Jul 28 10:05:38.029508 ip-10-0-141-50 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 28 10:05:38.029520 ip-10-0-141-50 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 28 10:05:38.029527 ip-10-0-141-50 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 28 10:05:38.029744 ip-10-0-141-50 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 28 10:05:48.249521 ip-10-0-141-50 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 28 10:05:48.249540 ip-10-0-141-50 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 4f5f4d18a06c45ecb3997a484c10b9b1 --
Jul 28 10:07:45.624840 ip-10-0-141-50 systemd[1]: Starting Kubernetes Kubelet...
Jul 28 10:07:45.984865 ip-10-0-141-50 kubenswrapper[2579]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 10:07:45.984865 ip-10-0-141-50 kubenswrapper[2579]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 28 10:07:45.984865 ip-10-0-141-50 kubenswrapper[2579]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 10:07:45.984865 ip-10-0-141-50 kubenswrapper[2579]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 28 10:07:45.984865 ip-10-0-141-50 kubenswrapper[2579]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 10:07:45.985775 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:45.985690    2579 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 28 10:07:45.990820 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.990804    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 10:07:45.990860 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.990821    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 10:07:45.990860 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.990827    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 10:07:45.990860 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.990830    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 10:07:45.990860 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.990834    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 10:07:45.990860 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.990837    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 10:07:45.990860 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.990840    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 10:07:45.990860 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.990842    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 10:07:45.990860 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.990845    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 10:07:45.990860 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.990848    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 10:07:45.990860 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.990850    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 10:07:45.990860 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.990853    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 10:07:45.990860 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.990856    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 10:07:45.990860 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.990858    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 10:07:45.990860 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.990861    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 10:07:45.990860 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.990865    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 10:07:45.990860 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.990868    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 10:07:45.991279 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.990871    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 10:07:45.991279 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.990874    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 10:07:45.991279 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.990877    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 10:07:45.991279 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.990879    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 10:07:45.991279 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.990882    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 10:07:45.991279 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.990885    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 10:07:45.991279 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.990888    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 10:07:45.991279 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.990890    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 10:07:45.991279 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.990893    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 10:07:45.991279 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.990895    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 10:07:45.991279 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.990898    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 10:07:45.991279 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.990901    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 10:07:45.991279 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.990903    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 10:07:45.991279 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.990906    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 10:07:45.991279 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.990908    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 10:07:45.991279 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.990911    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 10:07:45.991279 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.990913    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 10:07:45.991279 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.990916    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 10:07:45.991279 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.990919    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 10:07:45.991279 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.990921    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 10:07:45.991783 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.990923    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 10:07:45.991783 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.990927    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 10:07:45.991783 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.990929    2579 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 10:07:45.991783 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.990932    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 10:07:45.991783 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.990934    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 10:07:45.991783 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.990937    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 10:07:45.991783 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.990940    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 10:07:45.991783 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.990942    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 10:07:45.991783 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.990945    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 10:07:45.991783 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.990947    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 10:07:45.991783 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.990950    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 10:07:45.991783 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.990952    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 10:07:45.991783 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.990954    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 10:07:45.991783 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.990958    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 10:07:45.991783 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.990961    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 10:07:45.991783 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.990964    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 10:07:45.991783 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.990966    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 10:07:45.991783 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.990969    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 10:07:45.991783 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.990972    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 10:07:45.991783 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.990974    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 10:07:45.992264 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.990977    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 10:07:45.992264 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.990979    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 10:07:45.992264 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.990982    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 10:07:45.992264 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.990984    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 10:07:45.992264 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.990987    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 10:07:45.992264 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.990989    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 10:07:45.992264 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.990992    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 10:07:45.992264 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.990994    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 10:07:45.992264 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.990997    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 10:07:45.992264 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.991000    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 10:07:45.992264 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.991002    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 10:07:45.992264 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.991005    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 10:07:45.992264 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.991007    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 10:07:45.992264 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.991010    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 10:07:45.992264 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.991015    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 10:07:45.992264 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.991019    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 10:07:45.992264 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.991023    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 10:07:45.992264 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.991026    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 10:07:45.992264 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.991029    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 10:07:45.992751 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.991032    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 10:07:45.992751 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.991034    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 10:07:45.992751 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.991037    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 10:07:45.992751 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.991039    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 10:07:45.992751 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.991042    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 10:07:45.992751 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.991044    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 10:07:45.992751 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.991048    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 10:07:45.992751 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.991051    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 10:07:45.992751 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.991053    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 10:07:45.992751 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.991056    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 10:07:45.992751 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.991059    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 10:07:45.992751 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.991061    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 10:07:45.992751 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.991064    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 10:07:45.992751 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.991067    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 10:07:45.992751 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.991069    2579 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 10:07:45.992751 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.991072    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 10:07:45.992751 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.991074    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 10:07:45.992751 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.991076    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 10:07:45.992751 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.991079    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 10:07:45.992751 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.991081    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 10:07:45.993234 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.991084    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 10:07:45.993234 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.991086    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 10:07:45.993234 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.991089    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 10:07:45.993234 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.991819    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 10:07:45.993234 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.991825    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 10:07:45.993234 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.991828    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 10:07:45.993234 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.991831    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 10:07:45.993234 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.991834    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 10:07:45.993234 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.991837    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 10:07:45.993234 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.991840    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 10:07:45.993234 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.991844    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 10:07:45.993234 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.991846    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 10:07:45.993234 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.991849    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 10:07:45.993234 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.991851    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 10:07:45.993234 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.991855    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 10:07:45.993234 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.991858    2579 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 10:07:45.993234 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.991860    2579 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 10:07:45.993234 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.991863    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 10:07:45.993234 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.991865    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 10:07:45.993234 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.991868    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 10:07:45.993754 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.991870    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 10:07:45.993754 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.991873    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 10:07:45.993754 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.991875    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 10:07:45.993754 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.991878    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 10:07:45.993754 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.991880    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 10:07:45.993754 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.991883    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 10:07:45.993754 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.991885    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 10:07:45.993754 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.991888    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 10:07:45.993754 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.991890    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 10:07:45.993754 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.991893    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 10:07:45.993754 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.991895    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 10:07:45.993754 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.991898    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 10:07:45.993754 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.991900    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 10:07:45.993754 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.991903    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 10:07:45.993754 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.991905    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 10:07:45.993754 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.991907    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 10:07:45.993754 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.991910    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 10:07:45.993754 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.991913    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 10:07:45.993754 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.991915    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 10:07:45.993754 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.991918    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 10:07:45.994237 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.991920    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 10:07:45.994237 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.991924    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 10:07:45.994237 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.991927    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 10:07:45.994237 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.991930    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 10:07:45.994237 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.991932    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 10:07:45.994237 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.991935    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 10:07:45.994237 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.991937    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 10:07:45.994237 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.991940    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 10:07:45.994237 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.991942    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 10:07:45.994237 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.991945    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 10:07:45.994237 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.991947    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 10:07:45.994237 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.991950    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 10:07:45.994237 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.991952    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 10:07:45.994237 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.991956    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 10:07:45.994237 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.991958    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 10:07:45.994237 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.991961    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 10:07:45.994237 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.991963    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 10:07:45.994237 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.991965    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 10:07:45.994237 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.991968    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 10:07:45.994237 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.991971    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 10:07:45.994740 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.991973    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 10:07:45.994740 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.991975    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 10:07:45.994740 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.991978    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 10:07:45.994740 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.991980    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 10:07:45.994740 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.991983    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 10:07:45.994740 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.991987    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 10:07:45.994740 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.991991    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 10:07:45.994740 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.991993    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 10:07:45.994740 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.991996    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 10:07:45.994740 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.991999    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 10:07:45.994740 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.992002    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 10:07:45.994740 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.992005    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 10:07:45.994740 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.992008    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 10:07:45.994740 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.992011    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 10:07:45.994740 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.992015    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 10:07:45.994740 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.992017    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 10:07:45.994740 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.992019    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 10:07:45.994740 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.992022    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 10:07:45.994740 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.992024    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 10:07:45.995311 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.992027    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 10:07:45.995311 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.992029    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 10:07:45.995311 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.992032    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 10:07:45.995311 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.992034    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 10:07:45.995311 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.992037    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 10:07:45.995311 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.992040    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 10:07:45.995311 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.992042    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 10:07:45.995311 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.992044    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 10:07:45.995311 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.992047    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 10:07:45.995311 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.992049    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 10:07:45.995311 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.992053    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 10:07:45.995311 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.992056    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 10:07:45.995311 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.992061    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 10:07:45.995311 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.992063    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 10:07:45.995311 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.992066    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 10:07:45.995311 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.992069    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 10:07:45.995311 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.992071    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 10:07:45.995311 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.992074    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 10:07:45.995311 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.992077    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 10:07:45.995790 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.992079    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 10:07:45.995790 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.992082    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 10:07:45.995790 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.992086    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 10:07:45.995790 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.992090    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 10:07:45.995790 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.993031    2579 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 10:07:45.995790 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.993038    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 10:07:45.995790 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.993042    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 10:07:45.995790 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.993046    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 10:07:45.995790 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.993049    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 10:07:45.995790 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.993052    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 10:07:45.995790 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.993055    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 10:07:45.995790 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.993058    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 10:07:45.995790 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.993060    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 10:07:45.995790 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.993063    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 10:07:45.995790 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.993066    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 10:07:45.995790 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.993069    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 10:07:45.995790 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.993072    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 10:07:45.995790 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.993074    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 10:07:45.995790 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.993077    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 10:07:45.995790 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.993080    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 10:07:45.996279 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.993084    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 10:07:45.996279 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.993087    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 10:07:45.996279 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.993090    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 10:07:45.996279 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.993093    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 10:07:45.996279 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.993097    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 10:07:45.996279 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.993100    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 10:07:45.996279 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.993103    2579 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 10:07:45.996279 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.993106    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 10:07:45.996279 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.993109    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 10:07:45.996279 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.993112    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 10:07:45.996279 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.993115    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 10:07:45.996279 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.993118    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 10:07:45.996279 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.993121    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 10:07:45.996279 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.993124    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 10:07:45.996279 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.993126    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 10:07:45.996279 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.993129    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 10:07:45.996279 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.993132    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 10:07:45.996279 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.993134    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 10:07:45.996279 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.993137    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 10:07:45.996786 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.993139    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 10:07:45.996786 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.993142    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 10:07:45.996786 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.993145    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 10:07:45.996786 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.993147    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 10:07:45.996786 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.993150    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 10:07:45.996786 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.993152    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 10:07:45.996786 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.993154    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 10:07:45.996786 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.993157    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 10:07:45.996786 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.993159    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 10:07:45.996786 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.993162    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 10:07:45.996786 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.993165    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 10:07:45.996786 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.993168    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 10:07:45.996786 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.993170    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 10:07:45.996786 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.993173    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 10:07:45.996786 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.993175    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 10:07:45.996786 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.993178    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 10:07:45.996786 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.993180    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 10:07:45.996786 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.993182    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 10:07:45.996786 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.993185    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 10:07:45.996786 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.993187    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 10:07:45.997263 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.993190    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 10:07:45.997263 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.993193    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 10:07:45.997263 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.993195    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 10:07:45.997263 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.993198    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 10:07:45.997263 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.993201    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 10:07:45.997263 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.993203    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 10:07:45.997263 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.993206    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 10:07:45.997263 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.993209    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 10:07:45.997263 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.993211    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 10:07:45.997263 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.993214    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 10:07:45.997263 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.993216    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 10:07:45.997263 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.993219    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 10:07:45.997263 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.993221    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 10:07:45.997263 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.993224    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 10:07:45.997263 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.993226    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 10:07:45.997263 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.993230    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 10:07:45.997263 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.993232    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 10:07:45.997263 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.993235    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 10:07:45.997263 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.993238    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 10:07:45.997802 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.993240    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 10:07:45.997802 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.993243    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 10:07:45.997802 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.993247    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 10:07:45.997802 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.993249    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 10:07:45.997802 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.993252    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 10:07:45.997802 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.993254    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 10:07:45.997802 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.993257    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 10:07:45.997802 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.993259    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 10:07:45.997802 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.993262    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 10:07:45.997802 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.993264    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 10:07:45.997802 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.993267    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 10:07:45.997802 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.993269    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 10:07:45.997802 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.993272    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 10:07:45.997802 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.993274    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 10:07:45.997802 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.993277    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 10:07:45.997802 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.993279    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 10:07:45.997802 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.993281    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 10:07:45.997802 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.993284    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 10:07:45.997802 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.993286    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 10:07:45.997802 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.993289    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 10:07:45.998298 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.993291    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 10:07:45.998298 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.993293    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 10:07:45.998298 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.993296    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 10:07:45.998298 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.993298    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 10:07:45.998298 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.993301    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 10:07:45.998298 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:45.993364    2579 flags.go:64] FLAG: --address="0.0.0.0"
Jul 28 10:07:45.998298 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:45.993371    2579 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 28 10:07:45.998298 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:45.993378    2579 flags.go:64] FLAG: --anonymous-auth="true"
Jul 28 10:07:45.998298 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:45.993382    2579 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 28 10:07:45.998298 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:45.993387    2579 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 28 10:07:45.998298 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:45.993391    2579 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 28 10:07:45.998298 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:45.993395    2579 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 28 10:07:45.998298 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:45.993399    2579 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 28 10:07:45.998298 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:45.993403    2579 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 28 10:07:45.998298 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:45.993405    2579 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 28 10:07:45.998298 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:45.993409    2579 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 28 10:07:45.998298 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:45.993412    2579 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 28 10:07:45.998298 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:45.993415    2579 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 28 10:07:45.998298 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:45.993418    2579 flags.go:64] FLAG: --cgroup-root=""
Jul 28 10:07:45.998298 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:45.993421    2579 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 28 10:07:45.998298 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:45.993424    2579 flags.go:64] FLAG: --client-ca-file=""
Jul 28 10:07:45.998298 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:45.993427    2579 flags.go:64] FLAG: --cloud-provider="external"
Jul 28 10:07:45.998298 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:45.993430    2579 flags.go:64] FLAG: --cluster-dns="[]"
Jul 28 10:07:45.998871 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:45.993434    2579 flags.go:64] FLAG: --cluster-domain=""
Jul 28 10:07:45.998871 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:45.993437    2579 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 28 10:07:45.998871 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:45.993440    2579 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 28 10:07:45.998871 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:45.993443    2579 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 28 10:07:45.998871 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:45.993446    2579 flags.go:64] FLAG: --container-log-max-files="5"
Jul 28 10:07:45.998871 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:45.993450    2579 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 28 10:07:45.998871 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:45.993452    2579 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 28 10:07:45.998871 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:45.993455    2579 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 28 10:07:45.998871 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:45.993459    2579 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 28 10:07:45.998871 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:45.993462    2579 flags.go:64] FLAG: --contention-profiling="false"
Jul 28 10:07:45.998871 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:45.993465    2579 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 28 10:07:45.998871 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:45.993468    2579 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 28 10:07:45.998871 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:45.993471    2579 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 28 10:07:45.998871 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:45.993475    2579 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 28 10:07:45.998871 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:45.993479    2579 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 28 10:07:45.998871 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:45.993482    2579 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 28 10:07:45.998871 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:45.993485    2579 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 28 10:07:45.998871 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:45.993488    2579 flags.go:64] FLAG: --enable-load-reader="false"
Jul 28 10:07:45.998871 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:45.993492    2579 flags.go:64] FLAG: --enable-server="true"
Jul 28 10:07:45.998871 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:45.993495    2579 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 28 10:07:45.998871 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:45.993500    2579 flags.go:64] FLAG: --event-burst="100"
Jul 28 10:07:45.998871 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:45.993503    2579 flags.go:64] FLAG: --event-qps="50"
Jul 28 10:07:45.998871 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:45.993506    2579 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 28 10:07:45.998871 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:45.993509    2579 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 28 10:07:45.999448 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:45.993512    2579 flags.go:64] FLAG: --eviction-hard=""
Jul 28 10:07:45.999448 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:45.993516    2579 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 28 10:07:45.999448 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:45.993518    2579 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 28 10:07:45.999448 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:45.993522    2579 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 28 10:07:45.999448 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:45.993524    2579 flags.go:64] FLAG: --eviction-soft=""
Jul 28 10:07:45.999448 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:45.993527    2579 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 28 10:07:45.999448 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:45.993530    2579 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 28 10:07:45.999448 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:45.993545    2579 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 28 10:07:45.999448 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:45.993548    2579 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 28 10:07:45.999448 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:45.993551    2579 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 28 10:07:45.999448 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:45.993554    2579 flags.go:64] FLAG: --fail-swap-on="true"
Jul 28 10:07:45.999448 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:45.993556    2579 flags.go:64] FLAG: --feature-gates=""
Jul 28 10:07:45.999448 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:45.993560    2579 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 28 10:07:45.999448 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:45.993563    2579 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 28 10:07:45.999448 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:45.993566    2579 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 28 10:07:45.999448 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:45.993569    2579 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 28 10:07:45.999448 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:45.993576    2579 flags.go:64] FLAG: --healthz-port="10248"
Jul 28 10:07:45.999448 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:45.993579    2579 flags.go:64] FLAG: --help="false"
Jul 28 10:07:45.999448 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:45.993582    2579 flags.go:64] FLAG: --hostname-override="ip-10-0-141-50.ec2.internal"
Jul 28 10:07:45.999448 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:45.993585    2579 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 28 10:07:45.999448 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:45.993588    2579 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 28 10:07:45.999448 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:45.993591    2579 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 28 10:07:45.999448 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:45.993594    2579 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 28 10:07:45.999448 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:45.993598    2579 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 28 10:07:46.000066 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:45.993600    2579 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 28 10:07:46.000066 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:45.993603    2579 flags.go:64] FLAG: --image-service-endpoint=""
Jul 28 10:07:46.000066 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:45.993606    2579 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 28 10:07:46.000066 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:45.993609    2579 flags.go:64] FLAG: --kube-api-burst="100"
Jul 28 10:07:46.000066 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:45.993613    2579 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 28 10:07:46.000066 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:45.993616    2579 flags.go:64] FLAG: --kube-api-qps="50"
Jul 28 10:07:46.000066 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:45.993618    2579 flags.go:64] FLAG: --kube-reserved=""
Jul 28 10:07:46.000066 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:45.993621    2579 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 28 10:07:46.000066 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:45.993624    2579 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 28 10:07:46.000066 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:45.993627    2579 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 28 10:07:46.000066 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:45.993630    2579 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 28 10:07:46.000066 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:45.993633    2579 flags.go:64] FLAG: --lock-file=""
Jul 28 10:07:46.000066 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:45.993635    2579 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 28 10:07:46.000066 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:45.993638    2579 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 28 10:07:46.000066 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:45.993641    2579 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 28 10:07:46.000066 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:45.993647    2579 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 28 10:07:46.000066 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:45.993649    2579 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 28 10:07:46.000066 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:45.993652    2579 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 28 10:07:46.000066 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:45.993655    2579 flags.go:64] FLAG: --logging-format="text"
Jul 28 10:07:46.000066 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:45.993658    2579 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 28 10:07:46.000066 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:45.993662    2579 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 28 10:07:46.000066 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:45.993664    2579 flags.go:64] FLAG: --manifest-url=""
Jul 28 10:07:46.000066 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:45.993667    2579 flags.go:64] FLAG: --manifest-url-header=""
Jul 28 10:07:46.000066 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:45.993671    2579 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 28 10:07:46.000066 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:45.993676    2579 flags.go:64] FLAG: --max-open-files="1000000"
Jul 28 10:07:46.000724 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:45.993680    2579 flags.go:64] FLAG: --max-pods="110"
Jul 28 10:07:46.000724 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:45.993683    2579 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 28 10:07:46.000724 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:45.993686    2579 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 28 10:07:46.000724 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:45.993689    2579 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 28 10:07:46.000724 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:45.993692    2579 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 28 10:07:46.000724 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:45.993699    2579 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 28 10:07:46.000724 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:45.993702    2579 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 28 10:07:46.000724 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:45.993705    2579 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 28 10:07:46.000724 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:45.993711    2579 flags.go:64] FLAG: --node-status-max-images="50"
Jul 28 10:07:46.000724 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:45.993714    2579 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 28 10:07:46.000724 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:45.993717    2579 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 28 10:07:46.000724 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:45.993720    2579 flags.go:64] FLAG: --pod-cidr=""
Jul 28 10:07:46.000724 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:45.993724    2579 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb"
Jul 28 10:07:46.000724 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:45.993730    2579 flags.go:64] FLAG: --pod-manifest-path=""
Jul 28 10:07:46.000724 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:45.993733    2579 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 28 10:07:46.000724 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:45.993736    2579 flags.go:64] FLAG: --pods-per-core="0"
Jul 28 10:07:46.000724 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:45.993739    2579 flags.go:64] FLAG: --port="10250"
Jul 28 10:07:46.000724 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:45.993742    2579 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 28 10:07:46.000724 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:45.993745    2579 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-072f58415dfb05282"
Jul 28 10:07:46.000724 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:45.993748    2579 flags.go:64] FLAG: --qos-reserved=""
Jul 28 10:07:46.000724 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:45.993751    2579 flags.go:64] FLAG: --read-only-port="10255"
Jul 28 10:07:46.000724 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:45.993754    2579 flags.go:64] FLAG: --register-node="true"
Jul 28 10:07:46.000724 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:45.993757    2579 flags.go:64] FLAG: --register-with-taints=""
Jul 28 10:07:46.000724 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:45.993761    2579 flags.go:64] FLAG: --registry-burst="10"
Jul 28 10:07:46.001352 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:45.993764    2579 flags.go:64] FLAG: --registry-qps="5"
Jul 28 10:07:46.001352 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:45.993767    2579 flags.go:64] FLAG: --reserved-cpus=""
Jul 28 10:07:46.001352 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:45.993770    2579 flags.go:64] FLAG: --reserved-memory=""
Jul 28 10:07:46.001352 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:45.993779    2579 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 28 10:07:46.001352 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:45.993782    2579 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 28 10:07:46.001352 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:45.993785    2579 flags.go:64] FLAG: --rotate-certificates="false"
Jul 28 10:07:46.001352 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:45.993787    2579 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 28 10:07:46.001352 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:45.993790    2579 flags.go:64] FLAG: --runonce="false"
Jul 28 10:07:46.001352 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:45.993795    2579 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 28 10:07:46.001352 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:45.993798    2579 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 28 10:07:46.001352 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:45.993801    2579 flags.go:64] FLAG: --seccomp-default="false"
Jul 28 10:07:46.001352 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:45.993804    2579 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 28 10:07:46.001352 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:45.993806    2579 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 28 10:07:46.001352 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:45.993810    2579 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 28 10:07:46.001352 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:45.993814    2579 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 28 10:07:46.001352 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:45.993817    2579 flags.go:64] FLAG: --storage-driver-password="root"
Jul 28 10:07:46.001352 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:45.993820    2579 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 28 10:07:46.001352 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:45.993823    2579 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 28 10:07:46.001352 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:45.993826    2579 flags.go:64] FLAG: --storage-driver-user="root"
Jul 28 10:07:46.001352 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:45.993829    2579 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 28 10:07:46.001352 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:45.993832    2579 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 28 10:07:46.001352 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:45.993835    2579 flags.go:64] FLAG: --system-cgroups=""
Jul 28 10:07:46.001352 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:45.993838    2579 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 28 10:07:46.001352 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:45.993844    2579 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 28 10:07:46.001352 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:45.993846    2579 flags.go:64] FLAG: --tls-cert-file=""
Jul 28 10:07:46.001991 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:45.993849    2579 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 28 10:07:46.001991 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:45.993853    2579 flags.go:64] FLAG: --tls-min-version=""
Jul 28 10:07:46.001991 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:45.993855    2579 flags.go:64] FLAG: --tls-private-key-file=""
Jul 28 10:07:46.001991 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:45.993859    2579 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 28 10:07:46.001991 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:45.993861    2579 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 28 10:07:46.001991 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:45.993864    2579 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 28 10:07:46.001991 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:45.993867    2579 flags.go:64] FLAG: --v="2"
Jul 28 10:07:46.001991 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:45.993871    2579 flags.go:64] FLAG: --version="false"
Jul 28 10:07:46.001991 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:45.993875    2579 flags.go:64] FLAG: --vmodule=""
Jul 28 10:07:46.001991 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:45.993879    2579 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 28 10:07:46.001991 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:45.993882    2579 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 28 10:07:46.001991 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.993990    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 10:07:46.001991 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.993994    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 10:07:46.001991 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.993996    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 10:07:46.001991 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.993999    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 10:07:46.001991 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.994003    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 10:07:46.001991 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.994006    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 10:07:46.001991 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.994008    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 10:07:46.001991 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.994011    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 10:07:46.001991 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.994013    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 10:07:46.001991 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.994017    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 10:07:46.001991 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.994020    2579 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 10:07:46.001991 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.994022    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 10:07:46.002558 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.994026    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 10:07:46.002558 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.994028    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 10:07:46.002558 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.994031    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 10:07:46.002558 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.994033    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 10:07:46.002558 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.994036    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 10:07:46.002558 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.994038    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 10:07:46.002558 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.994042    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 10:07:46.002558 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.994044    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 10:07:46.002558 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.994047    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 10:07:46.002558 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.994050    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 10:07:46.002558 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.994053    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 10:07:46.002558 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.994055    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 10:07:46.002558 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.994058    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 10:07:46.002558 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.994060    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 10:07:46.002558 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.994063    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 10:07:46.002558 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.994065    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 10:07:46.002558 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.994068    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 10:07:46.002558 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.994070    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 10:07:46.002558 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.994073    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 10:07:46.003011 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.994075    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 10:07:46.003011 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.994077    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 10:07:46.003011 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.994080    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 10:07:46.003011 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.994082    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 10:07:46.003011 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.994085    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 10:07:46.003011 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.994090    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 10:07:46.003011 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.994093    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 10:07:46.003011 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.994097    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 10:07:46.003011 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.994099    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 10:07:46.003011 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.994102    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 10:07:46.003011 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.994106    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 10:07:46.003011 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.994108    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 10:07:46.003011 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.994111    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 10:07:46.003011 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.994114    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 10:07:46.003011 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.994116    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 10:07:46.003011 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.994119    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 10:07:46.003011 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.994121    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 10:07:46.003011 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.994124    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 10:07:46.003011 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.994126    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 10:07:46.003011 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.994129    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 10:07:46.003508 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.994132    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 10:07:46.003508 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.994135    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 10:07:46.003508 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.994137    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 10:07:46.003508 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.994140    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 10:07:46.003508 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.994142    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 10:07:46.003508 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.994145    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 10:07:46.003508 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.994147    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 10:07:46.003508 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.994150    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 10:07:46.003508 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.994152    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 10:07:46.003508 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.994155    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 10:07:46.003508 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.994158    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 10:07:46.003508 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.994160    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 10:07:46.003508 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.994162    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 10:07:46.003508 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.994165    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 10:07:46.003508 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.994167    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 10:07:46.003508 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.994170    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 10:07:46.003508 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.994172    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 10:07:46.003508 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.994176    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 10:07:46.003508 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.994178    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 10:07:46.003508 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.994181    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 10:07:46.004012 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.994183    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 10:07:46.004012 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.994186    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 10:07:46.004012 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.994191    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 10:07:46.004012 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.994194    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 10:07:46.004012 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.994197    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 10:07:46.004012 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.994199    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 10:07:46.004012 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.994202    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 10:07:46.004012 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.994205    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 10:07:46.004012 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.994208    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 10:07:46.004012 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.994210    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 10:07:46.004012 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.994213    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 10:07:46.004012 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.994215    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 10:07:46.004012 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.994218    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 10:07:46.004012 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.994221    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 10:07:46.004012 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.994223    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 10:07:46.004012 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.994226    2579 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 10:07:46.004012 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.994228    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 10:07:46.004012 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.994232    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 10:07:46.004012 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.994235    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 10:07:46.004485 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.994239    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 10:07:46.004485 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.994242    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 10:07:46.004485 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.994245    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 10:07:46.004485 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.994247    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 10:07:46.004485 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.994250    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 10:07:46.004485 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.994254    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 10:07:46.004485 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.994256    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 10:07:46.004485 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.994259    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 10:07:46.004485 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:45.994261    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 10:07:46.004485 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:45.994820    2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 10:07:46.004485 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:46.001567    2579 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 28 10:07:46.004485 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:46.001582    2579 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 28 10:07:46.004485 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:46.001598    2579 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 28 10:07:46.004485 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:46.001604    2579 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 28 10:07:46.004877 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.001662    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 10:07:46.004877 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.001667    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 10:07:46.004877 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.001671    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 10:07:46.004877 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.001674    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 10:07:46.004877 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.001677    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 10:07:46.004877 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.001680    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 10:07:46.004877 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.001683    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 10:07:46.004877 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.001687    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 10:07:46.004877 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.001692    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 10:07:46.004877 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.001695    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 10:07:46.004877 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.001698    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 10:07:46.004877 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.001701    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 10:07:46.004877 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.001706    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 10:07:46.004877 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.001710    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 10:07:46.004877 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.001713    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 10:07:46.004877 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.001716    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 10:07:46.004877 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.001719    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 10:07:46.004877 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.001722    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 10:07:46.005325 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.001724    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 10:07:46.005325 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.001726    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 10:07:46.005325 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.001729    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 10:07:46.005325 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.001732    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 10:07:46.005325 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.001734    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 10:07:46.005325 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.001737    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 10:07:46.005325 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.001740    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 10:07:46.005325 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.001742    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 10:07:46.005325 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.001745    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 10:07:46.005325 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.001747    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 10:07:46.005325 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.001750    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 10:07:46.005325 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.001753    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 10:07:46.005325 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.001756    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 10:07:46.005325 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.001759    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 10:07:46.005325 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.001761    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 10:07:46.005325 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.001764    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 10:07:46.005325 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.001766    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 10:07:46.005325 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.001769    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 10:07:46.005325 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.001771    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 10:07:46.005325 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.001774    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 10:07:46.005852 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.001776    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 10:07:46.005852 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.001779    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 10:07:46.005852 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.001781    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 10:07:46.005852 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.001784    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 10:07:46.005852 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.001787    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 10:07:46.005852 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.001789    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 10:07:46.005852 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.001793    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 10:07:46.005852 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.001797    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 10:07:46.005852 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.001800    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 10:07:46.005852 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.001803    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 10:07:46.005852 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.001806    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 10:07:46.005852 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.001809    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 10:07:46.005852 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.001811    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 10:07:46.005852 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.001814    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 10:07:46.005852 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.001816    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 10:07:46.005852 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.001819    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 10:07:46.005852 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.001821    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 10:07:46.005852 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.001824    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 10:07:46.005852 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.001826    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 10:07:46.006494 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.001830    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 10:07:46.006494 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.001832    2579 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 10:07:46.006494 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.001835    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 10:07:46.006494 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.001838    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 10:07:46.006494 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.001840    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 10:07:46.006494 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.001843    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 10:07:46.006494 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.001846    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 10:07:46.006494 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.001848    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 10:07:46.006494 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.001851    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 10:07:46.006494 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.001853    2579 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 10:07:46.006494 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.001856    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 10:07:46.006494 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.001858    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 10:07:46.006494 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.001861    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 10:07:46.006494 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.001863    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 10:07:46.006494 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.001866    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 10:07:46.006494 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.001869    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 10:07:46.006494 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.001872    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 10:07:46.006494 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.001874    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 10:07:46.006494 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.001877    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 10:07:46.006494 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.001880    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 10:07:46.007086 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.001882    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 10:07:46.007086 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.001885    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 10:07:46.007086 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.001888    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 10:07:46.007086 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.001890    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 10:07:46.007086 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.001893    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 10:07:46.007086 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.001895    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 10:07:46.007086 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.001898    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 10:07:46.007086 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.001900    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 10:07:46.007086 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.001903    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 10:07:46.007086 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.001905    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 10:07:46.007086 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.001908    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 10:07:46.007086 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.001910    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 10:07:46.007086 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.001913    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 10:07:46.007086 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.001915    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 10:07:46.007086 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.001918    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 10:07:46.007086 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.001920    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 10:07:46.007086 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.001923    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 10:07:46.007086 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.001926    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 10:07:46.007086 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.001928    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 10:07:46.007086 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.001931    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 10:07:46.007879 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.001933    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 10:07:46.007879 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.001935    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 10:07:46.007879 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:46.001941    2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 10:07:46.007879 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.002054    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 10:07:46.007879 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.002068    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 10:07:46.007879 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.002071    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 10:07:46.007879 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.002073    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 10:07:46.007879 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.002076    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 10:07:46.007879 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.002079    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 10:07:46.007879 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.002082    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 10:07:46.007879 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.002084    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 10:07:46.007879 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.002087    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 10:07:46.007879 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.002090    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 10:07:46.007879 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.002092    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 10:07:46.007879 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.002096    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 10:07:46.008260 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.002099    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 10:07:46.008260 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.002101    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 10:07:46.008260 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.002104    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 10:07:46.008260 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.002107    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 10:07:46.008260 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.002109    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 10:07:46.008260 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.002113    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 10:07:46.008260 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.002117    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 10:07:46.008260 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.002120    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 10:07:46.008260 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.002123    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 10:07:46.008260 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.002125    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 10:07:46.008260 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.002128    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 10:07:46.008260 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.002130    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 10:07:46.008260 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.002133    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 10:07:46.008260 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.002136    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 10:07:46.008260 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.002138    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 10:07:46.008260 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.002141    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 10:07:46.008260 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.002144    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 10:07:46.008260 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.002148    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 10:07:46.008260 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.002151    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 10:07:46.008874 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.002153    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 10:07:46.008874 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.002156    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 10:07:46.008874 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.002159    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 10:07:46.008874 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.002161    2579 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 10:07:46.008874 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.002164    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 10:07:46.008874 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.002166    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 10:07:46.008874 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.002169    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 10:07:46.008874 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.002171    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 10:07:46.008874 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.002174    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 10:07:46.008874 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.002178    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 10:07:46.008874 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.002181    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 10:07:46.008874 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.002184    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 10:07:46.008874 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.002187    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 10:07:46.008874 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.002190    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 10:07:46.008874 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.002193    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 10:07:46.008874 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.002195    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 10:07:46.008874 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.002198    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 10:07:46.008874 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.002201    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 10:07:46.008874 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.002204    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 10:07:46.008874 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.002207    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 10:07:46.009679 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.002209    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 10:07:46.009679 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.002212    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 10:07:46.009679 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.002215    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 10:07:46.009679 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.002217    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 10:07:46.009679 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.002219    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 10:07:46.009679 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.002222    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 10:07:46.009679 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.002224    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 10:07:46.009679 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.002227    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 10:07:46.009679 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.002229    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 10:07:46.009679 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.002231    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 10:07:46.009679 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.002234    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 10:07:46.009679 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.002236    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 10:07:46.009679 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.002239    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 10:07:46.009679 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.002241    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 10:07:46.009679 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.002243    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 10:07:46.009679 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.002246    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 10:07:46.009679 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.002248    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 10:07:46.009679 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.002251    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 10:07:46.009679 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.002253    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 10:07:46.009679 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.002256    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 10:07:46.010411 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.002258    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 10:07:46.010411 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.002261    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 10:07:46.010411 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.002263    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 10:07:46.010411 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.002266    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 10:07:46.010411 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.002268    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 10:07:46.010411 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.002271    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 10:07:46.010411 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.002273    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 10:07:46.010411 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.002276    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 10:07:46.010411 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.002278    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 10:07:46.010411 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.002281    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 10:07:46.010411 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.002284    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 10:07:46.010411 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.002286    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 10:07:46.010411 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.002289    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 10:07:46.010411 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.002291    2579 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 10:07:46.010411 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.002294    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 10:07:46.010411 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.002296    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 10:07:46.010411 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.002299    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 10:07:46.010411 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.002301    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 10:07:46.010411 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.002303    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 10:07:46.010411 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.002306    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 10:07:46.011130 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.002308    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 10:07:46.011130 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.002311    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 10:07:46.011130 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.002313    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 10:07:46.011130 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.002315    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 10:07:46.011130 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.002318    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 10:07:46.011130 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.002320    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 10:07:46.011130 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.002323    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 10:07:46.011130 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:46.002325    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 10:07:46.011130 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:46.002330    2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 10:07:46.011130 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:46.002429    2579 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 28 10:07:46.011130 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:46.004500    2579 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 28 10:07:46.011130 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:46.005817    2579 server.go:1019] "Starting client certificate rotation"
Jul 28 10:07:46.011130 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:46.005937    2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 28 10:07:46.011130 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:46.006026    2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 28 10:07:46.011491 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:46.010163    2579 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 28 10:07:46.015081 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:46.015062    2579 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 28 10:07:46.027905 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:46.027885    2579 log.go:25] "Validated CRI v1 runtime API"
Jul 28 10:07:46.030768 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:46.030755    2579 log.go:25] "Validated CRI v1 image API"
Jul 28 10:07:46.032879 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:46.032863    2579 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 28 10:07:46.035152 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:46.035132    2579 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 c310d2d9-3a38-4de3-a4eb-375d7eea86fc:/dev/nvme0n1p4 ccae3ba7-9e62-4807-80ce-f640418fe6f7:/dev/nvme0n1p3]
Jul 28 10:07:46.035208 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:46.035152    2579 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 28 10:07:46.040176 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:46.040158    2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 28 10:07:46.040687 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:46.040572    2579 manager.go:216] Machine: {Timestamp:2026-07-28 10:07:46.039316791 +0000 UTC m=+0.302713111 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101013 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2af33a5bd3fe559da179221baeb642 SystemUUID:ec2af33a-5bd3-fe55-9da1-79221baeb642 BootID:4f5f4d18-a06c-45ec-b399-7a484c10b9b1 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:51:05:24:bc:c5 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:51:05:24:bc:c5 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ee:16:43:7b:1c:fd Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 28 10:07:46.040687 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:46.040685    2579 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 28 10:07:46.040782 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:46.040766    2579 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 28 10:07:46.043355 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:46.043327    2579 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 28 10:07:46.043486 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:46.043357    2579 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-50.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 28 10:07:46.044091 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:46.044081    2579 topology_manager.go:138] "Creating topology manager with none policy"
Jul 28 10:07:46.044122 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:46.044094    2579 container_manager_linux.go:309] "Creating device plugin manager"
Jul 28 10:07:46.044122 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:46.044106    2579 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 28 10:07:46.044749 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:46.044735    2579 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 28 10:07:46.046055 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:46.046045    2579 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 28 10:07:46.047742 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:46.047731    2579 state_mem.go:36] "Initialized new in-memory state store"
Jul 28 10:07:46.047856 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:46.047847    2579 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 28 10:07:46.047888 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:46.047875    2579 kubelet.go:486] "Attempting to sync node with API server"
Jul 28 10:07:46.047888 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:46.047887    2579 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 28 10:07:46.047951 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:46.047902    2579 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 28 10:07:46.047951 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:46.047911    2579 kubelet.go:398] "Adding apiserver pod source"
Jul 28 10:07:46.047951 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:46.047918    2579 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 28 10:07:46.048412 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:46.048396    2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 28 10:07:46.048472 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:46.048426    2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 28 10:07:46.050660 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:46.050624    2579 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 28 10:07:46.054127 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:46.054106    2579 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 28 10:07:46.054195 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:46.054139    2579 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 28 10:07:46.055624 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:46.055611    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 28 10:07:46.055671 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:46.055629    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 28 10:07:46.055671 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:46.055636    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 28 10:07:46.055671 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:46.055642    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 28 10:07:46.055671 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:46.055650    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 28 10:07:46.055671 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:46.055656    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 28 10:07:46.055671 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:46.055664    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 28 10:07:46.055823 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:46.055696    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 28 10:07:46.055823 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:46.055704    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 28 10:07:46.055823 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:46.055710    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 28 10:07:46.055823 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:46.055718    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 28 10:07:46.055823 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:46.055723    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 28 10:07:46.056422 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:46.056414    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 28 10:07:46.056422 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:46.056423    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 28 10:07:46.059516 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:46.059502    2579 server.go:1268] "Started kubelet"
Jul 28 10:07:46.059618 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:46.059597    2579 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 28 10:07:46.059744 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:46.059697    2579 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 28 10:07:46.059789 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:46.059765    2579 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 28 10:07:46.060215 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:46.060195    2579 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-141-50.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 28 10:07:46.060334 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:07:46.060304    2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-141-50.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 28 10:07:46.060404 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:07:46.060379    2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 28 10:07:46.060420 ip-10-0-141-50 systemd[1]: Started Kubernetes Kubelet.
Jul 28 10:07:46.061585 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:46.061500    2579 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 28 10:07:46.061641 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:46.061621    2579 server.go:310] "Adding debug handlers to kubelet server"
Jul 28 10:07:46.065712 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:07:46.065694    2579 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 28 10:07:46.067625 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:07:46.066792    2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-50.ec2.internal.18c66c3268af8365  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-50.ec2.internal,UID:ip-10-0-141-50.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-141-50.ec2.internal,},FirstTimestamp:2026-07-28 10:07:46.059477861 +0000 UTC m=+0.322874180,LastTimestamp:2026-07-28 10:07:46.059477861 +0000 UTC m=+0.322874180,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-50.ec2.internal,}"
Jul 28 10:07:46.069341 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:46.069324    2579 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 28 10:07:46.069341 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:46.069338    2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 28 10:07:46.069470 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:07:46.069429    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-50.ec2.internal\" not found"
Jul 28 10:07:46.069795 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:46.069775    2579 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 28 10:07:46.069912 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:46.069779    2579 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 28 10:07:46.069912 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:46.069812    2579 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 28 10:07:46.070023 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:46.069931    2579 reconstruct.go:97] "Volume reconstruction finished"
Jul 28 10:07:46.070023 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:46.069942    2579 reconciler.go:29] "Reconciler: start to sync state"
Jul 28 10:07:46.070566 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:46.070523    2579 factory.go:55] Registering systemd factory
Jul 28 10:07:46.070637 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:46.070569    2579 factory.go:223] Registration of the systemd container factory successfully
Jul 28 10:07:46.071053 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:46.071000    2579 factory.go:153] Registering CRI-O factory
Jul 28 10:07:46.071053 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:46.071015    2579 factory.go:223] Registration of the crio container factory successfully
Jul 28 10:07:46.071204 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:46.071076    2579 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 28 10:07:46.071204 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:46.071099    2579 factory.go:103] Registering Raw factory
Jul 28 10:07:46.071204 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:46.071112    2579 manager.go:1196] Started watching for new ooms in manager
Jul 28 10:07:46.071506 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:46.071487    2579 manager.go:349] Starting recovery of all containers
Jul 28 10:07:46.073062 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:46.073038    2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-l6m99"
Jul 28 10:07:46.073466 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:07:46.073445    2579 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-141-50.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 28 10:07:46.073577 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:07:46.073553    2579 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 28 10:07:46.082106 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:46.082083    2579 manager.go:354] Recovery completed
Jul 28 10:07:46.082397 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:46.082375    2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-l6m99"
Jul 28 10:07:46.084196 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:07:46.084166    2579 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Jul 28 10:07:46.087109 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:46.087095    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 10:07:46.090209 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:46.090194    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-50.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 10:07:46.090279 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:46.090222    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-50.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 10:07:46.090279 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:46.090234    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-50.ec2.internal" event="NodeHasSufficientPID"
Jul 28 10:07:46.090711 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:46.090699    2579 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 28 10:07:46.090760 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:46.090712    2579 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 28 10:07:46.090760 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:46.090730    2579 state_mem.go:36] "Initialized new in-memory state store"
Jul 28 10:07:46.090855 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:46.090759    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 10:07:46.092297 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:46.092287    2579 policy_none.go:49] "None policy: Start"
Jul 28 10:07:46.092340 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:46.092301    2579 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 28 10:07:46.092340 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:46.092311    2579 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 28 10:07:46.093081 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:46.093066    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-50.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 10:07:46.093147 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:46.093091    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-50.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 10:07:46.093147 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:46.093101    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-50.ec2.internal" event="NodeHasSufficientPID"
Jul 28 10:07:46.093241 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:46.093230    2579 policy_none.go:47] "Start"
Jul 28 10:07:46.093578 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:07:46.093502    2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-50.ec2.internal.18c66c326a8471af  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-50.ec2.internal,UID:ip-10-0-141-50.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-141-50.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-141-50.ec2.internal,},FirstTimestamp:2026-07-28 10:07:46.090209711 +0000 UTC m=+0.353606030,LastTimestamp:2026-07-28 10:07:46.090209711 +0000 UTC m=+0.353606030,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-50.ec2.internal,}"
Jul 28 10:07:46.130800 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:46.130636    2579 manager.go:337] "Starting Device Plugin manager"
Jul 28 10:07:46.130882 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:07:46.130826    2579 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 28 10:07:46.130882 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:46.130840    2579 server.go:87] "Starting device plugin registration server"
Jul 28 10:07:46.131096 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:46.131084    2579 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 28 10:07:46.131130 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:46.131112    2579 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 28 10:07:46.131211 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:46.131192    2579 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 28 10:07:46.131294 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:46.131282    2579 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 28 10:07:46.131342 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:46.131298    2579 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 28 10:07:46.131811 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:07:46.131789    2579 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 28 10:07:46.131885 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:07:46.131848    2579 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-50.ec2.internal\" not found"
Jul 28 10:07:46.231782 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:46.231746    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 10:07:46.233102 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:46.232724    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-50.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 10:07:46.233102 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:46.232753    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-50.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 10:07:46.233102 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:46.232764    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-50.ec2.internal" event="NodeHasSufficientPID"
Jul 28 10:07:46.233102 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:46.232793    2579 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-141-50.ec2.internal"
Jul 28 10:07:46.239571 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:46.239484    2579 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-141-50.ec2.internal"
Jul 28 10:07:46.239571 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:07:46.239508    2579 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-141-50.ec2.internal\": node \"ip-10-0-141-50.ec2.internal\" not found"
Jul 28 10:07:46.256848 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:07:46.256832    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-50.ec2.internal\" not found"
Jul 28 10:07:46.267859 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:46.267834    2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 28 10:07:46.269621 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:46.269607    2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 28 10:07:46.269696 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:46.269649    2579 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 28 10:07:46.269696 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:46.269675    2579 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 28 10:07:46.269788 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:07:46.269715    2579 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 28 10:07:46.272245 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:46.272228    2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 10:07:46.357051 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:07:46.357012    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-50.ec2.internal\" not found"
Jul 28 10:07:46.370119 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:46.370091    2579 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-50.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-50.ec2.internal"]
Jul 28 10:07:46.370185 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:46.370161    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 10:07:46.371167 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:46.371143    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-50.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 10:07:46.371284 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:46.371178    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-50.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 10:07:46.371284 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:46.371189    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-50.ec2.internal" event="NodeHasSufficientPID"
Jul 28 10:07:46.372482 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:46.372469    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 10:07:46.372633 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:46.372616    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-50.ec2.internal"
Jul 28 10:07:46.372674 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:46.372653    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 10:07:46.373166 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:46.373149    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-50.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 10:07:46.373255 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:46.373173    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-50.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 10:07:46.373255 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:46.373187    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-50.ec2.internal" event="NodeHasSufficientPID"
Jul 28 10:07:46.373331 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:46.373262    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-50.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 10:07:46.373331 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:46.373285    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-50.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 10:07:46.373331 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:46.373295    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-50.ec2.internal" event="NodeHasSufficientPID"
Jul 28 10:07:46.374447 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:46.374431    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-50.ec2.internal"
Jul 28 10:07:46.374510 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:46.374455    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 10:07:46.375136 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:46.375110    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-50.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 10:07:46.375246 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:46.375144    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-50.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 10:07:46.375246 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:46.375158    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-50.ec2.internal" event="NodeHasSufficientPID"
Jul 28 10:07:46.398501 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:07:46.398477    2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-50.ec2.internal\" not found" node="ip-10-0-141-50.ec2.internal"
Jul 28 10:07:46.403317 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:07:46.403299    2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-50.ec2.internal\" not found" node="ip-10-0-141-50.ec2.internal"
Jul 28 10:07:46.457126 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:07:46.457096    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-50.ec2.internal\" not found"
Jul 28 10:07:46.471219 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:46.471194    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5d0794185faca7077a864950fb56bc94-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-50.ec2.internal\" (UID: \"5d0794185faca7077a864950fb56bc94\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-50.ec2.internal"
Jul 28 10:07:46.471299 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:46.471223    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5d0794185faca7077a864950fb56bc94-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-50.ec2.internal\" (UID: \"5d0794185faca7077a864950fb56bc94\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-50.ec2.internal"
Jul 28 10:07:46.471299 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:46.471243    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2efc845f7ae190bc5a73ce1b56253af6-config\") pod \"kube-apiserver-proxy-ip-10-0-141-50.ec2.internal\" (UID: \"2efc845f7ae190bc5a73ce1b56253af6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-50.ec2.internal"
Jul 28 10:07:46.557773 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:07:46.557691    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-50.ec2.internal\" not found"
Jul 28 10:07:46.572091 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:46.572060    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2efc845f7ae190bc5a73ce1b56253af6-config\") pod \"kube-apiserver-proxy-ip-10-0-141-50.ec2.internal\" (UID: \"2efc845f7ae190bc5a73ce1b56253af6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-50.ec2.internal"
Jul 28 10:07:46.572161 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:46.572096    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5d0794185faca7077a864950fb56bc94-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-50.ec2.internal\" (UID: \"5d0794185faca7077a864950fb56bc94\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-50.ec2.internal"
Jul 28 10:07:46.572161 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:46.572116    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5d0794185faca7077a864950fb56bc94-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-50.ec2.internal\" (UID: \"5d0794185faca7077a864950fb56bc94\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-50.ec2.internal"
Jul 28 10:07:46.572235 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:46.572167    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2efc845f7ae190bc5a73ce1b56253af6-config\") pod \"kube-apiserver-proxy-ip-10-0-141-50.ec2.internal\" (UID: \"2efc845f7ae190bc5a73ce1b56253af6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-50.ec2.internal"
Jul 28 10:07:46.572235 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:46.572180    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5d0794185faca7077a864950fb56bc94-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-50.ec2.internal\" (UID: \"5d0794185faca7077a864950fb56bc94\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-50.ec2.internal"
Jul 28 10:07:46.572235 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:46.572182    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5d0794185faca7077a864950fb56bc94-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-50.ec2.internal\" (UID: \"5d0794185faca7077a864950fb56bc94\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-50.ec2.internal"
Jul 28 10:07:46.664300 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:07:46.664250    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-50.ec2.internal\" not found"
Jul 28 10:07:46.703711 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:46.703677    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-50.ec2.internal"
Jul 28 10:07:46.708787 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:46.708438    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-50.ec2.internal"
Jul 28 10:07:46.764657 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:07:46.764618    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-50.ec2.internal\" not found"
Jul 28 10:07:46.868003 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:07:46.867890    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-50.ec2.internal\" not found"
Jul 28 10:07:46.968055 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:07:46.968000    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-50.ec2.internal\" not found"
Jul 28 10:07:47.005279 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:47.005248    2579 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 28 10:07:47.005716 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:47.005430    2579 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 28 10:07:47.068241 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:07:47.068208    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-50.ec2.internal\" not found"
Jul 28 10:07:47.087930 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:47.087894    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-27 10:02:46 +0000 UTC" deadline="2028-05-07 23:33:08.696749194 +0000 UTC"
Jul 28 10:07:47.087930 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:47.087922    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15589h25m21.608830586s"
Jul 28 10:07:47.169136 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:07:47.169060    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-50.ec2.internal\" not found"
Jul 28 10:07:47.269810 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:07:47.269784    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-50.ec2.internal\" not found"
Jul 28 10:07:47.277902 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:47.277871    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d0794185faca7077a864950fb56bc94.slice/crio-c088433ca2530b45316e90732f50c6d14c8c342f4b0715d5b1b31536ce343198 WatchSource:0}: Error finding container c088433ca2530b45316e90732f50c6d14c8c342f4b0715d5b1b31536ce343198: Status 404 returned error can't find the container with id c088433ca2530b45316e90732f50c6d14c8c342f4b0715d5b1b31536ce343198
Jul 28 10:07:47.282123 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:47.282109    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 10:07:47.337041 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:47.337009    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2efc845f7ae190bc5a73ce1b56253af6.slice/crio-647dfe6ba4327994f3b71d03fbe7827981a2ecd7a6383263b881cd56c326da16 WatchSource:0}: Error finding container 647dfe6ba4327994f3b71d03fbe7827981a2ecd7a6383263b881cd56c326da16: Status 404 returned error can't find the container with id 647dfe6ba4327994f3b71d03fbe7827981a2ecd7a6383263b881cd56c326da16
Jul 28 10:07:47.370050 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:07:47.370005    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-50.ec2.internal\" not found"
Jul 28 10:07:47.425659 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:47.425583    2579 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 10:07:47.425659 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:47.425600    2579 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 10:07:47.440380 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:47.440360    2579 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 10:07:47.470135 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:47.470101    2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-50.ec2.internal"
Jul 28 10:07:47.482378 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:47.482356    2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 28 10:07:47.483430 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:47.483416    2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-50.ec2.internal"
Jul 28 10:07:47.493553 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:47.493533    2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 28 10:07:48.015928 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.015895    2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 10:07:48.049226 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.049192    2579 apiserver.go:52] "Watching apiserver"
Jul 28 10:07:48.055160 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.055131    2579 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 28 10:07:48.058196 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.058164    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-cjlcq","kube-system/global-pull-secret-syncer-bffsx","kube-system/konnectivity-agent-2w2gl","kube-system/kube-apiserver-proxy-ip-10-0-141-50.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7sp7b","openshift-dns/node-resolver-2rf9h","openshift-multus/multus-gtxhl","openshift-cluster-node-tuning-operator/tuned-lcrsz","openshift-image-registry/node-ca-mkf4n","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-50.ec2.internal","openshift-multus/multus-additional-cni-plugins-fftp9","openshift-multus/network-metrics-daemon-vtxgv","openshift-network-diagnostics/network-check-target-4msch","openshift-network-operator/iptables-alerter-whcvk"]
Jul 28 10:07:48.060142 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.060115    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4msch"
Jul 28 10:07:48.060253 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:07:48.060214    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4msch" podUID="92436f2b-eb78-4e84-ab81-be7d1102f5ff"
Jul 28 10:07:48.061292 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.061266    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bffsx"
Jul 28 10:07:48.061388 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:07:48.061354    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bffsx" podUID="1fb2411c-4da6-44e6-882e-4f36fb1a26ea"
Jul 28 10:07:48.062604 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.062416    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-2w2gl"
Jul 28 10:07:48.063913 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.063889    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-cjlcq"
Jul 28 10:07:48.064647 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.064579    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 28 10:07:48.064742 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.064724    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-2dxng\""
Jul 28 10:07:48.064974 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.064954    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 28 10:07:48.065814 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.065793    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 28 10:07:48.065909 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.065820    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-xgb8w\""
Jul 28 10:07:48.066310 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.066269    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 28 10:07:48.066387 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.066372    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 28 10:07:48.066445 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.066422    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 28 10:07:48.066550 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.066294    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7sp7b"
Jul 28 10:07:48.066550 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.066531    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 28 10:07:48.066706 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.066674    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 28 10:07:48.067718 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.067696    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-2rf9h"
Jul 28 10:07:48.069640 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.068761    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 28 10:07:48.069640 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.068780    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-xbfxp\""
Jul 28 10:07:48.069640 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.069033    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-gtxhl"
Jul 28 10:07:48.069640 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.069042    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 28 10:07:48.069640 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.069039    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 28 10:07:48.070351 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.069660    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 28 10:07:48.070351 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.069771    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-4wbtr\""
Jul 28 10:07:48.070351 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.070303    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 28 10:07:48.072660 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.070603    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-lcrsz"
Jul 28 10:07:48.072660 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.071103    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 28 10:07:48.072660 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.071188    2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 28 10:07:48.072660 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.071307    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-q7gqj\""
Jul 28 10:07:48.072660 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.072047    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-mkf4n"
Jul 28 10:07:48.072660 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.072142    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 28 10:07:48.072660 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.072275    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 28 10:07:48.072660 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.072146    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 28 10:07:48.073380 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.072737    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-xxcmz\""
Jul 28 10:07:48.073380 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.072765    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 28 10:07:48.073380 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.072806    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 28 10:07:48.073380 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.073190    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-whcvk"
Jul 28 10:07:48.074254 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.074233    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 28 10:07:48.074254 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.074249    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 28 10:07:48.074746 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.074729    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-fftp9"
Jul 28 10:07:48.074994 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.074969    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-dfdvg\""
Jul 28 10:07:48.075106 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.075091    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 28 10:07:48.075172 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.074797    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 28 10:07:48.075672 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.075652    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 28 10:07:48.075762 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.075652    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-p92mv\""
Jul 28 10:07:48.075762 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.075753    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 28 10:07:48.076423 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.076404    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vtxgv"
Jul 28 10:07:48.076520 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:07:48.076486    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vtxgv" podUID="d67ab7ea-ff97-432b-b10a-df89bb204785"
Jul 28 10:07:48.077656 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.077593    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 28 10:07:48.078375 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.078259    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/20a3d596-59ed-46ce-b8f2-a9be7dc51b41-etc-modprobe-d\") pod \"tuned-lcrsz\" (UID: \"20a3d596-59ed-46ce-b8f2-a9be7dc51b41\") " pod="openshift-cluster-node-tuning-operator/tuned-lcrsz"
Jul 28 10:07:48.078375 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.078303    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/20a3d596-59ed-46ce-b8f2-a9be7dc51b41-run\") pod \"tuned-lcrsz\" (UID: \"20a3d596-59ed-46ce-b8f2-a9be7dc51b41\") " pod="openshift-cluster-node-tuning-operator/tuned-lcrsz"
Jul 28 10:07:48.078375 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.078336    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/20a3d596-59ed-46ce-b8f2-a9be7dc51b41-host\") pod \"tuned-lcrsz\" (UID: \"20a3d596-59ed-46ce-b8f2-a9be7dc51b41\") " pod="openshift-cluster-node-tuning-operator/tuned-lcrsz"
Jul 28 10:07:48.078375 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.078369    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1fb2411c-4da6-44e6-882e-4f36fb1a26ea-kubelet-config\") pod \"global-pull-secret-syncer-bffsx\" (UID: \"1fb2411c-4da6-44e6-882e-4f36fb1a26ea\") " pod="kube-system/global-pull-secret-syncer-bffsx"
Jul 28 10:07:48.078713 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.078401    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1d9e06b8-3c8a-41bd-8368-435c66ba1a52-host-run-netns\") pod \"multus-gtxhl\" (UID: \"1d9e06b8-3c8a-41bd-8368-435c66ba1a52\") " pod="openshift-multus/multus-gtxhl"
Jul 28 10:07:48.078713 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.078433    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f237d44f-f0d7-4d96-82b1-6a75f8c22704-host-cni-bin\") pod \"ovnkube-node-cjlcq\" (UID: \"f237d44f-f0d7-4d96-82b1-6a75f8c22704\") " pod="openshift-ovn-kubernetes/ovnkube-node-cjlcq"
Jul 28 10:07:48.078713 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.078463    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/aff18ade-5ca4-40b1-bee1-a593dc42d38c-konnectivity-ca\") pod \"konnectivity-agent-2w2gl\" (UID: \"aff18ade-5ca4-40b1-bee1-a593dc42d38c\") " pod="kube-system/konnectivity-agent-2w2gl"
Jul 28 10:07:48.078713 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.078494    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba296f38-5de8-4d11-b789-7a727a01b9cc-tmp-dir\") pod \"node-resolver-2rf9h\" (UID: \"ba296f38-5de8-4d11-b789-7a727a01b9cc\") " pod="openshift-dns/node-resolver-2rf9h"
Jul 28 10:07:48.078929 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.078728    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1d9e06b8-3c8a-41bd-8368-435c66ba1a52-os-release\") pod \"multus-gtxhl\" (UID: \"1d9e06b8-3c8a-41bd-8368-435c66ba1a52\") " pod="openshift-multus/multus-gtxhl"
Jul 28 10:07:48.078929 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.078757    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0e38cbfa-85ef-4a89-a8c0-5942584d4e30-socket-dir\") pod \"aws-ebs-csi-driver-node-7sp7b\" (UID: \"0e38cbfa-85ef-4a89-a8c0-5942584d4e30\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7sp7b"
Jul 28 10:07:48.078929 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.078785    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/20a3d596-59ed-46ce-b8f2-a9be7dc51b41-etc-kubernetes\") pod \"tuned-lcrsz\" (UID: \"20a3d596-59ed-46ce-b8f2-a9be7dc51b41\") " pod="openshift-cluster-node-tuning-operator/tuned-lcrsz"
Jul 28 10:07:48.079069 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.078960    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1d9e06b8-3c8a-41bd-8368-435c66ba1a52-host-var-lib-cni-multus\") pod \"multus-gtxhl\" (UID: \"1d9e06b8-3c8a-41bd-8368-435c66ba1a52\") " pod="openshift-multus/multus-gtxhl"
Jul 28 10:07:48.079069 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.079000    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f237d44f-f0d7-4d96-82b1-6a75f8c22704-host-run-ovn-kubernetes\") pod \"ovnkube-node-cjlcq\" (UID: \"f237d44f-f0d7-4d96-82b1-6a75f8c22704\") " pod="openshift-ovn-kubernetes/ovnkube-node-cjlcq"
Jul 28 10:07:48.079069 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.079032    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f237d44f-f0d7-4d96-82b1-6a75f8c22704-ovnkube-script-lib\") pod \"ovnkube-node-cjlcq\" (UID: \"f237d44f-f0d7-4d96-82b1-6a75f8c22704\") " pod="openshift-ovn-kubernetes/ovnkube-node-cjlcq"
Jul 28 10:07:48.079069 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.079059    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/20a3d596-59ed-46ce-b8f2-a9be7dc51b41-etc-sysconfig\") pod \"tuned-lcrsz\" (UID: \"20a3d596-59ed-46ce-b8f2-a9be7dc51b41\") " pod="openshift-cluster-node-tuning-operator/tuned-lcrsz"
Jul 28 10:07:48.079259 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.079206    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/20a3d596-59ed-46ce-b8f2-a9be7dc51b41-etc-systemd\") pod \"tuned-lcrsz\" (UID: \"20a3d596-59ed-46ce-b8f2-a9be7dc51b41\") " pod="openshift-cluster-node-tuning-operator/tuned-lcrsz"
Jul 28 10:07:48.079259 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.079237    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/20a3d596-59ed-46ce-b8f2-a9be7dc51b41-lib-modules\") pod \"tuned-lcrsz\" (UID: \"20a3d596-59ed-46ce-b8f2-a9be7dc51b41\") " pod="openshift-cluster-node-tuning-operator/tuned-lcrsz"
Jul 28 10:07:48.079354 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.079274    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f237d44f-f0d7-4d96-82b1-6a75f8c22704-env-overrides\") pod \"ovnkube-node-cjlcq\" (UID: \"f237d44f-f0d7-4d96-82b1-6a75f8c22704\") " pod="openshift-ovn-kubernetes/ovnkube-node-cjlcq"
Jul 28 10:07:48.079490 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.079458    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2s4l\" (UniqueName: \"kubernetes.io/projected/92436f2b-eb78-4e84-ab81-be7d1102f5ff-kube-api-access-h2s4l\") pod \"network-check-target-4msch\" (UID: \"92436f2b-eb78-4e84-ab81-be7d1102f5ff\") " pod="openshift-network-diagnostics/network-check-target-4msch"
Jul 28 10:07:48.079582 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.079496    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1d9e06b8-3c8a-41bd-8368-435c66ba1a52-multus-conf-dir\") pod \"multus-gtxhl\" (UID: \"1d9e06b8-3c8a-41bd-8368-435c66ba1a52\") " pod="openshift-multus/multus-gtxhl"
Jul 28 10:07:48.079582 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.079512    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 28 10:07:48.080089 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.079524    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f237d44f-f0d7-4d96-82b1-6a75f8c22704-run-openvswitch\") pod \"ovnkube-node-cjlcq\" (UID: \"f237d44f-f0d7-4d96-82b1-6a75f8c22704\") " pod="openshift-ovn-kubernetes/ovnkube-node-cjlcq"
Jul 28 10:07:48.080178 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.080117    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0e38cbfa-85ef-4a89-a8c0-5942584d4e30-registration-dir\") pod \"aws-ebs-csi-driver-node-7sp7b\" (UID: \"0e38cbfa-85ef-4a89-a8c0-5942584d4e30\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7sp7b"
Jul 28 10:07:48.080279 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.080261    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/20a3d596-59ed-46ce-b8f2-a9be7dc51b41-var-lib-kubelet\") pod \"tuned-lcrsz\" (UID: \"20a3d596-59ed-46ce-b8f2-a9be7dc51b41\") " pod="openshift-cluster-node-tuning-operator/tuned-lcrsz"
Jul 28 10:07:48.080351 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.080301    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f237d44f-f0d7-4d96-82b1-6a75f8c22704-var-lib-openvswitch\") pod \"ovnkube-node-cjlcq\" (UID: \"f237d44f-f0d7-4d96-82b1-6a75f8c22704\") " pod="openshift-ovn-kubernetes/ovnkube-node-cjlcq"
Jul 28 10:07:48.080351 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.080343    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f237d44f-f0d7-4d96-82b1-6a75f8c22704-host-cni-netd\") pod \"ovnkube-node-cjlcq\" (UID: \"f237d44f-f0d7-4d96-82b1-6a75f8c22704\") " pod="openshift-ovn-kubernetes/ovnkube-node-cjlcq"
Jul 28 10:07:48.080452 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.080369    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f237d44f-f0d7-4d96-82b1-6a75f8c22704-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-cjlcq\" (UID: \"f237d44f-f0d7-4d96-82b1-6a75f8c22704\") " pod="openshift-ovn-kubernetes/ovnkube-node-cjlcq"
Jul 28 10:07:48.080543 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.080520    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-rcwzq\""
Jul 28 10:07:48.080715 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.080530    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/20a3d596-59ed-46ce-b8f2-a9be7dc51b41-etc-sysctl-conf\") pod \"tuned-lcrsz\" (UID: \"20a3d596-59ed-46ce-b8f2-a9be7dc51b41\") " pod="openshift-cluster-node-tuning-operator/tuned-lcrsz"
Jul 28 10:07:48.080781 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.080743    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1d9e06b8-3c8a-41bd-8368-435c66ba1a52-system-cni-dir\") pod \"multus-gtxhl\" (UID: \"1d9e06b8-3c8a-41bd-8368-435c66ba1a52\") " pod="openshift-multus/multus-gtxhl"
Jul 28 10:07:48.080781 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.080770    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1d9e06b8-3c8a-41bd-8368-435c66ba1a52-cni-binary-copy\") pod \"multus-gtxhl\" (UID: \"1d9e06b8-3c8a-41bd-8368-435c66ba1a52\") " pod="openshift-multus/multus-gtxhl"
Jul 28 10:07:48.080892 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.080809    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1d9e06b8-3c8a-41bd-8368-435c66ba1a52-multus-socket-dir-parent\") pod \"multus-gtxhl\" (UID: \"1d9e06b8-3c8a-41bd-8368-435c66ba1a52\") " pod="openshift-multus/multus-gtxhl"
Jul 28 10:07:48.080892 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.080841    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1d9e06b8-3c8a-41bd-8368-435c66ba1a52-etc-kubernetes\") pod \"multus-gtxhl\" (UID: \"1d9e06b8-3c8a-41bd-8368-435c66ba1a52\") " pod="openshift-multus/multus-gtxhl"
Jul 28 10:07:48.080892 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.080881    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f237d44f-f0d7-4d96-82b1-6a75f8c22704-systemd-units\") pod \"ovnkube-node-cjlcq\" (UID: \"f237d44f-f0d7-4d96-82b1-6a75f8c22704\") " pod="openshift-ovn-kubernetes/ovnkube-node-cjlcq"
Jul 28 10:07:48.081053 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.080936    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1fb2411c-4da6-44e6-882e-4f36fb1a26ea-original-pull-secret\") pod \"global-pull-secret-syncer-bffsx\" (UID: \"1fb2411c-4da6-44e6-882e-4f36fb1a26ea\") " pod="kube-system/global-pull-secret-syncer-bffsx"
Jul 28 10:07:48.081053 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.080985    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f237d44f-f0d7-4d96-82b1-6a75f8c22704-ovn-node-metrics-cert\") pod \"ovnkube-node-cjlcq\" (UID: \"f237d44f-f0d7-4d96-82b1-6a75f8c22704\") " pod="openshift-ovn-kubernetes/ovnkube-node-cjlcq"
Jul 28 10:07:48.081053 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.081021    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/20a3d596-59ed-46ce-b8f2-a9be7dc51b41-etc-sysctl-d\") pod \"tuned-lcrsz\" (UID: \"20a3d596-59ed-46ce-b8f2-a9be7dc51b41\") " pod="openshift-cluster-node-tuning-operator/tuned-lcrsz"
Jul 28 10:07:48.081206 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.081052    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/20a3d596-59ed-46ce-b8f2-a9be7dc51b41-sys\") pod \"tuned-lcrsz\" (UID: \"20a3d596-59ed-46ce-b8f2-a9be7dc51b41\") " pod="openshift-cluster-node-tuning-operator/tuned-lcrsz"
Jul 28 10:07:48.081206 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.081086    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1d9e06b8-3c8a-41bd-8368-435c66ba1a52-multus-daemon-config\") pod \"multus-gtxhl\" (UID: \"1d9e06b8-3c8a-41bd-8368-435c66ba1a52\") " pod="openshift-multus/multus-gtxhl"
Jul 28 10:07:48.081206 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.081113    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f237d44f-f0d7-4d96-82b1-6a75f8c22704-log-socket\") pod \"ovnkube-node-cjlcq\" (UID: \"f237d44f-f0d7-4d96-82b1-6a75f8c22704\") " pod="openshift-ovn-kubernetes/ovnkube-node-cjlcq"
Jul 28 10:07:48.081206 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.081148    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1d9e06b8-3c8a-41bd-8368-435c66ba1a52-host-run-multus-certs\") pod \"multus-gtxhl\" (UID: \"1d9e06b8-3c8a-41bd-8368-435c66ba1a52\") " pod="openshift-multus/multus-gtxhl"
Jul 28 10:07:48.081713 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.081687    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0e38cbfa-85ef-4a89-a8c0-5942584d4e30-etc-selinux\") pod \"aws-ebs-csi-driver-node-7sp7b\" (UID: \"0e38cbfa-85ef-4a89-a8c0-5942584d4e30\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7sp7b"
Jul 28 10:07:48.081800 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.081744    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0e38cbfa-85ef-4a89-a8c0-5942584d4e30-device-dir\") pod \"aws-ebs-csi-driver-node-7sp7b\" (UID: \"0e38cbfa-85ef-4a89-a8c0-5942584d4e30\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7sp7b"
Jul 28 10:07:48.081800 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.081783    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1fb2411c-4da6-44e6-882e-4f36fb1a26ea-dbus\") pod \"global-pull-secret-syncer-bffsx\" (UID: \"1fb2411c-4da6-44e6-882e-4f36fb1a26ea\") " pod="kube-system/global-pull-secret-syncer-bffsx"
Jul 28 10:07:48.081933 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.081837    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f237d44f-f0d7-4d96-82b1-6a75f8c22704-host-run-netns\") pod \"ovnkube-node-cjlcq\" (UID: \"f237d44f-f0d7-4d96-82b1-6a75f8c22704\") " pod="openshift-ovn-kubernetes/ovnkube-node-cjlcq"
Jul 28 10:07:48.081933 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.081868    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f237d44f-f0d7-4d96-82b1-6a75f8c22704-etc-openvswitch\") pod \"ovnkube-node-cjlcq\" (UID: \"f237d44f-f0d7-4d96-82b1-6a75f8c22704\") " pod="openshift-ovn-kubernetes/ovnkube-node-cjlcq"
Jul 28 10:07:48.082036 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.081933    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/20a3d596-59ed-46ce-b8f2-a9be7dc51b41-tmp\") pod \"tuned-lcrsz\" (UID: \"20a3d596-59ed-46ce-b8f2-a9be7dc51b41\") " pod="openshift-cluster-node-tuning-operator/tuned-lcrsz"
Jul 28 10:07:48.082036 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.082000    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t25kd\" (UniqueName: \"kubernetes.io/projected/20a3d596-59ed-46ce-b8f2-a9be7dc51b41-kube-api-access-t25kd\") pod \"tuned-lcrsz\" (UID: \"20a3d596-59ed-46ce-b8f2-a9be7dc51b41\") " pod="openshift-cluster-node-tuning-operator/tuned-lcrsz"
Jul 28 10:07:48.082140 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.082062    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1d9e06b8-3c8a-41bd-8368-435c66ba1a52-host-var-lib-cni-bin\") pod \"multus-gtxhl\" (UID: \"1d9e06b8-3c8a-41bd-8368-435c66ba1a52\") " pod="openshift-multus/multus-gtxhl"
Jul 28 10:07:48.082140 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.082102    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f237d44f-f0d7-4d96-82b1-6a75f8c22704-node-log\") pod \"ovnkube-node-cjlcq\" (UID: \"f237d44f-f0d7-4d96-82b1-6a75f8c22704\") " pod="openshift-ovn-kubernetes/ovnkube-node-cjlcq"
Jul 28 10:07:48.082243 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.082148    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0e38cbfa-85ef-4a89-a8c0-5942584d4e30-sys-fs\") pod \"aws-ebs-csi-driver-node-7sp7b\" (UID: \"0e38cbfa-85ef-4a89-a8c0-5942584d4e30\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7sp7b"
Jul 28 10:07:48.082243 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.082205    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjlgp\" (UniqueName: \"kubernetes.io/projected/1d9e06b8-3c8a-41bd-8368-435c66ba1a52-kube-api-access-kjlgp\") pod \"multus-gtxhl\" (UID: \"1d9e06b8-3c8a-41bd-8368-435c66ba1a52\") " pod="openshift-multus/multus-gtxhl"
Jul 28 10:07:48.082338 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.082268    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f237d44f-f0d7-4d96-82b1-6a75f8c22704-run-systemd\") pod \"ovnkube-node-cjlcq\" (UID: \"f237d44f-f0d7-4d96-82b1-6a75f8c22704\") " pod="openshift-ovn-kubernetes/ovnkube-node-cjlcq"
Jul 28 10:07:48.082338 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.082316    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f237d44f-f0d7-4d96-82b1-6a75f8c22704-ovnkube-config\") pod \"ovnkube-node-cjlcq\" (UID: \"f237d44f-f0d7-4d96-82b1-6a75f8c22704\") " pod="openshift-ovn-kubernetes/ovnkube-node-cjlcq"
Jul 28 10:07:48.082430 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.082353    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0e38cbfa-85ef-4a89-a8c0-5942584d4e30-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7sp7b\" (UID: \"0e38cbfa-85ef-4a89-a8c0-5942584d4e30\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7sp7b"
Jul 28 10:07:48.082430 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.082387    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdhmh\" (UniqueName: \"kubernetes.io/projected/0e38cbfa-85ef-4a89-a8c0-5942584d4e30-kube-api-access-zdhmh\") pod \"aws-ebs-csi-driver-node-7sp7b\" (UID: \"0e38cbfa-85ef-4a89-a8c0-5942584d4e30\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7sp7b"
Jul 28 10:07:48.082536 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.082447    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/aff18ade-5ca4-40b1-bee1-a593dc42d38c-agent-certs\") pod \"konnectivity-agent-2w2gl\" (UID: \"aff18ade-5ca4-40b1-bee1-a593dc42d38c\") " pod="kube-system/konnectivity-agent-2w2gl"
Jul 28 10:07:48.082536 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.082501    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzjnq\" (UniqueName: \"kubernetes.io/projected/ba296f38-5de8-4d11-b789-7a727a01b9cc-kube-api-access-dzjnq\") pod \"node-resolver-2rf9h\" (UID: \"ba296f38-5de8-4d11-b789-7a727a01b9cc\") " pod="openshift-dns/node-resolver-2rf9h"
Jul 28 10:07:48.082638 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.082554    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1d9e06b8-3c8a-41bd-8368-435c66ba1a52-host-run-k8s-cni-cncf-io\") pod \"multus-gtxhl\" (UID: \"1d9e06b8-3c8a-41bd-8368-435c66ba1a52\") " pod="openshift-multus/multus-gtxhl"
Jul 28 10:07:48.082638 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.082604    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1d9e06b8-3c8a-41bd-8368-435c66ba1a52-cnibin\") pod \"multus-gtxhl\" (UID: \"1d9e06b8-3c8a-41bd-8368-435c66ba1a52\") " pod="openshift-multus/multus-gtxhl"
Jul 28 10:07:48.082746 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.082660    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f237d44f-f0d7-4d96-82b1-6a75f8c22704-run-ovn\") pod \"ovnkube-node-cjlcq\" (UID: \"f237d44f-f0d7-4d96-82b1-6a75f8c22704\") " pod="openshift-ovn-kubernetes/ovnkube-node-cjlcq"
Jul 28 10:07:48.082746 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.082691    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1d9e06b8-3c8a-41bd-8368-435c66ba1a52-multus-cni-dir\") pod \"multus-gtxhl\" (UID: \"1d9e06b8-3c8a-41bd-8368-435c66ba1a52\") " pod="openshift-multus/multus-gtxhl"
Jul 28 10:07:48.082746 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.082720    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1d9e06b8-3c8a-41bd-8368-435c66ba1a52-hostroot\") pod \"multus-gtxhl\" (UID: \"1d9e06b8-3c8a-41bd-8368-435c66ba1a52\") " pod="openshift-multus/multus-gtxhl"
Jul 28 10:07:48.082918 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.082749    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f237d44f-f0d7-4d96-82b1-6a75f8c22704-host-kubelet\") pod \"ovnkube-node-cjlcq\" (UID: \"f237d44f-f0d7-4d96-82b1-6a75f8c22704\") " pod="openshift-ovn-kubernetes/ovnkube-node-cjlcq"
Jul 28 10:07:48.082918 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.082801    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f237d44f-f0d7-4d96-82b1-6a75f8c22704-host-slash\") pod \"ovnkube-node-cjlcq\" (UID: \"f237d44f-f0d7-4d96-82b1-6a75f8c22704\") " pod="openshift-ovn-kubernetes/ovnkube-node-cjlcq"
Jul 28 10:07:48.082918 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.082827    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5b4tp\" (UniqueName: \"kubernetes.io/projected/f237d44f-f0d7-4d96-82b1-6a75f8c22704-kube-api-access-5b4tp\") pod \"ovnkube-node-cjlcq\" (UID: \"f237d44f-f0d7-4d96-82b1-6a75f8c22704\") " pod="openshift-ovn-kubernetes/ovnkube-node-cjlcq"
Jul 28 10:07:48.082918 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.082870    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/20a3d596-59ed-46ce-b8f2-a9be7dc51b41-etc-tuned\") pod \"tuned-lcrsz\" (UID: \"20a3d596-59ed-46ce-b8f2-a9be7dc51b41\") " pod="openshift-cluster-node-tuning-operator/tuned-lcrsz"
Jul 28 10:07:48.083105 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.082915    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ba296f38-5de8-4d11-b789-7a727a01b9cc-hosts-file\") pod \"node-resolver-2rf9h\" (UID: \"ba296f38-5de8-4d11-b789-7a727a01b9cc\") " pod="openshift-dns/node-resolver-2rf9h"
Jul 28 10:07:48.083105 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.083072    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1d9e06b8-3c8a-41bd-8368-435c66ba1a52-host-var-lib-kubelet\") pod \"multus-gtxhl\" (UID: \"1d9e06b8-3c8a-41bd-8368-435c66ba1a52\") " pod="openshift-multus/multus-gtxhl"
Jul 28 10:07:48.083951 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.083928    2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 28 10:07:48.106993 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.106967    2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-7wsfd"
Jul 28 10:07:48.117261 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.117231    2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-7wsfd"
Jul 28 10:07:48.171541 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.171513    2579 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 28 10:07:48.183379 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.183349    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1fb2411c-4da6-44e6-882e-4f36fb1a26ea-dbus\") pod \"global-pull-secret-syncer-bffsx\" (UID: \"1fb2411c-4da6-44e6-882e-4f36fb1a26ea\") " pod="kube-system/global-pull-secret-syncer-bffsx"
Jul 28 10:07:48.183573 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.183388    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f237d44f-f0d7-4d96-82b1-6a75f8c22704-host-run-netns\") pod \"ovnkube-node-cjlcq\" (UID: \"f237d44f-f0d7-4d96-82b1-6a75f8c22704\") " pod="openshift-ovn-kubernetes/ovnkube-node-cjlcq"
Jul 28 10:07:48.183573 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.183415    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f237d44f-f0d7-4d96-82b1-6a75f8c22704-etc-openvswitch\") pod \"ovnkube-node-cjlcq\" (UID: \"f237d44f-f0d7-4d96-82b1-6a75f8c22704\") " pod="openshift-ovn-kubernetes/ovnkube-node-cjlcq"
Jul 28 10:07:48.183573 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.183438    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/20a3d596-59ed-46ce-b8f2-a9be7dc51b41-tmp\") pod \"tuned-lcrsz\" (UID: \"20a3d596-59ed-46ce-b8f2-a9be7dc51b41\") " pod="openshift-cluster-node-tuning-operator/tuned-lcrsz"
Jul 28 10:07:48.183573 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.183463    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t25kd\" (UniqueName: \"kubernetes.io/projected/20a3d596-59ed-46ce-b8f2-a9be7dc51b41-kube-api-access-t25kd\") pod \"tuned-lcrsz\" (UID: \"20a3d596-59ed-46ce-b8f2-a9be7dc51b41\") " pod="openshift-cluster-node-tuning-operator/tuned-lcrsz"
Jul 28 10:07:48.183573 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.183486    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f237d44f-f0d7-4d96-82b1-6a75f8c22704-host-run-netns\") pod \"ovnkube-node-cjlcq\" (UID: \"f237d44f-f0d7-4d96-82b1-6a75f8c22704\") " pod="openshift-ovn-kubernetes/ovnkube-node-cjlcq"
Jul 28 10:07:48.183573 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.183518    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f237d44f-f0d7-4d96-82b1-6a75f8c22704-etc-openvswitch\") pod \"ovnkube-node-cjlcq\" (UID: \"f237d44f-f0d7-4d96-82b1-6a75f8c22704\") " pod="openshift-ovn-kubernetes/ovnkube-node-cjlcq"
Jul 28 10:07:48.183573 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.183489    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1d9e06b8-3c8a-41bd-8368-435c66ba1a52-host-var-lib-cni-bin\") pod \"multus-gtxhl\" (UID: \"1d9e06b8-3c8a-41bd-8368-435c66ba1a52\") " pod="openshift-multus/multus-gtxhl"
Jul 28 10:07:48.183573 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.183536    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1d9e06b8-3c8a-41bd-8368-435c66ba1a52-host-var-lib-cni-bin\") pod \"multus-gtxhl\" (UID: \"1d9e06b8-3c8a-41bd-8368-435c66ba1a52\") " pod="openshift-multus/multus-gtxhl"
Jul 28 10:07:48.183995 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.183600    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f237d44f-f0d7-4d96-82b1-6a75f8c22704-node-log\") pod \"ovnkube-node-cjlcq\" (UID: \"f237d44f-f0d7-4d96-82b1-6a75f8c22704\") " pod="openshift-ovn-kubernetes/ovnkube-node-cjlcq"
Jul 28 10:07:48.183995 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.183632    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0e38cbfa-85ef-4a89-a8c0-5942584d4e30-sys-fs\") pod \"aws-ebs-csi-driver-node-7sp7b\" (UID: \"0e38cbfa-85ef-4a89-a8c0-5942584d4e30\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7sp7b"
Jul 28 10:07:48.183995 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.183661    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjlgp\" (UniqueName: \"kubernetes.io/projected/1d9e06b8-3c8a-41bd-8368-435c66ba1a52-kube-api-access-kjlgp\") pod \"multus-gtxhl\" (UID: \"1d9e06b8-3c8a-41bd-8368-435c66ba1a52\") " pod="openshift-multus/multus-gtxhl"
Jul 28 10:07:48.183995 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.183682    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1fb2411c-4da6-44e6-882e-4f36fb1a26ea-dbus\") pod \"global-pull-secret-syncer-bffsx\" (UID: \"1fb2411c-4da6-44e6-882e-4f36fb1a26ea\") " pod="kube-system/global-pull-secret-syncer-bffsx"
Jul 28 10:07:48.183995 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.183693    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4f0d1f6f-a98f-4c02-a856-2465e90ce76e-cni-binary-copy\") pod \"multus-additional-cni-plugins-fftp9\" (UID: \"4f0d1f6f-a98f-4c02-a856-2465e90ce76e\") " pod="openshift-multus/multus-additional-cni-plugins-fftp9"
Jul 28 10:07:48.183995 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.183707    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f237d44f-f0d7-4d96-82b1-6a75f8c22704-node-log\") pod \"ovnkube-node-cjlcq\" (UID: \"f237d44f-f0d7-4d96-82b1-6a75f8c22704\") " pod="openshift-ovn-kubernetes/ovnkube-node-cjlcq"
Jul 28 10:07:48.183995 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.183735    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f237d44f-f0d7-4d96-82b1-6a75f8c22704-run-systemd\") pod \"ovnkube-node-cjlcq\" (UID: \"f237d44f-f0d7-4d96-82b1-6a75f8c22704\") " pod="openshift-ovn-kubernetes/ovnkube-node-cjlcq"
Jul 28 10:07:48.183995 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.183767    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f237d44f-f0d7-4d96-82b1-6a75f8c22704-ovnkube-config\") pod \"ovnkube-node-cjlcq\" (UID: \"f237d44f-f0d7-4d96-82b1-6a75f8c22704\") " pod="openshift-ovn-kubernetes/ovnkube-node-cjlcq"
Jul 28 10:07:48.183995 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.183807    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0e38cbfa-85ef-4a89-a8c0-5942584d4e30-sys-fs\") pod \"aws-ebs-csi-driver-node-7sp7b\" (UID: \"0e38cbfa-85ef-4a89-a8c0-5942584d4e30\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7sp7b"
Jul 28 10:07:48.183995 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.183831    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f237d44f-f0d7-4d96-82b1-6a75f8c22704-run-systemd\") pod \"ovnkube-node-cjlcq\" (UID: \"f237d44f-f0d7-4d96-82b1-6a75f8c22704\") " pod="openshift-ovn-kubernetes/ovnkube-node-cjlcq"
Jul 28 10:07:48.183995 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.183838    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0e38cbfa-85ef-4a89-a8c0-5942584d4e30-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7sp7b\" (UID: \"0e38cbfa-85ef-4a89-a8c0-5942584d4e30\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7sp7b"
Jul 28 10:07:48.183995 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.183868    2579 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 28 10:07:48.183995 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.183894    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0e38cbfa-85ef-4a89-a8c0-5942584d4e30-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7sp7b\" (UID: \"0e38cbfa-85ef-4a89-a8c0-5942584d4e30\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7sp7b"
Jul 28 10:07:48.183995 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.183888    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdhmh\" (UniqueName: \"kubernetes.io/projected/0e38cbfa-85ef-4a89-a8c0-5942584d4e30-kube-api-access-zdhmh\") pod \"aws-ebs-csi-driver-node-7sp7b\" (UID: \"0e38cbfa-85ef-4a89-a8c0-5942584d4e30\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7sp7b"
Jul 28 10:07:48.183995 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.183941    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/aff18ade-5ca4-40b1-bee1-a593dc42d38c-agent-certs\") pod \"konnectivity-agent-2w2gl\" (UID: \"aff18ade-5ca4-40b1-bee1-a593dc42d38c\") " pod="kube-system/konnectivity-agent-2w2gl"
Jul 28 10:07:48.183995 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.183971    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzjnq\" (UniqueName: \"kubernetes.io/projected/ba296f38-5de8-4d11-b789-7a727a01b9cc-kube-api-access-dzjnq\") pod \"node-resolver-2rf9h\" (UID: \"ba296f38-5de8-4d11-b789-7a727a01b9cc\") " pod="openshift-dns/node-resolver-2rf9h"
Jul 28 10:07:48.183995 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.183997    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1d9e06b8-3c8a-41bd-8368-435c66ba1a52-host-run-k8s-cni-cncf-io\") pod \"multus-gtxhl\" (UID: \"1d9e06b8-3c8a-41bd-8368-435c66ba1a52\") " pod="openshift-multus/multus-gtxhl"
Jul 28 10:07:48.184889 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.184027    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b00fc78c-64e8-4700-ba67-06a656c2d399-iptables-alerter-script\") pod \"iptables-alerter-whcvk\" (UID: \"b00fc78c-64e8-4700-ba67-06a656c2d399\") " pod="openshift-network-operator/iptables-alerter-whcvk"
Jul 28 10:07:48.184889 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.184057    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1d9e06b8-3c8a-41bd-8368-435c66ba1a52-cnibin\") pod \"multus-gtxhl\" (UID: \"1d9e06b8-3c8a-41bd-8368-435c66ba1a52\") " pod="openshift-multus/multus-gtxhl"
Jul 28 10:07:48.184889 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.184096    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f237d44f-f0d7-4d96-82b1-6a75f8c22704-run-ovn\") pod \"ovnkube-node-cjlcq\" (UID: \"f237d44f-f0d7-4d96-82b1-6a75f8c22704\") " pod="openshift-ovn-kubernetes/ovnkube-node-cjlcq"
Jul 28 10:07:48.184889 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.184155    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1d9e06b8-3c8a-41bd-8368-435c66ba1a52-multus-cni-dir\") pod \"multus-gtxhl\" (UID: \"1d9e06b8-3c8a-41bd-8368-435c66ba1a52\") " pod="openshift-multus/multus-gtxhl"
Jul 28 10:07:48.184889 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.184175    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1d9e06b8-3c8a-41bd-8368-435c66ba1a52-host-run-k8s-cni-cncf-io\") pod \"multus-gtxhl\" (UID: \"1d9e06b8-3c8a-41bd-8368-435c66ba1a52\") " pod="openshift-multus/multus-gtxhl"
Jul 28 10:07:48.184889 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.184182    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1d9e06b8-3c8a-41bd-8368-435c66ba1a52-hostroot\") pod \"multus-gtxhl\" (UID: \"1d9e06b8-3c8a-41bd-8368-435c66ba1a52\") " pod="openshift-multus/multus-gtxhl"
Jul 28 10:07:48.184889 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.184227    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1d9e06b8-3c8a-41bd-8368-435c66ba1a52-cnibin\") pod \"multus-gtxhl\" (UID: \"1d9e06b8-3c8a-41bd-8368-435c66ba1a52\") " pod="openshift-multus/multus-gtxhl"
Jul 28 10:07:48.184889 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.184233    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhz67\" (UniqueName: \"kubernetes.io/projected/b00fc78c-64e8-4700-ba67-06a656c2d399-kube-api-access-bhz67\") pod \"iptables-alerter-whcvk\" (UID: \"b00fc78c-64e8-4700-ba67-06a656c2d399\") " pod="openshift-network-operator/iptables-alerter-whcvk"
Jul 28 10:07:48.184889 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.184227    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f237d44f-f0d7-4d96-82b1-6a75f8c22704-run-ovn\") pod \"ovnkube-node-cjlcq\" (UID: \"f237d44f-f0d7-4d96-82b1-6a75f8c22704\") " pod="openshift-ovn-kubernetes/ovnkube-node-cjlcq"
Jul 28 10:07:48.184889 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.184263    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f237d44f-f0d7-4d96-82b1-6a75f8c22704-host-kubelet\") pod \"ovnkube-node-cjlcq\" (UID: \"f237d44f-f0d7-4d96-82b1-6a75f8c22704\") " pod="openshift-ovn-kubernetes/ovnkube-node-cjlcq"
Jul 28 10:07:48.184889 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.184282    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1d9e06b8-3c8a-41bd-8368-435c66ba1a52-multus-cni-dir\") pod \"multus-gtxhl\" (UID: \"1d9e06b8-3c8a-41bd-8368-435c66ba1a52\") " pod="openshift-multus/multus-gtxhl"
Jul 28 10:07:48.184889 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.184298    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1d9e06b8-3c8a-41bd-8368-435c66ba1a52-hostroot\") pod \"multus-gtxhl\" (UID: \"1d9e06b8-3c8a-41bd-8368-435c66ba1a52\") " pod="openshift-multus/multus-gtxhl"
Jul 28 10:07:48.184889 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.184343    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f237d44f-f0d7-4d96-82b1-6a75f8c22704-host-kubelet\") pod \"ovnkube-node-cjlcq\" (UID: \"f237d44f-f0d7-4d96-82b1-6a75f8c22704\") " pod="openshift-ovn-kubernetes/ovnkube-node-cjlcq"
Jul 28 10:07:48.184889 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.184342    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f237d44f-f0d7-4d96-82b1-6a75f8c22704-host-slash\") pod \"ovnkube-node-cjlcq\" (UID: \"f237d44f-f0d7-4d96-82b1-6a75f8c22704\") " pod="openshift-ovn-kubernetes/ovnkube-node-cjlcq"
Jul 28 10:07:48.184889 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.184365    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f237d44f-f0d7-4d96-82b1-6a75f8c22704-ovnkube-config\") pod \"ovnkube-node-cjlcq\" (UID: \"f237d44f-f0d7-4d96-82b1-6a75f8c22704\") " pod="openshift-ovn-kubernetes/ovnkube-node-cjlcq"
Jul 28 10:07:48.184889 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.184393    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f237d44f-f0d7-4d96-82b1-6a75f8c22704-host-slash\") pod \"ovnkube-node-cjlcq\" (UID: \"f237d44f-f0d7-4d96-82b1-6a75f8c22704\") " pod="openshift-ovn-kubernetes/ovnkube-node-cjlcq"
Jul 28 10:07:48.184889 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.184408    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5b4tp\" (UniqueName: \"kubernetes.io/projected/f237d44f-f0d7-4d96-82b1-6a75f8c22704-kube-api-access-5b4tp\") pod \"ovnkube-node-cjlcq\" (UID: \"f237d44f-f0d7-4d96-82b1-6a75f8c22704\") " pod="openshift-ovn-kubernetes/ovnkube-node-cjlcq"
Jul 28 10:07:48.184889 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.184451    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/20a3d596-59ed-46ce-b8f2-a9be7dc51b41-etc-tuned\") pod \"tuned-lcrsz\" (UID: \"20a3d596-59ed-46ce-b8f2-a9be7dc51b41\") " pod="openshift-cluster-node-tuning-operator/tuned-lcrsz"
Jul 28 10:07:48.185790 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.184476    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ba296f38-5de8-4d11-b789-7a727a01b9cc-hosts-file\") pod \"node-resolver-2rf9h\" (UID: \"ba296f38-5de8-4d11-b789-7a727a01b9cc\") " pod="openshift-dns/node-resolver-2rf9h"
Jul 28 10:07:48.185790 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.184508    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1d9e06b8-3c8a-41bd-8368-435c66ba1a52-host-var-lib-kubelet\") pod \"multus-gtxhl\" (UID: \"1d9e06b8-3c8a-41bd-8368-435c66ba1a52\") " pod="openshift-multus/multus-gtxhl"
Jul 28 10:07:48.185790 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.184544    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d67ab7ea-ff97-432b-b10a-df89bb204785-metrics-certs\") pod \"network-metrics-daemon-vtxgv\" (UID: \"d67ab7ea-ff97-432b-b10a-df89bb204785\") " pod="openshift-multus/network-metrics-daemon-vtxgv"
Jul 28 10:07:48.185790 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.184588    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/20a3d596-59ed-46ce-b8f2-a9be7dc51b41-etc-modprobe-d\") pod \"tuned-lcrsz\" (UID: \"20a3d596-59ed-46ce-b8f2-a9be7dc51b41\") " pod="openshift-cluster-node-tuning-operator/tuned-lcrsz"
Jul 28 10:07:48.185790 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.184611    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ba296f38-5de8-4d11-b789-7a727a01b9cc-hosts-file\") pod \"node-resolver-2rf9h\" (UID: \"ba296f38-5de8-4d11-b789-7a727a01b9cc\") " pod="openshift-dns/node-resolver-2rf9h"
Jul 28 10:07:48.185790 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.184612    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/20a3d596-59ed-46ce-b8f2-a9be7dc51b41-run\") pod \"tuned-lcrsz\" (UID: \"20a3d596-59ed-46ce-b8f2-a9be7dc51b41\") " pod="openshift-cluster-node-tuning-operator/tuned-lcrsz"
Jul 28 10:07:48.185790 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.184658    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/20a3d596-59ed-46ce-b8f2-a9be7dc51b41-host\") pod \"tuned-lcrsz\" (UID: \"20a3d596-59ed-46ce-b8f2-a9be7dc51b41\") " pod="openshift-cluster-node-tuning-operator/tuned-lcrsz"
Jul 28 10:07:48.185790 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.184666    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/20a3d596-59ed-46ce-b8f2-a9be7dc51b41-run\") pod \"tuned-lcrsz\" (UID: \"20a3d596-59ed-46ce-b8f2-a9be7dc51b41\") " pod="openshift-cluster-node-tuning-operator/tuned-lcrsz"
Jul 28 10:07:48.185790 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.184686    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1fb2411c-4da6-44e6-882e-4f36fb1a26ea-kubelet-config\") pod \"global-pull-secret-syncer-bffsx\" (UID: \"1fb2411c-4da6-44e6-882e-4f36fb1a26ea\") " pod="kube-system/global-pull-secret-syncer-bffsx"
Jul 28 10:07:48.185790 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.184729    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1d9e06b8-3c8a-41bd-8368-435c66ba1a52-host-var-lib-kubelet\") pod \"multus-gtxhl\" (UID: \"1d9e06b8-3c8a-41bd-8368-435c66ba1a52\") " pod="openshift-multus/multus-gtxhl"
Jul 28 10:07:48.185790 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.184736    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1d9e06b8-3c8a-41bd-8368-435c66ba1a52-host-run-netns\") pod \"multus-gtxhl\" (UID: \"1d9e06b8-3c8a-41bd-8368-435c66ba1a52\") " pod="openshift-multus/multus-gtxhl"
Jul 28 10:07:48.185790 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.184780    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4f0d1f6f-a98f-4c02-a856-2465e90ce76e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fftp9\" (UID: \"4f0d1f6f-a98f-4c02-a856-2465e90ce76e\") " pod="openshift-multus/multus-additional-cni-plugins-fftp9"
Jul 28 10:07:48.185790 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.184785    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/20a3d596-59ed-46ce-b8f2-a9be7dc51b41-host\") pod \"tuned-lcrsz\" (UID: \"20a3d596-59ed-46ce-b8f2-a9be7dc51b41\") " pod="openshift-cluster-node-tuning-operator/tuned-lcrsz"
Jul 28 10:07:48.185790 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.184808    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/beb002bc-8a51-44cf-8274-fd83f8d1d237-host\") pod \"node-ca-mkf4n\" (UID: \"beb002bc-8a51-44cf-8274-fd83f8d1d237\") " pod="openshift-image-registry/node-ca-mkf4n"
Jul 28 10:07:48.185790 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.184839    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1fb2411c-4da6-44e6-882e-4f36fb1a26ea-kubelet-config\") pod \"global-pull-secret-syncer-bffsx\" (UID: \"1fb2411c-4da6-44e6-882e-4f36fb1a26ea\") " pod="kube-system/global-pull-secret-syncer-bffsx"
Jul 28 10:07:48.185790 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.184858    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f237d44f-f0d7-4d96-82b1-6a75f8c22704-host-cni-bin\") pod \"ovnkube-node-cjlcq\" (UID: \"f237d44f-f0d7-4d96-82b1-6a75f8c22704\") " pod="openshift-ovn-kubernetes/ovnkube-node-cjlcq"
Jul 28 10:07:48.185790 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.184890    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1d9e06b8-3c8a-41bd-8368-435c66ba1a52-host-run-netns\") pod \"multus-gtxhl\" (UID: \"1d9e06b8-3c8a-41bd-8368-435c66ba1a52\") " pod="openshift-multus/multus-gtxhl"
Jul 28 10:07:48.185790 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.184867    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/20a3d596-59ed-46ce-b8f2-a9be7dc51b41-etc-modprobe-d\") pod \"tuned-lcrsz\" (UID: \"20a3d596-59ed-46ce-b8f2-a9be7dc51b41\") " pod="openshift-cluster-node-tuning-operator/tuned-lcrsz"
Jul 28 10:07:48.187075 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.184899    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/aff18ade-5ca4-40b1-bee1-a593dc42d38c-konnectivity-ca\") pod \"konnectivity-agent-2w2gl\" (UID: \"aff18ade-5ca4-40b1-bee1-a593dc42d38c\") " pod="kube-system/konnectivity-agent-2w2gl"
Jul 28 10:07:48.187075 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.184931    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f237d44f-f0d7-4d96-82b1-6a75f8c22704-host-cni-bin\") pod \"ovnkube-node-cjlcq\" (UID: \"f237d44f-f0d7-4d96-82b1-6a75f8c22704\") " pod="openshift-ovn-kubernetes/ovnkube-node-cjlcq"
Jul 28 10:07:48.187075 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.184942    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba296f38-5de8-4d11-b789-7a727a01b9cc-tmp-dir\") pod \"node-resolver-2rf9h\" (UID: \"ba296f38-5de8-4d11-b789-7a727a01b9cc\") " pod="openshift-dns/node-resolver-2rf9h"
Jul 28 10:07:48.187075 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.184973    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1d9e06b8-3c8a-41bd-8368-435c66ba1a52-os-release\") pod \"multus-gtxhl\" (UID: \"1d9e06b8-3c8a-41bd-8368-435c66ba1a52\") " pod="openshift-multus/multus-gtxhl"
Jul 28 10:07:48.187075 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.185016    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4f0d1f6f-a98f-4c02-a856-2465e90ce76e-system-cni-dir\") pod \"multus-additional-cni-plugins-fftp9\" (UID: \"4f0d1f6f-a98f-4c02-a856-2465e90ce76e\") " pod="openshift-multus/multus-additional-cni-plugins-fftp9"
Jul 28 10:07:48.187075 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.185019    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1d9e06b8-3c8a-41bd-8368-435c66ba1a52-os-release\") pod \"multus-gtxhl\" (UID: \"1d9e06b8-3c8a-41bd-8368-435c66ba1a52\") " pod="openshift-multus/multus-gtxhl"
Jul 28 10:07:48.187075 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.185047    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0e38cbfa-85ef-4a89-a8c0-5942584d4e30-socket-dir\") pod \"aws-ebs-csi-driver-node-7sp7b\" (UID: \"0e38cbfa-85ef-4a89-a8c0-5942584d4e30\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7sp7b"
Jul 28 10:07:48.187075 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.185070    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/20a3d596-59ed-46ce-b8f2-a9be7dc51b41-etc-kubernetes\") pod \"tuned-lcrsz\" (UID: \"20a3d596-59ed-46ce-b8f2-a9be7dc51b41\") " pod="openshift-cluster-node-tuning-operator/tuned-lcrsz"
Jul 28 10:07:48.187075 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.185098    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1d9e06b8-3c8a-41bd-8368-435c66ba1a52-host-var-lib-cni-multus\") pod \"multus-gtxhl\" (UID: \"1d9e06b8-3c8a-41bd-8368-435c66ba1a52\") " pod="openshift-multus/multus-gtxhl"
Jul 28 10:07:48.187075 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.185122    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f237d44f-f0d7-4d96-82b1-6a75f8c22704-host-run-ovn-kubernetes\") pod \"ovnkube-node-cjlcq\" (UID: \"f237d44f-f0d7-4d96-82b1-6a75f8c22704\") " pod="openshift-ovn-kubernetes/ovnkube-node-cjlcq"
Jul 28 10:07:48.187075 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.185144    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f237d44f-f0d7-4d96-82b1-6a75f8c22704-ovnkube-script-lib\") pod \"ovnkube-node-cjlcq\" (UID: \"f237d44f-f0d7-4d96-82b1-6a75f8c22704\") " pod="openshift-ovn-kubernetes/ovnkube-node-cjlcq"
Jul 28 10:07:48.187075 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.185164    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/20a3d596-59ed-46ce-b8f2-a9be7dc51b41-etc-sysconfig\") pod \"tuned-lcrsz\" (UID: \"20a3d596-59ed-46ce-b8f2-a9be7dc51b41\") " pod="openshift-cluster-node-tuning-operator/tuned-lcrsz"
Jul 28 10:07:48.187075 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.185185    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/20a3d596-59ed-46ce-b8f2-a9be7dc51b41-etc-systemd\") pod \"tuned-lcrsz\" (UID: \"20a3d596-59ed-46ce-b8f2-a9be7dc51b41\") " pod="openshift-cluster-node-tuning-operator/tuned-lcrsz"
Jul 28 10:07:48.187075 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.185200    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0e38cbfa-85ef-4a89-a8c0-5942584d4e30-socket-dir\") pod \"aws-ebs-csi-driver-node-7sp7b\" (UID: \"0e38cbfa-85ef-4a89-a8c0-5942584d4e30\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7sp7b"
Jul 28 10:07:48.187075 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.185213    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/20a3d596-59ed-46ce-b8f2-a9be7dc51b41-lib-modules\") pod \"tuned-lcrsz\" (UID: \"20a3d596-59ed-46ce-b8f2-a9be7dc51b41\") " pod="openshift-cluster-node-tuning-operator/tuned-lcrsz"
Jul 28 10:07:48.187075 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.185243    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvds8\" (UniqueName: \"kubernetes.io/projected/4f0d1f6f-a98f-4c02-a856-2465e90ce76e-kube-api-access-nvds8\") pod \"multus-additional-cni-plugins-fftp9\" (UID: \"4f0d1f6f-a98f-4c02-a856-2465e90ce76e\") " pod="openshift-multus/multus-additional-cni-plugins-fftp9"
Jul 28 10:07:48.187075 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.185255    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/20a3d596-59ed-46ce-b8f2-a9be7dc51b41-etc-kubernetes\") pod \"tuned-lcrsz\" (UID: \"20a3d596-59ed-46ce-b8f2-a9be7dc51b41\") " pod="openshift-cluster-node-tuning-operator/tuned-lcrsz"
Jul 28 10:07:48.188011 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.185270    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7777\" (UniqueName: \"kubernetes.io/projected/d67ab7ea-ff97-432b-b10a-df89bb204785-kube-api-access-x7777\") pod \"network-metrics-daemon-vtxgv\" (UID: \"d67ab7ea-ff97-432b-b10a-df89bb204785\") " pod="openshift-multus/network-metrics-daemon-vtxgv"
Jul 28 10:07:48.188011 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.185281    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba296f38-5de8-4d11-b789-7a727a01b9cc-tmp-dir\") pod \"node-resolver-2rf9h\" (UID: \"ba296f38-5de8-4d11-b789-7a727a01b9cc\") " pod="openshift-dns/node-resolver-2rf9h"
Jul 28 10:07:48.188011 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.185299    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f237d44f-f0d7-4d96-82b1-6a75f8c22704-env-overrides\") pod \"ovnkube-node-cjlcq\" (UID: \"f237d44f-f0d7-4d96-82b1-6a75f8c22704\") " pod="openshift-ovn-kubernetes/ovnkube-node-cjlcq"
Jul 28 10:07:48.188011 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.185311    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/20a3d596-59ed-46ce-b8f2-a9be7dc51b41-etc-sysconfig\") pod \"tuned-lcrsz\" (UID: \"20a3d596-59ed-46ce-b8f2-a9be7dc51b41\") " pod="openshift-cluster-node-tuning-operator/tuned-lcrsz"
Jul 28 10:07:48.188011 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.185330    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2s4l\" (UniqueName: \"kubernetes.io/projected/92436f2b-eb78-4e84-ab81-be7d1102f5ff-kube-api-access-h2s4l\") pod \"network-check-target-4msch\" (UID: \"92436f2b-eb78-4e84-ab81-be7d1102f5ff\") " pod="openshift-network-diagnostics/network-check-target-4msch"
Jul 28 10:07:48.188011 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.185356    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1d9e06b8-3c8a-41bd-8368-435c66ba1a52-host-var-lib-cni-multus\") pod \"multus-gtxhl\" (UID: \"1d9e06b8-3c8a-41bd-8368-435c66ba1a52\") " pod="openshift-multus/multus-gtxhl"
Jul 28 10:07:48.188011 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.185358    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1d9e06b8-3c8a-41bd-8368-435c66ba1a52-multus-conf-dir\") pod \"multus-gtxhl\" (UID: \"1d9e06b8-3c8a-41bd-8368-435c66ba1a52\") " pod="openshift-multus/multus-gtxhl"
Jul 28 10:07:48.188011 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.185403    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b00fc78c-64e8-4700-ba67-06a656c2d399-host-slash\") pod \"iptables-alerter-whcvk\" (UID: \"b00fc78c-64e8-4700-ba67-06a656c2d399\") " pod="openshift-network-operator/iptables-alerter-whcvk"
Jul 28 10:07:48.188011 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.185414    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1d9e06b8-3c8a-41bd-8368-435c66ba1a52-multus-conf-dir\") pod \"multus-gtxhl\" (UID: \"1d9e06b8-3c8a-41bd-8368-435c66ba1a52\") " pod="openshift-multus/multus-gtxhl"
Jul 28 10:07:48.188011 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.185430    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/20a3d596-59ed-46ce-b8f2-a9be7dc51b41-etc-systemd\") pod \"tuned-lcrsz\" (UID: \"20a3d596-59ed-46ce-b8f2-a9be7dc51b41\") " pod="openshift-cluster-node-tuning-operator/tuned-lcrsz"
Jul 28 10:07:48.188011 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.185450    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4f0d1f6f-a98f-4c02-a856-2465e90ce76e-cnibin\") pod \"multus-additional-cni-plugins-fftp9\" (UID: \"4f0d1f6f-a98f-4c02-a856-2465e90ce76e\") " pod="openshift-multus/multus-additional-cni-plugins-fftp9"
Jul 28 10:07:48.188011 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.185500    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4f0d1f6f-a98f-4c02-a856-2465e90ce76e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fftp9\" (UID: \"4f0d1f6f-a98f-4c02-a856-2465e90ce76e\") " pod="openshift-multus/multus-additional-cni-plugins-fftp9"
Jul 28 10:07:48.188011 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.185579    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f237d44f-f0d7-4d96-82b1-6a75f8c22704-run-openvswitch\") pod \"ovnkube-node-cjlcq\" (UID: \"f237d44f-f0d7-4d96-82b1-6a75f8c22704\") " pod="openshift-ovn-kubernetes/ovnkube-node-cjlcq"
Jul 28 10:07:48.188011 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.185457    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/aff18ade-5ca4-40b1-bee1-a593dc42d38c-konnectivity-ca\") pod \"konnectivity-agent-2w2gl\" (UID: \"aff18ade-5ca4-40b1-bee1-a593dc42d38c\") " pod="kube-system/konnectivity-agent-2w2gl"
Jul 28 10:07:48.188011 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.185646    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0e38cbfa-85ef-4a89-a8c0-5942584d4e30-registration-dir\") pod \"aws-ebs-csi-driver-node-7sp7b\" (UID: \"0e38cbfa-85ef-4a89-a8c0-5942584d4e30\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7sp7b"
Jul 28 10:07:48.188011 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.185681    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/20a3d596-59ed-46ce-b8f2-a9be7dc51b41-var-lib-kubelet\") pod \"tuned-lcrsz\" (UID: \"20a3d596-59ed-46ce-b8f2-a9be7dc51b41\") " pod="openshift-cluster-node-tuning-operator/tuned-lcrsz"
Jul 28 10:07:48.188011 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.185696    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0e38cbfa-85ef-4a89-a8c0-5942584d4e30-registration-dir\") pod \"aws-ebs-csi-driver-node-7sp7b\" (UID: \"0e38cbfa-85ef-4a89-a8c0-5942584d4e30\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7sp7b"
Jul 28 10:07:48.188915 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.185703    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f237d44f-f0d7-4d96-82b1-6a75f8c22704-var-lib-openvswitch\") pod \"ovnkube-node-cjlcq\" (UID: \"f237d44f-f0d7-4d96-82b1-6a75f8c22704\") " pod="openshift-ovn-kubernetes/ovnkube-node-cjlcq"
Jul 28 10:07:48.188915 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.185730    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f237d44f-f0d7-4d96-82b1-6a75f8c22704-host-cni-netd\") pod \"ovnkube-node-cjlcq\" (UID: \"f237d44f-f0d7-4d96-82b1-6a75f8c22704\") " pod="openshift-ovn-kubernetes/ovnkube-node-cjlcq"
Jul 28 10:07:48.188915 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.185758    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f237d44f-f0d7-4d96-82b1-6a75f8c22704-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-cjlcq\" (UID: \"f237d44f-f0d7-4d96-82b1-6a75f8c22704\") " pod="openshift-ovn-kubernetes/ovnkube-node-cjlcq"
Jul 28 10:07:48.188915 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.185592    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/20a3d596-59ed-46ce-b8f2-a9be7dc51b41-lib-modules\") pod \"tuned-lcrsz\" (UID: \"20a3d596-59ed-46ce-b8f2-a9be7dc51b41\") " pod="openshift-cluster-node-tuning-operator/tuned-lcrsz"
Jul 28 10:07:48.188915 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.185863    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f237d44f-f0d7-4d96-82b1-6a75f8c22704-var-lib-openvswitch\") pod \"ovnkube-node-cjlcq\" (UID: \"f237d44f-f0d7-4d96-82b1-6a75f8c22704\") " pod="openshift-ovn-kubernetes/ovnkube-node-cjlcq"
Jul 28 10:07:48.188915 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.185902    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f237d44f-f0d7-4d96-82b1-6a75f8c22704-host-cni-netd\") pod \"ovnkube-node-cjlcq\" (UID: \"f237d44f-f0d7-4d96-82b1-6a75f8c22704\") " pod="openshift-ovn-kubernetes/ovnkube-node-cjlcq"
Jul 28 10:07:48.188915 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.185940    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f237d44f-f0d7-4d96-82b1-6a75f8c22704-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-cjlcq\" (UID: \"f237d44f-f0d7-4d96-82b1-6a75f8c22704\") " pod="openshift-ovn-kubernetes/ovnkube-node-cjlcq"
Jul 28 10:07:48.188915 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.185944    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f237d44f-f0d7-4d96-82b1-6a75f8c22704-env-overrides\") pod \"ovnkube-node-cjlcq\" (UID: \"f237d44f-f0d7-4d96-82b1-6a75f8c22704\") " pod="openshift-ovn-kubernetes/ovnkube-node-cjlcq"
Jul 28 10:07:48.188915 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.185955    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/20a3d596-59ed-46ce-b8f2-a9be7dc51b41-etc-sysctl-conf\") pod \"tuned-lcrsz\" (UID: \"20a3d596-59ed-46ce-b8f2-a9be7dc51b41\") " pod="openshift-cluster-node-tuning-operator/tuned-lcrsz"
Jul 28 10:07:48.188915 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.185977    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f237d44f-f0d7-4d96-82b1-6a75f8c22704-host-run-ovn-kubernetes\") pod \"ovnkube-node-cjlcq\" (UID: \"f237d44f-f0d7-4d96-82b1-6a75f8c22704\") " pod="openshift-ovn-kubernetes/ovnkube-node-cjlcq"
Jul 28 10:07:48.188915 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.185810    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/20a3d596-59ed-46ce-b8f2-a9be7dc51b41-etc-sysctl-conf\") pod \"tuned-lcrsz\" (UID: \"20a3d596-59ed-46ce-b8f2-a9be7dc51b41\") " pod="openshift-cluster-node-tuning-operator/tuned-lcrsz"
Jul 28 10:07:48.188915 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.186012    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f237d44f-f0d7-4d96-82b1-6a75f8c22704-run-openvswitch\") pod \"ovnkube-node-cjlcq\" (UID: \"f237d44f-f0d7-4d96-82b1-6a75f8c22704\") " pod="openshift-ovn-kubernetes/ovnkube-node-cjlcq"
Jul 28 10:07:48.188915 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.186018    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/20a3d596-59ed-46ce-b8f2-a9be7dc51b41-var-lib-kubelet\") pod \"tuned-lcrsz\" (UID: \"20a3d596-59ed-46ce-b8f2-a9be7dc51b41\") " pod="openshift-cluster-node-tuning-operator/tuned-lcrsz"
Jul 28 10:07:48.188915 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.186014    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1d9e06b8-3c8a-41bd-8368-435c66ba1a52-system-cni-dir\") pod \"multus-gtxhl\" (UID: \"1d9e06b8-3c8a-41bd-8368-435c66ba1a52\") " pod="openshift-multus/multus-gtxhl"
Jul 28 10:07:48.188915 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.186062    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1d9e06b8-3c8a-41bd-8368-435c66ba1a52-cni-binary-copy\") pod \"multus-gtxhl\" (UID: \"1d9e06b8-3c8a-41bd-8368-435c66ba1a52\") " pod="openshift-multus/multus-gtxhl"
Jul 28 10:07:48.188915 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.186075    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1d9e06b8-3c8a-41bd-8368-435c66ba1a52-system-cni-dir\") pod \"multus-gtxhl\" (UID: \"1d9e06b8-3c8a-41bd-8368-435c66ba1a52\") " pod="openshift-multus/multus-gtxhl"
Jul 28 10:07:48.188915 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.186087    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1d9e06b8-3c8a-41bd-8368-435c66ba1a52-multus-socket-dir-parent\") pod \"multus-gtxhl\" (UID: \"1d9e06b8-3c8a-41bd-8368-435c66ba1a52\") " pod="openshift-multus/multus-gtxhl"
Jul 28 10:07:48.189849 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.186116    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1d9e06b8-3c8a-41bd-8368-435c66ba1a52-etc-kubernetes\") pod \"multus-gtxhl\" (UID: \"1d9e06b8-3c8a-41bd-8368-435c66ba1a52\") " pod="openshift-multus/multus-gtxhl"
Jul 28 10:07:48.189849 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.186143    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f237d44f-f0d7-4d96-82b1-6a75f8c22704-systemd-units\") pod \"ovnkube-node-cjlcq\" (UID: \"f237d44f-f0d7-4d96-82b1-6a75f8c22704\") " pod="openshift-ovn-kubernetes/ovnkube-node-cjlcq"
Jul 28 10:07:48.189849 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.186171    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1fb2411c-4da6-44e6-882e-4f36fb1a26ea-original-pull-secret\") pod \"global-pull-secret-syncer-bffsx\" (UID: \"1fb2411c-4da6-44e6-882e-4f36fb1a26ea\") " pod="kube-system/global-pull-secret-syncer-bffsx"
Jul 28 10:07:48.189849 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.186198    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4f0d1f6f-a98f-4c02-a856-2465e90ce76e-os-release\") pod \"multus-additional-cni-plugins-fftp9\" (UID: \"4f0d1f6f-a98f-4c02-a856-2465e90ce76e\") " pod="openshift-multus/multus-additional-cni-plugins-fftp9"
Jul 28 10:07:48.189849 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.186225    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4f0d1f6f-a98f-4c02-a856-2465e90ce76e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fftp9\" (UID: \"4f0d1f6f-a98f-4c02-a856-2465e90ce76e\") " pod="openshift-multus/multus-additional-cni-plugins-fftp9"
Jul 28 10:07:48.189849 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.186276    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f237d44f-f0d7-4d96-82b1-6a75f8c22704-ovn-node-metrics-cert\") pod \"ovnkube-node-cjlcq\" (UID: \"f237d44f-f0d7-4d96-82b1-6a75f8c22704\") " pod="openshift-ovn-kubernetes/ovnkube-node-cjlcq"
Jul 28 10:07:48.189849 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.186292    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f237d44f-f0d7-4d96-82b1-6a75f8c22704-ovnkube-script-lib\") pod \"ovnkube-node-cjlcq\" (UID: \"f237d44f-f0d7-4d96-82b1-6a75f8c22704\") " pod="openshift-ovn-kubernetes/ovnkube-node-cjlcq"
Jul 28 10:07:48.189849 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.186302    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/20a3d596-59ed-46ce-b8f2-a9be7dc51b41-etc-sysctl-d\") pod \"tuned-lcrsz\" (UID: \"20a3d596-59ed-46ce-b8f2-a9be7dc51b41\") " pod="openshift-cluster-node-tuning-operator/tuned-lcrsz"
Jul 28 10:07:48.189849 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.186327    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/20a3d596-59ed-46ce-b8f2-a9be7dc51b41-sys\") pod \"tuned-lcrsz\" (UID: \"20a3d596-59ed-46ce-b8f2-a9be7dc51b41\") " pod="openshift-cluster-node-tuning-operator/tuned-lcrsz"
Jul 28 10:07:48.189849 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.186355    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1d9e06b8-3c8a-41bd-8368-435c66ba1a52-multus-daemon-config\") pod \"multus-gtxhl\" (UID: \"1d9e06b8-3c8a-41bd-8368-435c66ba1a52\") " pod="openshift-multus/multus-gtxhl"
Jul 28 10:07:48.189849 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.186379    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/beb002bc-8a51-44cf-8274-fd83f8d1d237-serviceca\") pod \"node-ca-mkf4n\" (UID: \"beb002bc-8a51-44cf-8274-fd83f8d1d237\") " pod="openshift-image-registry/node-ca-mkf4n"
Jul 28 10:07:48.189849 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:07:48.186412    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 10:07:48.189849 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:07:48.186497    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1fb2411c-4da6-44e6-882e-4f36fb1a26ea-original-pull-secret podName:1fb2411c-4da6-44e6-882e-4f36fb1a26ea nodeName:}" failed. No retries permitted until 2026-07-28 10:07:48.686475717 +0000 UTC m=+2.949872048 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1fb2411c-4da6-44e6-882e-4f36fb1a26ea-original-pull-secret") pod "global-pull-secret-syncer-bffsx" (UID: "1fb2411c-4da6-44e6-882e-4f36fb1a26ea") : object "kube-system"/"original-pull-secret" not registered
Jul 28 10:07:48.189849 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.186553    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1d9e06b8-3c8a-41bd-8368-435c66ba1a52-multus-socket-dir-parent\") pod \"multus-gtxhl\" (UID: \"1d9e06b8-3c8a-41bd-8368-435c66ba1a52\") " pod="openshift-multus/multus-gtxhl"
Jul 28 10:07:48.189849 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.186575    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/20a3d596-59ed-46ce-b8f2-a9be7dc51b41-sys\") pod \"tuned-lcrsz\" (UID: \"20a3d596-59ed-46ce-b8f2-a9be7dc51b41\") " pod="openshift-cluster-node-tuning-operator/tuned-lcrsz"
Jul 28 10:07:48.189849 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.186633    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f237d44f-f0d7-4d96-82b1-6a75f8c22704-systemd-units\") pod \"ovnkube-node-cjlcq\" (UID: \"f237d44f-f0d7-4d96-82b1-6a75f8c22704\") " pod="openshift-ovn-kubernetes/ovnkube-node-cjlcq"
Jul 28 10:07:48.189849 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.186407    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqwlp\" (UniqueName: \"kubernetes.io/projected/beb002bc-8a51-44cf-8274-fd83f8d1d237-kube-api-access-vqwlp\") pod \"node-ca-mkf4n\" (UID: \"beb002bc-8a51-44cf-8274-fd83f8d1d237\") " pod="openshift-image-registry/node-ca-mkf4n"
Jul 28 10:07:48.190725 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.186651    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1d9e06b8-3c8a-41bd-8368-435c66ba1a52-etc-kubernetes\") pod \"multus-gtxhl\" (UID: \"1d9e06b8-3c8a-41bd-8368-435c66ba1a52\") " pod="openshift-multus/multus-gtxhl"
Jul 28 10:07:48.190725 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.186677    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f237d44f-f0d7-4d96-82b1-6a75f8c22704-log-socket\") pod \"ovnkube-node-cjlcq\" (UID: \"f237d44f-f0d7-4d96-82b1-6a75f8c22704\") " pod="openshift-ovn-kubernetes/ovnkube-node-cjlcq"
Jul 28 10:07:48.190725 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.186702    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1d9e06b8-3c8a-41bd-8368-435c66ba1a52-host-run-multus-certs\") pod \"multus-gtxhl\" (UID: \"1d9e06b8-3c8a-41bd-8368-435c66ba1a52\") " pod="openshift-multus/multus-gtxhl"
Jul 28 10:07:48.190725 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.186729    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0e38cbfa-85ef-4a89-a8c0-5942584d4e30-etc-selinux\") pod \"aws-ebs-csi-driver-node-7sp7b\" (UID: \"0e38cbfa-85ef-4a89-a8c0-5942584d4e30\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7sp7b"
Jul 28 10:07:48.190725 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.186733    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/20a3d596-59ed-46ce-b8f2-a9be7dc51b41-etc-sysctl-d\") pod \"tuned-lcrsz\" (UID: \"20a3d596-59ed-46ce-b8f2-a9be7dc51b41\") " pod="openshift-cluster-node-tuning-operator/tuned-lcrsz"
Jul 28 10:07:48.190725 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.186746    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1d9e06b8-3c8a-41bd-8368-435c66ba1a52-cni-binary-copy\") pod \"multus-gtxhl\" (UID: \"1d9e06b8-3c8a-41bd-8368-435c66ba1a52\") " pod="openshift-multus/multus-gtxhl"
Jul 28 10:07:48.190725 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.186755    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0e38cbfa-85ef-4a89-a8c0-5942584d4e30-device-dir\") pod \"aws-ebs-csi-driver-node-7sp7b\" (UID: \"0e38cbfa-85ef-4a89-a8c0-5942584d4e30\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7sp7b"
Jul 28 10:07:48.190725 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.186831    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f237d44f-f0d7-4d96-82b1-6a75f8c22704-log-socket\") pod \"ovnkube-node-cjlcq\" (UID: \"f237d44f-f0d7-4d96-82b1-6a75f8c22704\") " pod="openshift-ovn-kubernetes/ovnkube-node-cjlcq"
Jul 28 10:07:48.190725 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.186875    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0e38cbfa-85ef-4a89-a8c0-5942584d4e30-device-dir\") pod \"aws-ebs-csi-driver-node-7sp7b\" (UID: \"0e38cbfa-85ef-4a89-a8c0-5942584d4e30\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7sp7b"
Jul 28 10:07:48.190725 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.186914    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0e38cbfa-85ef-4a89-a8c0-5942584d4e30-etc-selinux\") pod \"aws-ebs-csi-driver-node-7sp7b\" (UID: \"0e38cbfa-85ef-4a89-a8c0-5942584d4e30\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7sp7b"
Jul 28 10:07:48.190725 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.186920    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1d9e06b8-3c8a-41bd-8368-435c66ba1a52-host-run-multus-certs\") pod \"multus-gtxhl\" (UID: \"1d9e06b8-3c8a-41bd-8368-435c66ba1a52\") " pod="openshift-multus/multus-gtxhl"
Jul 28 10:07:48.190725 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.187187    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1d9e06b8-3c8a-41bd-8368-435c66ba1a52-multus-daemon-config\") pod \"multus-gtxhl\" (UID: \"1d9e06b8-3c8a-41bd-8368-435c66ba1a52\") " pod="openshift-multus/multus-gtxhl"
Jul 28 10:07:48.190725 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.188280    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/20a3d596-59ed-46ce-b8f2-a9be7dc51b41-tmp\") pod \"tuned-lcrsz\" (UID: \"20a3d596-59ed-46ce-b8f2-a9be7dc51b41\") " pod="openshift-cluster-node-tuning-operator/tuned-lcrsz"
Jul 28 10:07:48.190725 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.189189    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/aff18ade-5ca4-40b1-bee1-a593dc42d38c-agent-certs\") pod \"konnectivity-agent-2w2gl\" (UID: \"aff18ade-5ca4-40b1-bee1-a593dc42d38c\") " pod="kube-system/konnectivity-agent-2w2gl"
Jul 28 10:07:48.190725 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.189706    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f237d44f-f0d7-4d96-82b1-6a75f8c22704-ovn-node-metrics-cert\") pod \"ovnkube-node-cjlcq\" (UID: \"f237d44f-f0d7-4d96-82b1-6a75f8c22704\") " pod="openshift-ovn-kubernetes/ovnkube-node-cjlcq"
Jul 28 10:07:48.190725 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.190297    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/20a3d596-59ed-46ce-b8f2-a9be7dc51b41-etc-tuned\") pod \"tuned-lcrsz\" (UID: \"20a3d596-59ed-46ce-b8f2-a9be7dc51b41\") " pod="openshift-cluster-node-tuning-operator/tuned-lcrsz"
Jul 28 10:07:48.192099 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:07:48.191958    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 10:07:48.192099 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:07:48.191984    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 10:07:48.192099 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:07:48.191998    2579 projected.go:196] Error preparing data for projected volume kube-api-access-h2s4l for pod openshift-network-diagnostics/network-check-target-4msch: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 10:07:48.192099 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:07:48.192071    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/92436f2b-eb78-4e84-ab81-be7d1102f5ff-kube-api-access-h2s4l podName:92436f2b-eb78-4e84-ab81-be7d1102f5ff nodeName:}" failed. No retries permitted until 2026-07-28 10:07:48.692053603 +0000 UTC m=+2.955449939 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-h2s4l" (UniqueName: "kubernetes.io/projected/92436f2b-eb78-4e84-ab81-be7d1102f5ff-kube-api-access-h2s4l") pod "network-check-target-4msch" (UID: "92436f2b-eb78-4e84-ab81-be7d1102f5ff") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 10:07:48.194392 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.194366    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t25kd\" (UniqueName: \"kubernetes.io/projected/20a3d596-59ed-46ce-b8f2-a9be7dc51b41-kube-api-access-t25kd\") pod \"tuned-lcrsz\" (UID: \"20a3d596-59ed-46ce-b8f2-a9be7dc51b41\") " pod="openshift-cluster-node-tuning-operator/tuned-lcrsz"
Jul 28 10:07:48.195624 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.195593    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjlgp\" (UniqueName: \"kubernetes.io/projected/1d9e06b8-3c8a-41bd-8368-435c66ba1a52-kube-api-access-kjlgp\") pod \"multus-gtxhl\" (UID: \"1d9e06b8-3c8a-41bd-8368-435c66ba1a52\") " pod="openshift-multus/multus-gtxhl"
Jul 28 10:07:48.196592 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.196568    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdhmh\" (UniqueName: \"kubernetes.io/projected/0e38cbfa-85ef-4a89-a8c0-5942584d4e30-kube-api-access-zdhmh\") pod \"aws-ebs-csi-driver-node-7sp7b\" (UID: \"0e38cbfa-85ef-4a89-a8c0-5942584d4e30\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7sp7b"
Jul 28 10:07:48.196680 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.196608    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzjnq\" (UniqueName: \"kubernetes.io/projected/ba296f38-5de8-4d11-b789-7a727a01b9cc-kube-api-access-dzjnq\") pod \"node-resolver-2rf9h\" (UID: \"ba296f38-5de8-4d11-b789-7a727a01b9cc\") " pod="openshift-dns/node-resolver-2rf9h"
Jul 28 10:07:48.197160 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.197140    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5b4tp\" (UniqueName: \"kubernetes.io/projected/f237d44f-f0d7-4d96-82b1-6a75f8c22704-kube-api-access-5b4tp\") pod \"ovnkube-node-cjlcq\" (UID: \"f237d44f-f0d7-4d96-82b1-6a75f8c22704\") " pod="openshift-ovn-kubernetes/ovnkube-node-cjlcq"
Jul 28 10:07:48.274470 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.274364    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-50.ec2.internal" event={"ID":"2efc845f7ae190bc5a73ce1b56253af6","Type":"ContainerStarted","Data":"647dfe6ba4327994f3b71d03fbe7827981a2ecd7a6383263b881cd56c326da16"}
Jul 28 10:07:48.275638 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.275597    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-50.ec2.internal" event={"ID":"5d0794185faca7077a864950fb56bc94","Type":"ContainerStarted","Data":"c088433ca2530b45316e90732f50c6d14c8c342f4b0715d5b1b31536ce343198"}
Jul 28 10:07:48.287618 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.287580    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4f0d1f6f-a98f-4c02-a856-2465e90ce76e-cni-binary-copy\") pod \"multus-additional-cni-plugins-fftp9\" (UID: \"4f0d1f6f-a98f-4c02-a856-2465e90ce76e\") " pod="openshift-multus/multus-additional-cni-plugins-fftp9"
Jul 28 10:07:48.287767 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.287639    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b00fc78c-64e8-4700-ba67-06a656c2d399-iptables-alerter-script\") pod \"iptables-alerter-whcvk\" (UID: \"b00fc78c-64e8-4700-ba67-06a656c2d399\") " pod="openshift-network-operator/iptables-alerter-whcvk"
Jul 28 10:07:48.287767 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.287670    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhz67\" (UniqueName: \"kubernetes.io/projected/b00fc78c-64e8-4700-ba67-06a656c2d399-kube-api-access-bhz67\") pod \"iptables-alerter-whcvk\" (UID: \"b00fc78c-64e8-4700-ba67-06a656c2d399\") " pod="openshift-network-operator/iptables-alerter-whcvk"
Jul 28 10:07:48.287767 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.287700    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d67ab7ea-ff97-432b-b10a-df89bb204785-metrics-certs\") pod \"network-metrics-daemon-vtxgv\" (UID: \"d67ab7ea-ff97-432b-b10a-df89bb204785\") " pod="openshift-multus/network-metrics-daemon-vtxgv"
Jul 28 10:07:48.287767 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.287733    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4f0d1f6f-a98f-4c02-a856-2465e90ce76e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fftp9\" (UID: \"4f0d1f6f-a98f-4c02-a856-2465e90ce76e\") " pod="openshift-multus/multus-additional-cni-plugins-fftp9"
Jul 28 10:07:48.287767 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.287760    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/beb002bc-8a51-44cf-8274-fd83f8d1d237-host\") pod \"node-ca-mkf4n\" (UID: \"beb002bc-8a51-44cf-8274-fd83f8d1d237\") " pod="openshift-image-registry/node-ca-mkf4n"
Jul 28 10:07:48.287998 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.287816    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/beb002bc-8a51-44cf-8274-fd83f8d1d237-host\") pod \"node-ca-mkf4n\" (UID: \"beb002bc-8a51-44cf-8274-fd83f8d1d237\") " pod="openshift-image-registry/node-ca-mkf4n"
Jul 28 10:07:48.287998 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:07:48.287861    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 10:07:48.287998 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:07:48.287950    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d67ab7ea-ff97-432b-b10a-df89bb204785-metrics-certs podName:d67ab7ea-ff97-432b-b10a-df89bb204785 nodeName:}" failed. No retries permitted until 2026-07-28 10:07:48.787927113 +0000 UTC m=+3.051323425 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d67ab7ea-ff97-432b-b10a-df89bb204785-metrics-certs") pod "network-metrics-daemon-vtxgv" (UID: "d67ab7ea-ff97-432b-b10a-df89bb204785") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 10:07:48.288157 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.288069    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4f0d1f6f-a98f-4c02-a856-2465e90ce76e-system-cni-dir\") pod \"multus-additional-cni-plugins-fftp9\" (UID: \"4f0d1f6f-a98f-4c02-a856-2465e90ce76e\") " pod="openshift-multus/multus-additional-cni-plugins-fftp9"
Jul 28 10:07:48.288157 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.288111    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvds8\" (UniqueName: \"kubernetes.io/projected/4f0d1f6f-a98f-4c02-a856-2465e90ce76e-kube-api-access-nvds8\") pod \"multus-additional-cni-plugins-fftp9\" (UID: \"4f0d1f6f-a98f-4c02-a856-2465e90ce76e\") " pod="openshift-multus/multus-additional-cni-plugins-fftp9"
Jul 28 10:07:48.288157 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.288139    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7777\" (UniqueName: \"kubernetes.io/projected/d67ab7ea-ff97-432b-b10a-df89bb204785-kube-api-access-x7777\") pod \"network-metrics-daemon-vtxgv\" (UID: \"d67ab7ea-ff97-432b-b10a-df89bb204785\") " pod="openshift-multus/network-metrics-daemon-vtxgv"
Jul 28 10:07:48.288292 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.288179    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b00fc78c-64e8-4700-ba67-06a656c2d399-host-slash\") pod \"iptables-alerter-whcvk\" (UID: \"b00fc78c-64e8-4700-ba67-06a656c2d399\") " pod="openshift-network-operator/iptables-alerter-whcvk"
Jul 28 10:07:48.288292 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.288193    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4f0d1f6f-a98f-4c02-a856-2465e90ce76e-cni-binary-copy\") pod \"multus-additional-cni-plugins-fftp9\" (UID: \"4f0d1f6f-a98f-4c02-a856-2465e90ce76e\") " pod="openshift-multus/multus-additional-cni-plugins-fftp9"
Jul 28 10:07:48.288292 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.288205    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4f0d1f6f-a98f-4c02-a856-2465e90ce76e-cnibin\") pod \"multus-additional-cni-plugins-fftp9\" (UID: \"4f0d1f6f-a98f-4c02-a856-2465e90ce76e\") " pod="openshift-multus/multus-additional-cni-plugins-fftp9"
Jul 28 10:07:48.288292 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.288232    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4f0d1f6f-a98f-4c02-a856-2465e90ce76e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fftp9\" (UID: \"4f0d1f6f-a98f-4c02-a856-2465e90ce76e\") " pod="openshift-multus/multus-additional-cni-plugins-fftp9"
Jul 28 10:07:48.288292 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.288273    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4f0d1f6f-a98f-4c02-a856-2465e90ce76e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fftp9\" (UID: \"4f0d1f6f-a98f-4c02-a856-2465e90ce76e\") " pod="openshift-multus/multus-additional-cni-plugins-fftp9"
Jul 28 10:07:48.288292 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.288286    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4f0d1f6f-a98f-4c02-a856-2465e90ce76e-os-release\") pod \"multus-additional-cni-plugins-fftp9\" (UID: \"4f0d1f6f-a98f-4c02-a856-2465e90ce76e\") " pod="openshift-multus/multus-additional-cni-plugins-fftp9"
Jul 28 10:07:48.288597 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.288313    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4f0d1f6f-a98f-4c02-a856-2465e90ce76e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fftp9\" (UID: \"4f0d1f6f-a98f-4c02-a856-2465e90ce76e\") " pod="openshift-multus/multus-additional-cni-plugins-fftp9"
Jul 28 10:07:48.288597 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.288335    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4f0d1f6f-a98f-4c02-a856-2465e90ce76e-cnibin\") pod \"multus-additional-cni-plugins-fftp9\" (UID: \"4f0d1f6f-a98f-4c02-a856-2465e90ce76e\") " pod="openshift-multus/multus-additional-cni-plugins-fftp9"
Jul 28 10:07:48.288597 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.288345    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/beb002bc-8a51-44cf-8274-fd83f8d1d237-serviceca\") pod \"node-ca-mkf4n\" (UID: \"beb002bc-8a51-44cf-8274-fd83f8d1d237\") " pod="openshift-image-registry/node-ca-mkf4n"
Jul 28 10:07:48.288597 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.288373    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqwlp\" (UniqueName: \"kubernetes.io/projected/beb002bc-8a51-44cf-8274-fd83f8d1d237-kube-api-access-vqwlp\") pod \"node-ca-mkf4n\" (UID: \"beb002bc-8a51-44cf-8274-fd83f8d1d237\") " pod="openshift-image-registry/node-ca-mkf4n"
Jul 28 10:07:48.288597 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.288380    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b00fc78c-64e8-4700-ba67-06a656c2d399-host-slash\") pod \"iptables-alerter-whcvk\" (UID: \"b00fc78c-64e8-4700-ba67-06a656c2d399\") " pod="openshift-network-operator/iptables-alerter-whcvk"
Jul 28 10:07:48.288597 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.288204    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4f0d1f6f-a98f-4c02-a856-2465e90ce76e-system-cni-dir\") pod \"multus-additional-cni-plugins-fftp9\" (UID: \"4f0d1f6f-a98f-4c02-a856-2465e90ce76e\") " pod="openshift-multus/multus-additional-cni-plugins-fftp9"
Jul 28 10:07:48.288597 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.288468    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4f0d1f6f-a98f-4c02-a856-2465e90ce76e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fftp9\" (UID: \"4f0d1f6f-a98f-4c02-a856-2465e90ce76e\") " pod="openshift-multus/multus-additional-cni-plugins-fftp9"
Jul 28 10:07:48.288597 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.288534    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4f0d1f6f-a98f-4c02-a856-2465e90ce76e-os-release\") pod \"multus-additional-cni-plugins-fftp9\" (UID: \"4f0d1f6f-a98f-4c02-a856-2465e90ce76e\") " pod="openshift-multus/multus-additional-cni-plugins-fftp9"
Jul 28 10:07:48.289028 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.289004    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4f0d1f6f-a98f-4c02-a856-2465e90ce76e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fftp9\" (UID: \"4f0d1f6f-a98f-4c02-a856-2465e90ce76e\") " pod="openshift-multus/multus-additional-cni-plugins-fftp9"
Jul 28 10:07:48.289090 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.289011    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b00fc78c-64e8-4700-ba67-06a656c2d399-iptables-alerter-script\") pod \"iptables-alerter-whcvk\" (UID: \"b00fc78c-64e8-4700-ba67-06a656c2d399\") " pod="openshift-network-operator/iptables-alerter-whcvk"
Jul 28 10:07:48.289139 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.289095    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/beb002bc-8a51-44cf-8274-fd83f8d1d237-serviceca\") pod \"node-ca-mkf4n\" (UID: \"beb002bc-8a51-44cf-8274-fd83f8d1d237\") " pod="openshift-image-registry/node-ca-mkf4n"
Jul 28 10:07:48.299190 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.299153    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhz67\" (UniqueName: \"kubernetes.io/projected/b00fc78c-64e8-4700-ba67-06a656c2d399-kube-api-access-bhz67\") pod \"iptables-alerter-whcvk\" (UID: \"b00fc78c-64e8-4700-ba67-06a656c2d399\") " pod="openshift-network-operator/iptables-alerter-whcvk"
Jul 28 10:07:48.299649 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.299606    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvds8\" (UniqueName: \"kubernetes.io/projected/4f0d1f6f-a98f-4c02-a856-2465e90ce76e-kube-api-access-nvds8\") pod \"multus-additional-cni-plugins-fftp9\" (UID: \"4f0d1f6f-a98f-4c02-a856-2465e90ce76e\") " pod="openshift-multus/multus-additional-cni-plugins-fftp9"
Jul 28 10:07:48.300260 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.300234    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7777\" (UniqueName: \"kubernetes.io/projected/d67ab7ea-ff97-432b-b10a-df89bb204785-kube-api-access-x7777\") pod \"network-metrics-daemon-vtxgv\" (UID: \"d67ab7ea-ff97-432b-b10a-df89bb204785\") " pod="openshift-multus/network-metrics-daemon-vtxgv"
Jul 28 10:07:48.300348 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.300299    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqwlp\" (UniqueName: \"kubernetes.io/projected/beb002bc-8a51-44cf-8274-fd83f8d1d237-kube-api-access-vqwlp\") pod \"node-ca-mkf4n\" (UID: \"beb002bc-8a51-44cf-8274-fd83f8d1d237\") " pod="openshift-image-registry/node-ca-mkf4n"
Jul 28 10:07:48.379181 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.379140    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-2w2gl"
Jul 28 10:07:48.388102 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.388075    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-cjlcq"
Jul 28 10:07:48.398789 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.398764    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7sp7b"
Jul 28 10:07:48.403493 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.403459    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-2rf9h"
Jul 28 10:07:48.411128 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.411105    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-gtxhl"
Jul 28 10:07:48.418821 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.418801    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-lcrsz"
Jul 28 10:07:48.427679 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.427654    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-mkf4n"
Jul 28 10:07:48.436343 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.436320    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-whcvk"
Jul 28 10:07:48.443204 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.443185    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-fftp9"
Jul 28 10:07:48.692144 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.692052    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1fb2411c-4da6-44e6-882e-4f36fb1a26ea-original-pull-secret\") pod \"global-pull-secret-syncer-bffsx\" (UID: \"1fb2411c-4da6-44e6-882e-4f36fb1a26ea\") " pod="kube-system/global-pull-secret-syncer-bffsx"
Jul 28 10:07:48.692144 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.692141    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2s4l\" (UniqueName: \"kubernetes.io/projected/92436f2b-eb78-4e84-ab81-be7d1102f5ff-kube-api-access-h2s4l\") pod \"network-check-target-4msch\" (UID: \"92436f2b-eb78-4e84-ab81-be7d1102f5ff\") " pod="openshift-network-diagnostics/network-check-target-4msch"
Jul 28 10:07:48.692377 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:07:48.692228    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 10:07:48.692377 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:07:48.692270    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 10:07:48.692377 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:07:48.692289    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 10:07:48.692377 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:07:48.692301    2579 projected.go:196] Error preparing data for projected volume kube-api-access-h2s4l for pod openshift-network-diagnostics/network-check-target-4msch: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 10:07:48.692377 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:07:48.692311    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1fb2411c-4da6-44e6-882e-4f36fb1a26ea-original-pull-secret podName:1fb2411c-4da6-44e6-882e-4f36fb1a26ea nodeName:}" failed. No retries permitted until 2026-07-28 10:07:49.692287289 +0000 UTC m=+3.955683599 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1fb2411c-4da6-44e6-882e-4f36fb1a26ea-original-pull-secret") pod "global-pull-secret-syncer-bffsx" (UID: "1fb2411c-4da6-44e6-882e-4f36fb1a26ea") : object "kube-system"/"original-pull-secret" not registered
Jul 28 10:07:48.692377 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:07:48.692367    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/92436f2b-eb78-4e84-ab81-be7d1102f5ff-kube-api-access-h2s4l podName:92436f2b-eb78-4e84-ab81-be7d1102f5ff nodeName:}" failed. No retries permitted until 2026-07-28 10:07:49.692338573 +0000 UTC m=+3.955734886 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-h2s4l" (UniqueName: "kubernetes.io/projected/92436f2b-eb78-4e84-ab81-be7d1102f5ff-kube-api-access-h2s4l") pod "network-check-target-4msch" (UID: "92436f2b-eb78-4e84-ab81-be7d1102f5ff") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 10:07:48.793367 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:48.793324    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d67ab7ea-ff97-432b-b10a-df89bb204785-metrics-certs\") pod \"network-metrics-daemon-vtxgv\" (UID: \"d67ab7ea-ff97-432b-b10a-df89bb204785\") " pod="openshift-multus/network-metrics-daemon-vtxgv"
Jul 28 10:07:48.793635 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:07:48.793464    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 10:07:48.793635 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:07:48.793569    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d67ab7ea-ff97-432b-b10a-df89bb204785-metrics-certs podName:d67ab7ea-ff97-432b-b10a-df89bb204785 nodeName:}" failed. No retries permitted until 2026-07-28 10:07:49.79353262 +0000 UTC m=+4.056928930 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d67ab7ea-ff97-432b-b10a-df89bb204785-metrics-certs") pod "network-metrics-daemon-vtxgv" (UID: "d67ab7ea-ff97-432b-b10a-df89bb204785") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 10:07:49.119996 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:49.119895    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 10:02:48 +0000 UTC" deadline="2028-01-31 21:57:15.86724488 +0000 UTC"
Jul 28 10:07:49.119996 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:49.119938    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13259h49m26.74731049s"
Jul 28 10:07:49.199387 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:49.199307    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20a3d596_59ed_46ce_b8f2_a9be7dc51b41.slice/crio-a1c91e5aad27870d79eda829b07bbdabc63d017db61c3ec2be8ef209e67948cc WatchSource:0}: Error finding container a1c91e5aad27870d79eda829b07bbdabc63d017db61c3ec2be8ef209e67948cc: Status 404 returned error can't find the container with id a1c91e5aad27870d79eda829b07bbdabc63d017db61c3ec2be8ef209e67948cc
Jul 28 10:07:49.200848 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:49.200821    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d9e06b8_3c8a_41bd_8368_435c66ba1a52.slice/crio-73319df809db53cc4dd2db3b3461b7260f4beebe0bcabc34474406ddfc3b276b WatchSource:0}: Error finding container 73319df809db53cc4dd2db3b3461b7260f4beebe0bcabc34474406ddfc3b276b: Status 404 returned error can't find the container with id 73319df809db53cc4dd2db3b3461b7260f4beebe0bcabc34474406ddfc3b276b
Jul 28 10:07:49.201903 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:49.201870    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb00fc78c_64e8_4700_ba67_06a656c2d399.slice/crio-f64f4227d13e6648cd4511e95b473bc680047e2783f30f6249998e338fad25cd WatchSource:0}: Error finding container f64f4227d13e6648cd4511e95b473bc680047e2783f30f6249998e338fad25cd: Status 404 returned error can't find the container with id f64f4227d13e6648cd4511e95b473bc680047e2783f30f6249998e338fad25cd
Jul 28 10:07:49.203209 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:49.203092    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba296f38_5de8_4d11_b789_7a727a01b9cc.slice/crio-a303ce30aa308e37e8122cd691c5620d239c22abf1bf92ecdb8b729a8ce2dd79 WatchSource:0}: Error finding container a303ce30aa308e37e8122cd691c5620d239c22abf1bf92ecdb8b729a8ce2dd79: Status 404 returned error can't find the container with id a303ce30aa308e37e8122cd691c5620d239c22abf1bf92ecdb8b729a8ce2dd79
Jul 28 10:07:49.204890 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:49.204565    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e38cbfa_85ef_4a89_a8c0_5942584d4e30.slice/crio-65bd1af2087337676525356554c4f14475f1254cd68bc3d2e6176e33331aeed4 WatchSource:0}: Error finding container 65bd1af2087337676525356554c4f14475f1254cd68bc3d2e6176e33331aeed4: Status 404 returned error can't find the container with id 65bd1af2087337676525356554c4f14475f1254cd68bc3d2e6176e33331aeed4
Jul 28 10:07:49.207749 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:07:49.207697    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f0d1f6f_a98f_4c02_a856_2465e90ce76e.slice/crio-8979dfbb797ea0e93ae5286cc028f6c0f14e2a086e854c15eccad95c883729d6 WatchSource:0}: Error finding container 8979dfbb797ea0e93ae5286cc028f6c0f14e2a086e854c15eccad95c883729d6: Status 404 returned error can't find the container with id 8979dfbb797ea0e93ae5286cc028f6c0f14e2a086e854c15eccad95c883729d6
Jul 28 10:07:49.270633 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:49.270588    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bffsx"
Jul 28 10:07:49.270733 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:07:49.270699    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bffsx" podUID="1fb2411c-4da6-44e6-882e-4f36fb1a26ea"
Jul 28 10:07:49.270787 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:49.270754    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vtxgv"
Jul 28 10:07:49.270869 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:07:49.270851    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vtxgv" podUID="d67ab7ea-ff97-432b-b10a-df89bb204785"
Jul 28 10:07:49.280269 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:49.280230    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-2w2gl" event={"ID":"aff18ade-5ca4-40b1-bee1-a593dc42d38c","Type":"ContainerStarted","Data":"614398e6625204da78428109b7f20bdecd254dddd2a632ae7cf8182ad9212e87"}
Jul 28 10:07:49.281647 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:49.281621    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cjlcq" event={"ID":"f237d44f-f0d7-4d96-82b1-6a75f8c22704","Type":"ContainerStarted","Data":"a2b9856d4c74f2853fa4626a93a9573245add7085140b5e9dc6f7b97a580084c"}
Jul 28 10:07:49.282711 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:49.282681    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-mkf4n" event={"ID":"beb002bc-8a51-44cf-8274-fd83f8d1d237","Type":"ContainerStarted","Data":"9f93778e37df2610de4ada4ff9bd722845cbaf0320bc178528f7e0c8bd443373"}
Jul 28 10:07:49.283979 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:49.283959    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7sp7b" event={"ID":"0e38cbfa-85ef-4a89-a8c0-5942584d4e30","Type":"ContainerStarted","Data":"65bd1af2087337676525356554c4f14475f1254cd68bc3d2e6176e33331aeed4"}
Jul 28 10:07:49.285244 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:49.285217    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-whcvk" event={"ID":"b00fc78c-64e8-4700-ba67-06a656c2d399","Type":"ContainerStarted","Data":"f64f4227d13e6648cd4511e95b473bc680047e2783f30f6249998e338fad25cd"}
Jul 28 10:07:49.286401 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:49.286377    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-gtxhl" event={"ID":"1d9e06b8-3c8a-41bd-8368-435c66ba1a52","Type":"ContainerStarted","Data":"73319df809db53cc4dd2db3b3461b7260f4beebe0bcabc34474406ddfc3b276b"}
Jul 28 10:07:49.287549 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:49.287528    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-2rf9h" event={"ID":"ba296f38-5de8-4d11-b789-7a727a01b9cc","Type":"ContainerStarted","Data":"a303ce30aa308e37e8122cd691c5620d239c22abf1bf92ecdb8b729a8ce2dd79"}
Jul 28 10:07:49.288749 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:49.288722    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fftp9" event={"ID":"4f0d1f6f-a98f-4c02-a856-2465e90ce76e","Type":"ContainerStarted","Data":"8979dfbb797ea0e93ae5286cc028f6c0f14e2a086e854c15eccad95c883729d6"}
Jul 28 10:07:49.290066 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:49.290033    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-lcrsz" event={"ID":"20a3d596-59ed-46ce-b8f2-a9be7dc51b41","Type":"ContainerStarted","Data":"a1c91e5aad27870d79eda829b07bbdabc63d017db61c3ec2be8ef209e67948cc"}
Jul 28 10:07:49.701687 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:49.701255    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1fb2411c-4da6-44e6-882e-4f36fb1a26ea-original-pull-secret\") pod \"global-pull-secret-syncer-bffsx\" (UID: \"1fb2411c-4da6-44e6-882e-4f36fb1a26ea\") " pod="kube-system/global-pull-secret-syncer-bffsx"
Jul 28 10:07:49.701687 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:07:49.701509    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 10:07:49.701687 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:07:49.701599    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1fb2411c-4da6-44e6-882e-4f36fb1a26ea-original-pull-secret podName:1fb2411c-4da6-44e6-882e-4f36fb1a26ea nodeName:}" failed. No retries permitted until 2026-07-28 10:07:51.701578883 +0000 UTC m=+5.964975205 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1fb2411c-4da6-44e6-882e-4f36fb1a26ea-original-pull-secret") pod "global-pull-secret-syncer-bffsx" (UID: "1fb2411c-4da6-44e6-882e-4f36fb1a26ea") : object "kube-system"/"original-pull-secret" not registered
Jul 28 10:07:49.701687 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:49.701656    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2s4l\" (UniqueName: \"kubernetes.io/projected/92436f2b-eb78-4e84-ab81-be7d1102f5ff-kube-api-access-h2s4l\") pod \"network-check-target-4msch\" (UID: \"92436f2b-eb78-4e84-ab81-be7d1102f5ff\") " pod="openshift-network-diagnostics/network-check-target-4msch"
Jul 28 10:07:49.702011 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:07:49.701818    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 10:07:49.702011 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:07:49.701834    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 10:07:49.702011 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:07:49.701860    2579 projected.go:196] Error preparing data for projected volume kube-api-access-h2s4l for pod openshift-network-diagnostics/network-check-target-4msch: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 10:07:49.702011 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:07:49.701901    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/92436f2b-eb78-4e84-ab81-be7d1102f5ff-kube-api-access-h2s4l podName:92436f2b-eb78-4e84-ab81-be7d1102f5ff nodeName:}" failed. No retries permitted until 2026-07-28 10:07:51.701887567 +0000 UTC m=+5.965283879 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-h2s4l" (UniqueName: "kubernetes.io/projected/92436f2b-eb78-4e84-ab81-be7d1102f5ff-kube-api-access-h2s4l") pod "network-check-target-4msch" (UID: "92436f2b-eb78-4e84-ab81-be7d1102f5ff") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 10:07:49.802085 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:49.802022    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d67ab7ea-ff97-432b-b10a-df89bb204785-metrics-certs\") pod \"network-metrics-daemon-vtxgv\" (UID: \"d67ab7ea-ff97-432b-b10a-df89bb204785\") " pod="openshift-multus/network-metrics-daemon-vtxgv"
Jul 28 10:07:49.802270 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:07:49.802210    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 10:07:49.802270 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:07:49.802273    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d67ab7ea-ff97-432b-b10a-df89bb204785-metrics-certs podName:d67ab7ea-ff97-432b-b10a-df89bb204785 nodeName:}" failed. No retries permitted until 2026-07-28 10:07:51.802251929 +0000 UTC m=+6.065648251 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d67ab7ea-ff97-432b-b10a-df89bb204785-metrics-certs") pod "network-metrics-daemon-vtxgv" (UID: "d67ab7ea-ff97-432b-b10a-df89bb204785") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 10:07:50.120372 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:50.120282    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 10:02:48 +0000 UTC" deadline="2028-04-17 13:43:22.85048945 +0000 UTC"
Jul 28 10:07:50.120372 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:50.120341    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15099h35m32.730152368s"
Jul 28 10:07:50.270313 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:50.270229    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4msch"
Jul 28 10:07:50.270464 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:07:50.270358    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4msch" podUID="92436f2b-eb78-4e84-ab81-be7d1102f5ff"
Jul 28 10:07:50.307639 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:50.307306    2579 generic.go:356] "Generic (PLEG): container finished" podID="5d0794185faca7077a864950fb56bc94" containerID="6ac3f4e628da8995eba12ee940c00d1bdbe1bfb3d70beb4b71c0cb1c00cb25a0" exitCode=0
Jul 28 10:07:50.307639 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:50.307432    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-50.ec2.internal" event={"ID":"5d0794185faca7077a864950fb56bc94","Type":"ContainerDied","Data":"6ac3f4e628da8995eba12ee940c00d1bdbe1bfb3d70beb4b71c0cb1c00cb25a0"}
Jul 28 10:07:50.329774 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:50.329150    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-50.ec2.internal" event={"ID":"2efc845f7ae190bc5a73ce1b56253af6","Type":"ContainerStarted","Data":"8944ef70011549b48a63f87ad1493daf7a4cfdcc1c9d5a7922e30f9fb8f88adc"}
Jul 28 10:07:50.345183 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:50.345048    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-50.ec2.internal" podStartSLOduration=3.345027408 podStartE2EDuration="3.345027408s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 10:07:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:07:50.344718825 +0000 UTC m=+4.608115155" watchObservedRunningTime="2026-07-28 10:07:50.345027408 +0000 UTC m=+4.608423738"
Jul 28 10:07:51.271191 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:51.270519    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bffsx"
Jul 28 10:07:51.271191 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:07:51.270659    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bffsx" podUID="1fb2411c-4da6-44e6-882e-4f36fb1a26ea"
Jul 28 10:07:51.271191 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:51.271037    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vtxgv"
Jul 28 10:07:51.271191 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:07:51.271136    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vtxgv" podUID="d67ab7ea-ff97-432b-b10a-df89bb204785"
Jul 28 10:07:51.343892 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:51.343378    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-50.ec2.internal" event={"ID":"5d0794185faca7077a864950fb56bc94","Type":"ContainerStarted","Data":"1586a72a1225a73c32cdc213f1a6e2309708f0e086533f45405cf7d0578916c3"}
Jul 28 10:07:51.719062 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:51.718979    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2s4l\" (UniqueName: \"kubernetes.io/projected/92436f2b-eb78-4e84-ab81-be7d1102f5ff-kube-api-access-h2s4l\") pod \"network-check-target-4msch\" (UID: \"92436f2b-eb78-4e84-ab81-be7d1102f5ff\") " pod="openshift-network-diagnostics/network-check-target-4msch"
Jul 28 10:07:51.719062 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:51.719037    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1fb2411c-4da6-44e6-882e-4f36fb1a26ea-original-pull-secret\") pod \"global-pull-secret-syncer-bffsx\" (UID: \"1fb2411c-4da6-44e6-882e-4f36fb1a26ea\") " pod="kube-system/global-pull-secret-syncer-bffsx"
Jul 28 10:07:51.719335 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:07:51.719168    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 10:07:51.719335 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:07:51.719198    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 10:07:51.719335 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:07:51.719198    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 10:07:51.719335 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:07:51.719211    2579 projected.go:196] Error preparing data for projected volume kube-api-access-h2s4l for pod openshift-network-diagnostics/network-check-target-4msch: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 10:07:51.719335 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:07:51.719261    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1fb2411c-4da6-44e6-882e-4f36fb1a26ea-original-pull-secret podName:1fb2411c-4da6-44e6-882e-4f36fb1a26ea nodeName:}" failed. No retries permitted until 2026-07-28 10:07:55.719242051 +0000 UTC m=+9.982638365 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1fb2411c-4da6-44e6-882e-4f36fb1a26ea-original-pull-secret") pod "global-pull-secret-syncer-bffsx" (UID: "1fb2411c-4da6-44e6-882e-4f36fb1a26ea") : object "kube-system"/"original-pull-secret" not registered
Jul 28 10:07:51.719335 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:07:51.719281    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/92436f2b-eb78-4e84-ab81-be7d1102f5ff-kube-api-access-h2s4l podName:92436f2b-eb78-4e84-ab81-be7d1102f5ff nodeName:}" failed. No retries permitted until 2026-07-28 10:07:55.719270308 +0000 UTC m=+9.982666618 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-h2s4l" (UniqueName: "kubernetes.io/projected/92436f2b-eb78-4e84-ab81-be7d1102f5ff-kube-api-access-h2s4l") pod "network-check-target-4msch" (UID: "92436f2b-eb78-4e84-ab81-be7d1102f5ff") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 10:07:51.820573 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:51.819843    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d67ab7ea-ff97-432b-b10a-df89bb204785-metrics-certs\") pod \"network-metrics-daemon-vtxgv\" (UID: \"d67ab7ea-ff97-432b-b10a-df89bb204785\") " pod="openshift-multus/network-metrics-daemon-vtxgv"
Jul 28 10:07:51.820573 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:07:51.820067    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 10:07:51.820573 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:07:51.820137    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d67ab7ea-ff97-432b-b10a-df89bb204785-metrics-certs podName:d67ab7ea-ff97-432b-b10a-df89bb204785 nodeName:}" failed. No retries permitted until 2026-07-28 10:07:55.820118465 +0000 UTC m=+10.083514785 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d67ab7ea-ff97-432b-b10a-df89bb204785-metrics-certs") pod "network-metrics-daemon-vtxgv" (UID: "d67ab7ea-ff97-432b-b10a-df89bb204785") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 10:07:52.275255 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:52.274758    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4msch"
Jul 28 10:07:52.275255 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:07:52.274892    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4msch" podUID="92436f2b-eb78-4e84-ab81-be7d1102f5ff"
Jul 28 10:07:53.270114 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:53.270075    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bffsx"
Jul 28 10:07:53.270328 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:07:53.270262    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bffsx" podUID="1fb2411c-4da6-44e6-882e-4f36fb1a26ea"
Jul 28 10:07:53.270404 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:53.270351    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vtxgv"
Jul 28 10:07:53.270492 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:07:53.270469    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vtxgv" podUID="d67ab7ea-ff97-432b-b10a-df89bb204785"
Jul 28 10:07:54.270513 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:54.270012    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4msch"
Jul 28 10:07:54.270513 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:07:54.270151    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4msch" podUID="92436f2b-eb78-4e84-ab81-be7d1102f5ff"
Jul 28 10:07:55.270577 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:55.270274    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bffsx"
Jul 28 10:07:55.270577 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:07:55.270408    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bffsx" podUID="1fb2411c-4da6-44e6-882e-4f36fb1a26ea"
Jul 28 10:07:55.270577 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:55.270492    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vtxgv"
Jul 28 10:07:55.271133 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:07:55.270602    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vtxgv" podUID="d67ab7ea-ff97-432b-b10a-df89bb204785"
Jul 28 10:07:55.755101 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:55.754764    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2s4l\" (UniqueName: \"kubernetes.io/projected/92436f2b-eb78-4e84-ab81-be7d1102f5ff-kube-api-access-h2s4l\") pod \"network-check-target-4msch\" (UID: \"92436f2b-eb78-4e84-ab81-be7d1102f5ff\") " pod="openshift-network-diagnostics/network-check-target-4msch"
Jul 28 10:07:55.755101 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:55.754827    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1fb2411c-4da6-44e6-882e-4f36fb1a26ea-original-pull-secret\") pod \"global-pull-secret-syncer-bffsx\" (UID: \"1fb2411c-4da6-44e6-882e-4f36fb1a26ea\") " pod="kube-system/global-pull-secret-syncer-bffsx"
Jul 28 10:07:55.755101 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:07:55.754940    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 10:07:55.755101 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:07:55.754967    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 10:07:55.755101 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:07:55.754968    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 10:07:55.755101 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:07:55.754981    2579 projected.go:196] Error preparing data for projected volume kube-api-access-h2s4l for pod openshift-network-diagnostics/network-check-target-4msch: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 10:07:55.755101 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:07:55.755041    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/92436f2b-eb78-4e84-ab81-be7d1102f5ff-kube-api-access-h2s4l podName:92436f2b-eb78-4e84-ab81-be7d1102f5ff nodeName:}" failed. No retries permitted until 2026-07-28 10:08:03.755023477 +0000 UTC m=+18.018419788 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-h2s4l" (UniqueName: "kubernetes.io/projected/92436f2b-eb78-4e84-ab81-be7d1102f5ff-kube-api-access-h2s4l") pod "network-check-target-4msch" (UID: "92436f2b-eb78-4e84-ab81-be7d1102f5ff") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 10:07:55.755101 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:07:55.755061    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1fb2411c-4da6-44e6-882e-4f36fb1a26ea-original-pull-secret podName:1fb2411c-4da6-44e6-882e-4f36fb1a26ea nodeName:}" failed. No retries permitted until 2026-07-28 10:08:03.755050479 +0000 UTC m=+18.018446789 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1fb2411c-4da6-44e6-882e-4f36fb1a26ea-original-pull-secret") pod "global-pull-secret-syncer-bffsx" (UID: "1fb2411c-4da6-44e6-882e-4f36fb1a26ea") : object "kube-system"/"original-pull-secret" not registered
Jul 28 10:07:55.855386 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:55.855348    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d67ab7ea-ff97-432b-b10a-df89bb204785-metrics-certs\") pod \"network-metrics-daemon-vtxgv\" (UID: \"d67ab7ea-ff97-432b-b10a-df89bb204785\") " pod="openshift-multus/network-metrics-daemon-vtxgv"
Jul 28 10:07:55.855633 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:07:55.855584    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 10:07:55.855695 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:07:55.855658    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d67ab7ea-ff97-432b-b10a-df89bb204785-metrics-certs podName:d67ab7ea-ff97-432b-b10a-df89bb204785 nodeName:}" failed. No retries permitted until 2026-07-28 10:08:03.855639296 +0000 UTC m=+18.119035605 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d67ab7ea-ff97-432b-b10a-df89bb204785-metrics-certs") pod "network-metrics-daemon-vtxgv" (UID: "d67ab7ea-ff97-432b-b10a-df89bb204785") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 10:07:56.271858 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:56.271825    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4msch"
Jul 28 10:07:56.272308 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:07:56.271977    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4msch" podUID="92436f2b-eb78-4e84-ab81-be7d1102f5ff"
Jul 28 10:07:57.270131 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:57.270098    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vtxgv"
Jul 28 10:07:57.270310 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:07:57.270216    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vtxgv" podUID="d67ab7ea-ff97-432b-b10a-df89bb204785"
Jul 28 10:07:57.270494 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:57.270382    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bffsx"
Jul 28 10:07:57.270494 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:07:57.270487    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bffsx" podUID="1fb2411c-4da6-44e6-882e-4f36fb1a26ea"
Jul 28 10:07:58.270094 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:58.270047    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4msch"
Jul 28 10:07:58.270619 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:07:58.270182    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4msch" podUID="92436f2b-eb78-4e84-ab81-be7d1102f5ff"
Jul 28 10:07:59.270421 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:59.270381    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vtxgv"
Jul 28 10:07:59.270941 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:07:59.270381    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bffsx"
Jul 28 10:07:59.270941 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:07:59.270513    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vtxgv" podUID="d67ab7ea-ff97-432b-b10a-df89bb204785"
Jul 28 10:07:59.270941 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:07:59.270579    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bffsx" podUID="1fb2411c-4da6-44e6-882e-4f36fb1a26ea"
Jul 28 10:08:00.270455 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:00.270367    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4msch"
Jul 28 10:08:00.270906 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:08:00.270503    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4msch" podUID="92436f2b-eb78-4e84-ab81-be7d1102f5ff"
Jul 28 10:08:01.270327 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:01.270291    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bffsx"
Jul 28 10:08:01.270327 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:01.270315    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vtxgv"
Jul 28 10:08:01.270885 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:08:01.270433    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bffsx" podUID="1fb2411c-4da6-44e6-882e-4f36fb1a26ea"
Jul 28 10:08:01.270885 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:08:01.270554    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vtxgv" podUID="d67ab7ea-ff97-432b-b10a-df89bb204785"
Jul 28 10:08:02.270965 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:02.270929    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4msch"
Jul 28 10:08:02.271458 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:08:02.271061    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4msch" podUID="92436f2b-eb78-4e84-ab81-be7d1102f5ff"
Jul 28 10:08:03.270458 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:03.270430    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vtxgv"
Jul 28 10:08:03.270734 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:03.270564    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bffsx"
Jul 28 10:08:03.270734 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:08:03.270570    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vtxgv" podUID="d67ab7ea-ff97-432b-b10a-df89bb204785"
Jul 28 10:08:03.270734 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:08:03.270642    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bffsx" podUID="1fb2411c-4da6-44e6-882e-4f36fb1a26ea"
Jul 28 10:08:03.816649 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:03.816531    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1fb2411c-4da6-44e6-882e-4f36fb1a26ea-original-pull-secret\") pod \"global-pull-secret-syncer-bffsx\" (UID: \"1fb2411c-4da6-44e6-882e-4f36fb1a26ea\") " pod="kube-system/global-pull-secret-syncer-bffsx"
Jul 28 10:08:03.817111 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:03.816651    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2s4l\" (UniqueName: \"kubernetes.io/projected/92436f2b-eb78-4e84-ab81-be7d1102f5ff-kube-api-access-h2s4l\") pod \"network-check-target-4msch\" (UID: \"92436f2b-eb78-4e84-ab81-be7d1102f5ff\") " pod="openshift-network-diagnostics/network-check-target-4msch"
Jul 28 10:08:03.817111 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:08:03.816662    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 10:08:03.817111 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:08:03.816734    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1fb2411c-4da6-44e6-882e-4f36fb1a26ea-original-pull-secret podName:1fb2411c-4da6-44e6-882e-4f36fb1a26ea nodeName:}" failed. No retries permitted until 2026-07-28 10:08:19.816716955 +0000 UTC m=+34.080113274 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1fb2411c-4da6-44e6-882e-4f36fb1a26ea-original-pull-secret") pod "global-pull-secret-syncer-bffsx" (UID: "1fb2411c-4da6-44e6-882e-4f36fb1a26ea") : object "kube-system"/"original-pull-secret" not registered
Jul 28 10:08:03.817111 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:08:03.816785    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 10:08:03.817111 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:08:03.816806    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 10:08:03.817111 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:08:03.816818    2579 projected.go:196] Error preparing data for projected volume kube-api-access-h2s4l for pod openshift-network-diagnostics/network-check-target-4msch: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 10:08:03.817111 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:08:03.816877    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/92436f2b-eb78-4e84-ab81-be7d1102f5ff-kube-api-access-h2s4l podName:92436f2b-eb78-4e84-ab81-be7d1102f5ff nodeName:}" failed. No retries permitted until 2026-07-28 10:08:19.816860746 +0000 UTC m=+34.080257054 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-h2s4l" (UniqueName: "kubernetes.io/projected/92436f2b-eb78-4e84-ab81-be7d1102f5ff-kube-api-access-h2s4l") pod "network-check-target-4msch" (UID: "92436f2b-eb78-4e84-ab81-be7d1102f5ff") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 10:08:03.917528 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:03.917485    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d67ab7ea-ff97-432b-b10a-df89bb204785-metrics-certs\") pod \"network-metrics-daemon-vtxgv\" (UID: \"d67ab7ea-ff97-432b-b10a-df89bb204785\") " pod="openshift-multus/network-metrics-daemon-vtxgv"
Jul 28 10:08:03.917706 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:08:03.917679    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 10:08:03.917773 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:08:03.917758    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d67ab7ea-ff97-432b-b10a-df89bb204785-metrics-certs podName:d67ab7ea-ff97-432b-b10a-df89bb204785 nodeName:}" failed. No retries permitted until 2026-07-28 10:08:19.917737212 +0000 UTC m=+34.181133524 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d67ab7ea-ff97-432b-b10a-df89bb204785-metrics-certs") pod "network-metrics-daemon-vtxgv" (UID: "d67ab7ea-ff97-432b-b10a-df89bb204785") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 10:08:04.270109 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:04.270020    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4msch"
Jul 28 10:08:04.270262 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:08:04.270168    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4msch" podUID="92436f2b-eb78-4e84-ab81-be7d1102f5ff"
Jul 28 10:08:05.270508 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:05.270468    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bffsx"
Jul 28 10:08:05.271012 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:05.270469    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vtxgv"
Jul 28 10:08:05.271012 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:08:05.270608    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bffsx" podUID="1fb2411c-4da6-44e6-882e-4f36fb1a26ea"
Jul 28 10:08:05.271012 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:08:05.270727    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vtxgv" podUID="d67ab7ea-ff97-432b-b10a-df89bb204785"
Jul 28 10:08:06.271806 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:06.271768    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4msch"
Jul 28 10:08:06.272246 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:08:06.271884    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4msch" podUID="92436f2b-eb78-4e84-ab81-be7d1102f5ff"
Jul 28 10:08:07.270433 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:07.270381    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bffsx"
Jul 28 10:08:07.270607 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:08:07.270528    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bffsx" podUID="1fb2411c-4da6-44e6-882e-4f36fb1a26ea"
Jul 28 10:08:07.270607 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:07.270583    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vtxgv"
Jul 28 10:08:07.270717 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:08:07.270690    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vtxgv" podUID="d67ab7ea-ff97-432b-b10a-df89bb204785"
Jul 28 10:08:08.270361 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:08.269937    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4msch"
Jul 28 10:08:08.270978 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:08:08.270477    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4msch" podUID="92436f2b-eb78-4e84-ab81-be7d1102f5ff"
Jul 28 10:08:08.378129 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:08.378087    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-2w2gl" event={"ID":"aff18ade-5ca4-40b1-bee1-a593dc42d38c","Type":"ContainerStarted","Data":"b3733ae49c4d578ac0c56265e05964c7e7184380bbc46c3cef952efb89352040"}
Jul 28 10:08:08.381746 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:08.381712    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cjlcq" event={"ID":"f237d44f-f0d7-4d96-82b1-6a75f8c22704","Type":"ContainerStarted","Data":"2437dd17018b5f07eb136a4b12845a3ddeb4947980c9e0e682567d506a63b7aa"}
Jul 28 10:08:08.381881 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:08.381782    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cjlcq" event={"ID":"f237d44f-f0d7-4d96-82b1-6a75f8c22704","Type":"ContainerStarted","Data":"8112d59f5fa1819048b0fd5a28e205eac43d8b5c9e80acc38472d152ab0eee62"}
Jul 28 10:08:08.381881 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:08.381830    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cjlcq" event={"ID":"f237d44f-f0d7-4d96-82b1-6a75f8c22704","Type":"ContainerStarted","Data":"b68c99b7699ccbc5d83aab21e65bbf8c4ef94097a4f53566ee35d6ba3a17fb00"}
Jul 28 10:08:08.381881 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:08.381867    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cjlcq" event={"ID":"f237d44f-f0d7-4d96-82b1-6a75f8c22704","Type":"ContainerStarted","Data":"3db908de0ebcd2fede22dc713e11eca136385a44d3e1789942794e5a9f956916"}
Jul 28 10:08:08.382034 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:08.381904    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cjlcq" event={"ID":"f237d44f-f0d7-4d96-82b1-6a75f8c22704","Type":"ContainerStarted","Data":"8077e2f98f97f905c02dbaba957d1dd9203a9bd992974b831c6de11d24adaa84"}
Jul 28 10:08:08.382034 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:08.381934    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cjlcq" event={"ID":"f237d44f-f0d7-4d96-82b1-6a75f8c22704","Type":"ContainerStarted","Data":"ede42a10892ab11d1c6459031ac07ff80e6ff43896f4633f5acdcf8d2661c6fa"}
Jul 28 10:08:08.383480 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:08.383455    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-mkf4n" event={"ID":"beb002bc-8a51-44cf-8274-fd83f8d1d237","Type":"ContainerStarted","Data":"8cceb8fa9675133f06d5c575c564775552706cf04ea00bf9767db719d819d198"}
Jul 28 10:08:08.385192 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:08.385169    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7sp7b" event={"ID":"0e38cbfa-85ef-4a89-a8c0-5942584d4e30","Type":"ContainerStarted","Data":"1fa209a2c441de8185ba4bdf52d06e0d45545399b5f470a50ef7ac9b36261fe5"}
Jul 28 10:08:08.386760 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:08.386728    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-gtxhl" event={"ID":"1d9e06b8-3c8a-41bd-8368-435c66ba1a52","Type":"ContainerStarted","Data":"f893e4b6a9357dd12dfd75e03fee65bc8b241882706f485274d14b34da5283bb"}
Jul 28 10:08:08.388349 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:08.388327    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-2rf9h" event={"ID":"ba296f38-5de8-4d11-b789-7a727a01b9cc","Type":"ContainerStarted","Data":"b04b24d4b04e2a9f2cfb96c33becda9e3cb86b825a47f621098886a880f7b56b"}
Jul 28 10:08:08.390032 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:08.390009    2579 generic.go:356] "Generic (PLEG): container finished" podID="4f0d1f6f-a98f-4c02-a856-2465e90ce76e" containerID="bcac5bec50c52aef29e0e071db6ab4ab8f4ef7bbdd3a56194ee1f0ffc1cc142d" exitCode=0
Jul 28 10:08:08.390122 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:08.390092    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fftp9" event={"ID":"4f0d1f6f-a98f-4c02-a856-2465e90ce76e","Type":"ContainerDied","Data":"bcac5bec50c52aef29e0e071db6ab4ab8f4ef7bbdd3a56194ee1f0ffc1cc142d"}
Jul 28 10:08:08.391677 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:08.391657    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-lcrsz" event={"ID":"20a3d596-59ed-46ce-b8f2-a9be7dc51b41","Type":"ContainerStarted","Data":"fa835c13f778e4a4e0bc47ad2ae60226fe86906fcc503640be064d76ea95163f"}
Jul 28 10:08:08.402910 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:08.402830    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-50.ec2.internal" podStartSLOduration=21.40281398 podStartE2EDuration="21.40281398s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 10:07:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:07:51.365669469 +0000 UTC m=+5.629065800" watchObservedRunningTime="2026-07-28 10:08:08.40281398 +0000 UTC m=+22.666210311"
Jul 28 10:08:08.403280 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:08.403245    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-2w2gl" podStartSLOduration=8.083202158 podStartE2EDuration="22.403234814s" totalImagesPullingTime="14.320032656s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:07:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:08:08.401640361 +0000 UTC m=+22.665036688" watchObservedRunningTime="2026-07-28 10:08:08.403234814 +0000 UTC m=+22.666631144"
Jul 28 10:08:08.434016 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:08.433967    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-gtxhl" podStartSLOduration=3.939943484 podStartE2EDuration="22.433949956s" totalImagesPullingTime="18.494006472s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 10:07:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:08:08.433287737 +0000 UTC m=+22.696684067" watchObservedRunningTime="2026-07-28 10:08:08.433949956 +0000 UTC m=+22.697346285"
Jul 28 10:08:08.464885 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:08.464820    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-lcrsz" podStartSLOduration=4.022481701 podStartE2EDuration="22.464800062s" totalImagesPullingTime="18.442318361s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 10:07:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:08:08.464678967 +0000 UTC m=+22.728075299" watchObservedRunningTime="2026-07-28 10:08:08.464800062 +0000 UTC m=+22.728196394"
Jul 28 10:08:08.482189 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:08.482115    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-2rf9h" podStartSLOduration=4.040833994 podStartE2EDuration="22.482096961s" totalImagesPullingTime="18.441262967s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 10:07:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:08:08.481751814 +0000 UTC m=+22.745148146" watchObservedRunningTime="2026-07-28 10:08:08.482096961 +0000 UTC m=+22.745493291"
Jul 28 10:08:08.495805 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:08.495746    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-mkf4n" podStartSLOduration=4.067873683 podStartE2EDuration="22.495727144s" totalImagesPullingTime="18.427853461s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 10:07:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:08:08.495229601 +0000 UTC m=+22.758625931" watchObservedRunningTime="2026-07-28 10:08:08.495727144 +0000 UTC m=+22.759123476"
Jul 28 10:08:08.960381 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:08.960355    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-2w2gl"
Jul 28 10:08:08.961109 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:08.961085    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-2w2gl"
Jul 28 10:08:09.014725 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:09.014699    2579 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 28 10:08:09.144569 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:09.144396    2579 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-28T10:08:09.014720324Z","UUID":"9e7e96a7-6ce9-42dd-9484-857c54f0f9f5","Handler":null,"Name":"","Endpoint":""}
Jul 28 10:08:09.145762 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:09.145741    2579 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 28 10:08:09.145871 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:09.145778    2579 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 28 10:08:09.270448 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:09.270377    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vtxgv"
Jul 28 10:08:09.270448 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:09.270407    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bffsx"
Jul 28 10:08:09.270927 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:08:09.270485    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vtxgv" podUID="d67ab7ea-ff97-432b-b10a-df89bb204785"
Jul 28 10:08:09.270927 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:08:09.270620    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bffsx" podUID="1fb2411c-4da6-44e6-882e-4f36fb1a26ea"
Jul 28 10:08:09.395179 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:09.395145    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7sp7b" event={"ID":"0e38cbfa-85ef-4a89-a8c0-5942584d4e30","Type":"ContainerStarted","Data":"11d6c5444adde48c1eaab4e90571980e241486a243e6e2d099e943ef5e174fc9"}
Jul 28 10:08:09.396436 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:09.396410    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-whcvk" event={"ID":"b00fc78c-64e8-4700-ba67-06a656c2d399","Type":"ContainerStarted","Data":"56dbf8331ed2e3d0b6337d970a338156146d1dce234857ee4feb829e183008a8"}
Jul 28 10:08:09.396963 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:09.396944    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-2w2gl"
Jul 28 10:08:09.397459 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:09.397444    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-2w2gl"
Jul 28 10:08:09.412649 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:09.412608    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-whcvk" podStartSLOduration=4.975370643 podStartE2EDuration="23.412596152s" totalImagesPullingTime="18.437225509s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 10:07:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:08:09.412341425 +0000 UTC m=+23.675737779" watchObservedRunningTime="2026-07-28 10:08:09.412596152 +0000 UTC m=+23.675992480"
Jul 28 10:08:10.270167 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:10.270129    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4msch"
Jul 28 10:08:10.270345 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:08:10.270250    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4msch" podUID="92436f2b-eb78-4e84-ab81-be7d1102f5ff"
Jul 28 10:08:10.401834 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:10.401800    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7sp7b" event={"ID":"0e38cbfa-85ef-4a89-a8c0-5942584d4e30","Type":"ContainerStarted","Data":"93ceaf300b7dc235f42ed8db69d27e5bf48b66506447a6d1e369e1d780e2a585"}
Jul 28 10:08:10.424140 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:10.424087    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7sp7b" podStartSLOduration=3.898119088 podStartE2EDuration="24.424067057s" totalImagesPullingTime="20.525947969s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 10:07:46 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:08:10.423662386 +0000 UTC m=+24.687058715" watchObservedRunningTime="2026-07-28 10:08:10.424067057 +0000 UTC m=+24.687463388"
Jul 28 10:08:11.270497 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:11.270465    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vtxgv"
Jul 28 10:08:11.270688 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:08:11.270601    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vtxgv" podUID="d67ab7ea-ff97-432b-b10a-df89bb204785"
Jul 28 10:08:11.270688 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:11.270654    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bffsx"
Jul 28 10:08:11.270786 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:08:11.270770    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bffsx" podUID="1fb2411c-4da6-44e6-882e-4f36fb1a26ea"
Jul 28 10:08:11.406841 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:11.406811    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cjlcq" event={"ID":"f237d44f-f0d7-4d96-82b1-6a75f8c22704","Type":"ContainerStarted","Data":"6eca1d56eae569fe966f0a31ce5c590812307b0f2e143fa079758f28d7d57bb5"}
Jul 28 10:08:12.270283 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:12.270066    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4msch"
Jul 28 10:08:12.270475 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:08:12.270388    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4msch" podUID="92436f2b-eb78-4e84-ab81-be7d1102f5ff"
Jul 28 10:08:13.270163 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:13.270132    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bffsx"
Jul 28 10:08:13.271338 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:08:13.270236    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bffsx" podUID="1fb2411c-4da6-44e6-882e-4f36fb1a26ea"
Jul 28 10:08:13.271338 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:13.270261    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vtxgv"
Jul 28 10:08:13.271338 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:08:13.270326    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vtxgv" podUID="d67ab7ea-ff97-432b-b10a-df89bb204785"
Jul 28 10:08:13.413549 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:13.413505    2579 generic.go:356] "Generic (PLEG): container finished" podID="4f0d1f6f-a98f-4c02-a856-2465e90ce76e" containerID="00430e538bc4421ba9c45c422fca5bc7998f291d8831026abdddb21a20295b65" exitCode=0
Jul 28 10:08:13.413746 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:13.413594    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fftp9" event={"ID":"4f0d1f6f-a98f-4c02-a856-2465e90ce76e","Type":"ContainerDied","Data":"00430e538bc4421ba9c45c422fca5bc7998f291d8831026abdddb21a20295b65"}
Jul 28 10:08:13.417299 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:13.417276    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cjlcq" event={"ID":"f237d44f-f0d7-4d96-82b1-6a75f8c22704","Type":"ContainerStarted","Data":"529f73985d045b77034f3091237b6cdd99d0607b19909a515b067c0d25e3bd09"}
Jul 28 10:08:13.417531 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:13.417516    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-cjlcq"
Jul 28 10:08:13.432437 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:13.432415    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-cjlcq"
Jul 28 10:08:13.470400 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:13.470287    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-cjlcq" podStartSLOduration=8.970470504 podStartE2EDuration="27.470272023s" totalImagesPullingTime="18.499801519s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 10:07:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:08:13.469528032 +0000 UTC m=+27.732924351" watchObservedRunningTime="2026-07-28 10:08:13.470272023 +0000 UTC m=+27.733668352"
Jul 28 10:08:14.270433 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:14.270248    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4msch"
Jul 28 10:08:14.270905 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:08:14.270524    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4msch" podUID="92436f2b-eb78-4e84-ab81-be7d1102f5ff"
Jul 28 10:08:14.419982 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:14.419912    2579 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 28 10:08:14.420432 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:14.420407    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-cjlcq"
Jul 28 10:08:14.435788 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:14.435766    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-cjlcq"
Jul 28 10:08:15.269900 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:15.269868    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bffsx"
Jul 28 10:08:15.269900 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:15.269887    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vtxgv"
Jul 28 10:08:15.270090 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:08:15.269970    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bffsx" podUID="1fb2411c-4da6-44e6-882e-4f36fb1a26ea"
Jul 28 10:08:15.270090 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:08:15.270040    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vtxgv" podUID="d67ab7ea-ff97-432b-b10a-df89bb204785"
Jul 28 10:08:15.423722 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:15.423689    2579 generic.go:356] "Generic (PLEG): container finished" podID="4f0d1f6f-a98f-4c02-a856-2465e90ce76e" containerID="72a3b4b4227bf486ef6ad69637a0fa40a2c5fac216e49444ac85d34d32ea8ac8" exitCode=0
Jul 28 10:08:15.424125 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:15.423773    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fftp9" event={"ID":"4f0d1f6f-a98f-4c02-a856-2465e90ce76e","Type":"ContainerDied","Data":"72a3b4b4227bf486ef6ad69637a0fa40a2c5fac216e49444ac85d34d32ea8ac8"}
Jul 28 10:08:15.424125 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:15.423822    2579 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 28 10:08:15.452419 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:15.452389    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-bffsx"]
Jul 28 10:08:15.452588 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:15.452526    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bffsx"
Jul 28 10:08:15.452670 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:08:15.452648    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bffsx" podUID="1fb2411c-4da6-44e6-882e-4f36fb1a26ea"
Jul 28 10:08:15.453214 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:15.453186    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-4msch"]
Jul 28 10:08:15.453329 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:15.453257    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4msch"
Jul 28 10:08:15.453393 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:08:15.453358    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4msch" podUID="92436f2b-eb78-4e84-ab81-be7d1102f5ff"
Jul 28 10:08:15.468662 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:15.468445    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-vtxgv"]
Jul 28 10:08:15.468934 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:15.468908    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vtxgv"
Jul 28 10:08:15.469114 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:08:15.469069    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vtxgv" podUID="d67ab7ea-ff97-432b-b10a-df89bb204785"
Jul 28 10:08:15.839343 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:15.839264    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-cjlcq"
Jul 28 10:08:17.108575 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:17.108412    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:17.210854 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:17.210822    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:17.270702 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:17.270619    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4msch"
Jul 28 10:08:17.270702 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:17.270657    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bffsx"
Jul 28 10:08:17.270898 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:17.270623    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vtxgv"
Jul 28 10:08:17.270898 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:08:17.270731    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4msch" podUID="92436f2b-eb78-4e84-ab81-be7d1102f5ff"
Jul 28 10:08:17.270898 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:08:17.270800    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vtxgv" podUID="d67ab7ea-ff97-432b-b10a-df89bb204785"
Jul 28 10:08:17.270898 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:08:17.270848    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bffsx" podUID="1fb2411c-4da6-44e6-882e-4f36fb1a26ea"
Jul 28 10:08:17.311886 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:17.311836    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:17.429576 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:17.429531    2579 generic.go:356] "Generic (PLEG): container finished" podID="4f0d1f6f-a98f-4c02-a856-2465e90ce76e" containerID="ecd0274006ae964a3f6fb82bf281798fff61bea39199e3761886fca52b5cb8b3" exitCode=0
Jul 28 10:08:17.429710 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:17.429628    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fftp9" event={"ID":"4f0d1f6f-a98f-4c02-a856-2465e90ce76e","Type":"ContainerDied","Data":"ecd0274006ae964a3f6fb82bf281798fff61bea39199e3761886fca52b5cb8b3"}
Jul 28 10:08:19.270912 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:19.270863    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bffsx"
Jul 28 10:08:19.271334 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:19.270871    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vtxgv"
Jul 28 10:08:19.271334 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:08:19.270984    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bffsx" podUID="1fb2411c-4da6-44e6-882e-4f36fb1a26ea"
Jul 28 10:08:19.271334 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:19.270878    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4msch"
Jul 28 10:08:19.271334 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:08:19.271090    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vtxgv" podUID="d67ab7ea-ff97-432b-b10a-df89bb204785"
Jul 28 10:08:19.271334 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:08:19.271166    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4msch" podUID="92436f2b-eb78-4e84-ab81-be7d1102f5ff"
Jul 28 10:08:19.281280 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:19.281247    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:19.377822 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:19.377787    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:19.835513 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:19.835481    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2s4l\" (UniqueName: \"kubernetes.io/projected/92436f2b-eb78-4e84-ab81-be7d1102f5ff-kube-api-access-h2s4l\") pod \"network-check-target-4msch\" (UID: \"92436f2b-eb78-4e84-ab81-be7d1102f5ff\") " pod="openshift-network-diagnostics/network-check-target-4msch"
Jul 28 10:08:19.835513 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:19.835517    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1fb2411c-4da6-44e6-882e-4f36fb1a26ea-original-pull-secret\") pod \"global-pull-secret-syncer-bffsx\" (UID: \"1fb2411c-4da6-44e6-882e-4f36fb1a26ea\") " pod="kube-system/global-pull-secret-syncer-bffsx"
Jul 28 10:08:19.835766 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:08:19.835630    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 10:08:19.835766 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:08:19.835658    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 10:08:19.835766 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:08:19.835677    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 10:08:19.835766 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:08:19.835686    2579 projected.go:196] Error preparing data for projected volume kube-api-access-h2s4l for pod openshift-network-diagnostics/network-check-target-4msch: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 10:08:19.835766 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:08:19.835695    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1fb2411c-4da6-44e6-882e-4f36fb1a26ea-original-pull-secret podName:1fb2411c-4da6-44e6-882e-4f36fb1a26ea nodeName:}" failed. No retries permitted until 2026-07-28 10:08:51.835670304 +0000 UTC m=+66.099066611 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1fb2411c-4da6-44e6-882e-4f36fb1a26ea-original-pull-secret") pod "global-pull-secret-syncer-bffsx" (UID: "1fb2411c-4da6-44e6-882e-4f36fb1a26ea") : object "kube-system"/"original-pull-secret" not registered
Jul 28 10:08:19.835766 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:08:19.835737    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/92436f2b-eb78-4e84-ab81-be7d1102f5ff-kube-api-access-h2s4l podName:92436f2b-eb78-4e84-ab81-be7d1102f5ff nodeName:}" failed. No retries permitted until 2026-07-28 10:08:51.835721575 +0000 UTC m=+66.099117889 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-h2s4l" (UniqueName: "kubernetes.io/projected/92436f2b-eb78-4e84-ab81-be7d1102f5ff-kube-api-access-h2s4l") pod "network-check-target-4msch" (UID: "92436f2b-eb78-4e84-ab81-be7d1102f5ff") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 10:08:19.936268 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:19.936191    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d67ab7ea-ff97-432b-b10a-df89bb204785-metrics-certs\") pod \"network-metrics-daemon-vtxgv\" (UID: \"d67ab7ea-ff97-432b-b10a-df89bb204785\") " pod="openshift-multus/network-metrics-daemon-vtxgv"
Jul 28 10:08:19.936469 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:08:19.936326    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 10:08:19.936469 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:08:19.936395    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d67ab7ea-ff97-432b-b10a-df89bb204785-metrics-certs podName:d67ab7ea-ff97-432b-b10a-df89bb204785 nodeName:}" failed. No retries permitted until 2026-07-28 10:08:51.936373995 +0000 UTC m=+66.199770305 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d67ab7ea-ff97-432b-b10a-df89bb204785-metrics-certs") pod "network-metrics-daemon-vtxgv" (UID: "d67ab7ea-ff97-432b-b10a-df89bb204785") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 10:08:20.546156 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:20.546078    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-50.ec2.internal" event="NodeReady"
Jul 28 10:08:20.546607 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:20.546232    2579 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 28 10:08:20.594480 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:20.594442    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-x9gk6"]
Jul 28 10:08:20.600669 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:20.600640    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-2qvgv"]
Jul 28 10:08:20.600818 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:20.600770    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-x9gk6"
Jul 28 10:08:20.602376 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:20.602351    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-2qvgv"
Jul 28 10:08:20.603196 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:20.603173    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-dv6fr\""
Jul 28 10:08:20.603306 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:20.603183    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 28 10:08:20.603306 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:20.603205    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 28 10:08:20.604808 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:20.604764    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-dfkl4\""
Jul 28 10:08:20.604808 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:20.604774    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 28 10:08:20.604808 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:20.604778    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 28 10:08:20.605046 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:20.604846    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 28 10:08:20.608338 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:20.608318    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-x9gk6"]
Jul 28 10:08:20.611630 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:20.611606    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-2qvgv"]
Jul 28 10:08:20.743304 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:20.743268    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69a2e04d-f363-4cfe-9dc0-61efcb33fc75-tmp-dir\") pod \"dns-default-x9gk6\" (UID: \"69a2e04d-f363-4cfe-9dc0-61efcb33fc75\") " pod="openshift-dns/dns-default-x9gk6"
Jul 28 10:08:20.743304 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:20.743309    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/69a2e04d-f363-4cfe-9dc0-61efcb33fc75-metrics-tls\") pod \"dns-default-x9gk6\" (UID: \"69a2e04d-f363-4cfe-9dc0-61efcb33fc75\") " pod="openshift-dns/dns-default-x9gk6"
Jul 28 10:08:20.743530 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:20.743338    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27lpd\" (UniqueName: \"kubernetes.io/projected/985feeac-ca2d-40aa-8ee6-53797559bc72-kube-api-access-27lpd\") pod \"ingress-canary-2qvgv\" (UID: \"985feeac-ca2d-40aa-8ee6-53797559bc72\") " pod="openshift-ingress-canary/ingress-canary-2qvgv"
Jul 28 10:08:20.743530 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:20.743427    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mlpq\" (UniqueName: \"kubernetes.io/projected/69a2e04d-f363-4cfe-9dc0-61efcb33fc75-kube-api-access-7mlpq\") pod \"dns-default-x9gk6\" (UID: \"69a2e04d-f363-4cfe-9dc0-61efcb33fc75\") " pod="openshift-dns/dns-default-x9gk6"
Jul 28 10:08:20.743530 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:20.743469    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/985feeac-ca2d-40aa-8ee6-53797559bc72-cert\") pod \"ingress-canary-2qvgv\" (UID: \"985feeac-ca2d-40aa-8ee6-53797559bc72\") " pod="openshift-ingress-canary/ingress-canary-2qvgv"
Jul 28 10:08:20.743690 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:20.743589    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/69a2e04d-f363-4cfe-9dc0-61efcb33fc75-config-volume\") pod \"dns-default-x9gk6\" (UID: \"69a2e04d-f363-4cfe-9dc0-61efcb33fc75\") " pod="openshift-dns/dns-default-x9gk6"
Jul 28 10:08:20.844715 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:20.844629    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69a2e04d-f363-4cfe-9dc0-61efcb33fc75-tmp-dir\") pod \"dns-default-x9gk6\" (UID: \"69a2e04d-f363-4cfe-9dc0-61efcb33fc75\") " pod="openshift-dns/dns-default-x9gk6"
Jul 28 10:08:20.844715 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:20.844693    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/69a2e04d-f363-4cfe-9dc0-61efcb33fc75-metrics-tls\") pod \"dns-default-x9gk6\" (UID: \"69a2e04d-f363-4cfe-9dc0-61efcb33fc75\") " pod="openshift-dns/dns-default-x9gk6"
Jul 28 10:08:20.844715 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:20.844718    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27lpd\" (UniqueName: \"kubernetes.io/projected/985feeac-ca2d-40aa-8ee6-53797559bc72-kube-api-access-27lpd\") pod \"ingress-canary-2qvgv\" (UID: \"985feeac-ca2d-40aa-8ee6-53797559bc72\") " pod="openshift-ingress-canary/ingress-canary-2qvgv"
Jul 28 10:08:20.845050 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:20.844755    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mlpq\" (UniqueName: \"kubernetes.io/projected/69a2e04d-f363-4cfe-9dc0-61efcb33fc75-kube-api-access-7mlpq\") pod \"dns-default-x9gk6\" (UID: \"69a2e04d-f363-4cfe-9dc0-61efcb33fc75\") " pod="openshift-dns/dns-default-x9gk6"
Jul 28 10:08:20.845050 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:20.844783    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/985feeac-ca2d-40aa-8ee6-53797559bc72-cert\") pod \"ingress-canary-2qvgv\" (UID: \"985feeac-ca2d-40aa-8ee6-53797559bc72\") " pod="openshift-ingress-canary/ingress-canary-2qvgv"
Jul 28 10:08:20.845050 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:20.844827    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/69a2e04d-f363-4cfe-9dc0-61efcb33fc75-config-volume\") pod \"dns-default-x9gk6\" (UID: \"69a2e04d-f363-4cfe-9dc0-61efcb33fc75\") " pod="openshift-dns/dns-default-x9gk6"
Jul 28 10:08:20.845050 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:08:20.844883    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 10:08:20.845050 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:08:20.844964    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69a2e04d-f363-4cfe-9dc0-61efcb33fc75-metrics-tls podName:69a2e04d-f363-4cfe-9dc0-61efcb33fc75 nodeName:}" failed. No retries permitted until 2026-07-28 10:08:21.344942922 +0000 UTC m=+35.608339242 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/69a2e04d-f363-4cfe-9dc0-61efcb33fc75-metrics-tls") pod "dns-default-x9gk6" (UID: "69a2e04d-f363-4cfe-9dc0-61efcb33fc75") : secret "dns-default-metrics-tls" not found
Jul 28 10:08:20.845050 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:20.845037    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69a2e04d-f363-4cfe-9dc0-61efcb33fc75-tmp-dir\") pod \"dns-default-x9gk6\" (UID: \"69a2e04d-f363-4cfe-9dc0-61efcb33fc75\") " pod="openshift-dns/dns-default-x9gk6"
Jul 28 10:08:20.845416 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:20.845394    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/69a2e04d-f363-4cfe-9dc0-61efcb33fc75-config-volume\") pod \"dns-default-x9gk6\" (UID: \"69a2e04d-f363-4cfe-9dc0-61efcb33fc75\") " pod="openshift-dns/dns-default-x9gk6"
Jul 28 10:08:20.849972 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:20.849951    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/985feeac-ca2d-40aa-8ee6-53797559bc72-cert\") pod \"ingress-canary-2qvgv\" (UID: \"985feeac-ca2d-40aa-8ee6-53797559bc72\") " pod="openshift-ingress-canary/ingress-canary-2qvgv"
Jul 28 10:08:20.854755 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:20.854733    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mlpq\" (UniqueName: \"kubernetes.io/projected/69a2e04d-f363-4cfe-9dc0-61efcb33fc75-kube-api-access-7mlpq\") pod \"dns-default-x9gk6\" (UID: \"69a2e04d-f363-4cfe-9dc0-61efcb33fc75\") " pod="openshift-dns/dns-default-x9gk6"
Jul 28 10:08:20.855200 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:20.855176    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-27lpd\" (UniqueName: \"kubernetes.io/projected/985feeac-ca2d-40aa-8ee6-53797559bc72-kube-api-access-27lpd\") pod \"ingress-canary-2qvgv\" (UID: \"985feeac-ca2d-40aa-8ee6-53797559bc72\") " pod="openshift-ingress-canary/ingress-canary-2qvgv"
Jul 28 10:08:20.919102 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:20.919060    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-2qvgv"
Jul 28 10:08:21.006305 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:21.006269    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:21.108262 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:21.108183    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:21.270316 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:21.270292    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4msch"
Jul 28 10:08:21.270455 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:21.270292    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vtxgv"
Jul 28 10:08:21.270509 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:21.270292    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bffsx"
Jul 28 10:08:21.273894 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:21.273810    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 28 10:08:21.273894 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:21.273840    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-l65kq\""
Jul 28 10:08:21.273894 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:21.273854    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 28 10:08:21.274114 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:21.273916    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 28 10:08:21.274114 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:21.273843    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-xsq5w\""
Jul 28 10:08:21.274178 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:21.274165    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 28 10:08:21.349310 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:21.349282    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/69a2e04d-f363-4cfe-9dc0-61efcb33fc75-metrics-tls\") pod \"dns-default-x9gk6\" (UID: \"69a2e04d-f363-4cfe-9dc0-61efcb33fc75\") " pod="openshift-dns/dns-default-x9gk6"
Jul 28 10:08:21.349416 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:08:21.349399    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 10:08:21.349488 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:08:21.349477    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69a2e04d-f363-4cfe-9dc0-61efcb33fc75-metrics-tls podName:69a2e04d-f363-4cfe-9dc0-61efcb33fc75 nodeName:}" failed. No retries permitted until 2026-07-28 10:08:22.349456326 +0000 UTC m=+36.612852634 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/69a2e04d-f363-4cfe-9dc0-61efcb33fc75-metrics-tls") pod "dns-default-x9gk6" (UID: "69a2e04d-f363-4cfe-9dc0-61efcb33fc75") : secret "dns-default-metrics-tls" not found
Jul 28 10:08:21.428112 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:21.427897    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-2qvgv"]
Jul 28 10:08:21.432852 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:08:21.432827    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod985feeac_ca2d_40aa_8ee6_53797559bc72.slice/crio-4c7d05bf4e726d5b6fd378c995221db5d038f026191636f605e78441312f1286 WatchSource:0}: Error finding container 4c7d05bf4e726d5b6fd378c995221db5d038f026191636f605e78441312f1286: Status 404 returned error can't find the container with id 4c7d05bf4e726d5b6fd378c995221db5d038f026191636f605e78441312f1286
Jul 28 10:08:21.442668 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:21.442638    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fftp9" event={"ID":"4f0d1f6f-a98f-4c02-a856-2465e90ce76e","Type":"ContainerStarted","Data":"d8e0f4ce07275e891bd6f789b38d02eeaece2a720400c6782f9a5e1206407788"}
Jul 28 10:08:21.443771 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:21.443749    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-2qvgv" event={"ID":"985feeac-ca2d-40aa-8ee6-53797559bc72","Type":"ContainerStarted","Data":"4c7d05bf4e726d5b6fd378c995221db5d038f026191636f605e78441312f1286"}
Jul 28 10:08:21.565683 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:21.565655    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:21.736814 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:21.736707    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:21.859632 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:21.859603    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:22.357603 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:22.357561    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/69a2e04d-f363-4cfe-9dc0-61efcb33fc75-metrics-tls\") pod \"dns-default-x9gk6\" (UID: \"69a2e04d-f363-4cfe-9dc0-61efcb33fc75\") " pod="openshift-dns/dns-default-x9gk6"
Jul 28 10:08:22.357822 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:08:22.357777    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 10:08:22.357887 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:08:22.357850    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69a2e04d-f363-4cfe-9dc0-61efcb33fc75-metrics-tls podName:69a2e04d-f363-4cfe-9dc0-61efcb33fc75 nodeName:}" failed. No retries permitted until 2026-07-28 10:08:24.357831006 +0000 UTC m=+38.621227332 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/69a2e04d-f363-4cfe-9dc0-61efcb33fc75-metrics-tls") pod "dns-default-x9gk6" (UID: "69a2e04d-f363-4cfe-9dc0-61efcb33fc75") : secret "dns-default-metrics-tls" not found
Jul 28 10:08:22.449127 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:22.449088    2579 generic.go:356] "Generic (PLEG): container finished" podID="4f0d1f6f-a98f-4c02-a856-2465e90ce76e" containerID="d8e0f4ce07275e891bd6f789b38d02eeaece2a720400c6782f9a5e1206407788" exitCode=0
Jul 28 10:08:22.449127 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:22.449141    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fftp9" event={"ID":"4f0d1f6f-a98f-4c02-a856-2465e90ce76e","Type":"ContainerDied","Data":"d8e0f4ce07275e891bd6f789b38d02eeaece2a720400c6782f9a5e1206407788"}
Jul 28 10:08:22.655406 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:22.655330    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-mtcd6"]
Jul 28 10:08:22.657354 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:22.657333    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-mtcd6"
Jul 28 10:08:22.666790 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:22.666764    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 28 10:08:22.666923 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:22.666851    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-q8lkz\""
Jul 28 10:08:22.667095 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:22.667066    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 28 10:08:22.667710 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:22.667688    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 28 10:08:22.669548 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:22.669520    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 28 10:08:22.704935 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:22.704889    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:22.705409 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:22.705377    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-wdgll"]
Jul 28 10:08:22.707256 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:22.707235    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-wdgll"
Jul 28 10:08:22.711156 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:22.711120    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-w9kbz\""
Jul 28 10:08:22.711266 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:22.711231    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Jul 28 10:08:22.712348 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:22.712315    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Jul 28 10:08:22.731418 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:22.731387    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-mtcd6"]
Jul 28 10:08:22.758233 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:22.758198    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-wdgll"]
Jul 28 10:08:22.761601 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:22.761572    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3fd0cb5a-eb68-45ec-89f2-df8fcf761def-crio-socket\") pod \"insights-runtime-extractor-mtcd6\" (UID: \"3fd0cb5a-eb68-45ec-89f2-df8fcf761def\") " pod="openshift-insights/insights-runtime-extractor-mtcd6"
Jul 28 10:08:22.761750 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:22.761622    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3fd0cb5a-eb68-45ec-89f2-df8fcf761def-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mtcd6\" (UID: \"3fd0cb5a-eb68-45ec-89f2-df8fcf761def\") " pod="openshift-insights/insights-runtime-extractor-mtcd6"
Jul 28 10:08:22.761750 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:22.761734    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3fd0cb5a-eb68-45ec-89f2-df8fcf761def-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mtcd6\" (UID: \"3fd0cb5a-eb68-45ec-89f2-df8fcf761def\") " pod="openshift-insights/insights-runtime-extractor-mtcd6"
Jul 28 10:08:22.761859 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:22.761780    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3fd0cb5a-eb68-45ec-89f2-df8fcf761def-data-volume\") pod \"insights-runtime-extractor-mtcd6\" (UID: \"3fd0cb5a-eb68-45ec-89f2-df8fcf761def\") " pod="openshift-insights/insights-runtime-extractor-mtcd6"
Jul 28 10:08:22.761859 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:22.761806    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-844xc\" (UniqueName: \"kubernetes.io/projected/3fd0cb5a-eb68-45ec-89f2-df8fcf761def-kube-api-access-844xc\") pod \"insights-runtime-extractor-mtcd6\" (UID: \"3fd0cb5a-eb68-45ec-89f2-df8fcf761def\") " pod="openshift-insights/insights-runtime-extractor-mtcd6"
Jul 28 10:08:22.862912 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:22.862872    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3fd0cb5a-eb68-45ec-89f2-df8fcf761def-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mtcd6\" (UID: \"3fd0cb5a-eb68-45ec-89f2-df8fcf761def\") " pod="openshift-insights/insights-runtime-extractor-mtcd6"
Jul 28 10:08:22.862912 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:22.862918    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3fd0cb5a-eb68-45ec-89f2-df8fcf761def-data-volume\") pod \"insights-runtime-extractor-mtcd6\" (UID: \"3fd0cb5a-eb68-45ec-89f2-df8fcf761def\") " pod="openshift-insights/insights-runtime-extractor-mtcd6"
Jul 28 10:08:22.863147 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:22.862938    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-844xc\" (UniqueName: \"kubernetes.io/projected/3fd0cb5a-eb68-45ec-89f2-df8fcf761def-kube-api-access-844xc\") pod \"insights-runtime-extractor-mtcd6\" (UID: \"3fd0cb5a-eb68-45ec-89f2-df8fcf761def\") " pod="openshift-insights/insights-runtime-extractor-mtcd6"
Jul 28 10:08:22.863147 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:08:22.863058    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 10:08:22.863147 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:08:22.863131    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3fd0cb5a-eb68-45ec-89f2-df8fcf761def-insights-runtime-extractor-tls podName:3fd0cb5a-eb68-45ec-89f2-df8fcf761def nodeName:}" failed. No retries permitted until 2026-07-28 10:08:23.363109904 +0000 UTC m=+37.626506234 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3fd0cb5a-eb68-45ec-89f2-df8fcf761def-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mtcd6" (UID: "3fd0cb5a-eb68-45ec-89f2-df8fcf761def") : secret "insights-runtime-extractor-tls" not found
Jul 28 10:08:22.863333 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:22.863195    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3fd0cb5a-eb68-45ec-89f2-df8fcf761def-crio-socket\") pod \"insights-runtime-extractor-mtcd6\" (UID: \"3fd0cb5a-eb68-45ec-89f2-df8fcf761def\") " pod="openshift-insights/insights-runtime-extractor-mtcd6"
Jul 28 10:08:22.863333 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:22.863239    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3fd0cb5a-eb68-45ec-89f2-df8fcf761def-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mtcd6\" (UID: \"3fd0cb5a-eb68-45ec-89f2-df8fcf761def\") " pod="openshift-insights/insights-runtime-extractor-mtcd6"
Jul 28 10:08:22.863333 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:22.863286    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3fd0cb5a-eb68-45ec-89f2-df8fcf761def-data-volume\") pod \"insights-runtime-extractor-mtcd6\" (UID: \"3fd0cb5a-eb68-45ec-89f2-df8fcf761def\") " pod="openshift-insights/insights-runtime-extractor-mtcd6"
Jul 28 10:08:22.863333 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:22.863299    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdxqk\" (UniqueName: \"kubernetes.io/projected/e1e132cc-95bb-466a-a573-f40cf5cf8e60-kube-api-access-jdxqk\") pod \"migrator-6865c9967-wdgll\" (UID: \"e1e132cc-95bb-466a-a573-f40cf5cf8e60\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-wdgll"
Jul 28 10:08:22.863555 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:08:22.863347    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 10:08:22.863555 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:08:22.863397    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3fd0cb5a-eb68-45ec-89f2-df8fcf761def-kube-rbac-proxy-cm podName:3fd0cb5a-eb68-45ec-89f2-df8fcf761def nodeName:}" failed. No retries permitted until 2026-07-28 10:08:23.363384403 +0000 UTC m=+37.626780743 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3fd0cb5a-eb68-45ec-89f2-df8fcf761def-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mtcd6" (UID: "3fd0cb5a-eb68-45ec-89f2-df8fcf761def") : configmap "kube-rbac-proxy" not found
Jul 28 10:08:22.863555 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:22.863470    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3fd0cb5a-eb68-45ec-89f2-df8fcf761def-crio-socket\") pod \"insights-runtime-extractor-mtcd6\" (UID: \"3fd0cb5a-eb68-45ec-89f2-df8fcf761def\") " pod="openshift-insights/insights-runtime-extractor-mtcd6"
Jul 28 10:08:22.880422 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:22.880394    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-844xc\" (UniqueName: \"kubernetes.io/projected/3fd0cb5a-eb68-45ec-89f2-df8fcf761def-kube-api-access-844xc\") pod \"insights-runtime-extractor-mtcd6\" (UID: \"3fd0cb5a-eb68-45ec-89f2-df8fcf761def\") " pod="openshift-insights/insights-runtime-extractor-mtcd6"
Jul 28 10:08:22.947794 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:22.947715    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:22.964679 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:22.964653    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdxqk\" (UniqueName: \"kubernetes.io/projected/e1e132cc-95bb-466a-a573-f40cf5cf8e60-kube-api-access-jdxqk\") pod \"migrator-6865c9967-wdgll\" (UID: \"e1e132cc-95bb-466a-a573-f40cf5cf8e60\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-wdgll"
Jul 28 10:08:22.992057 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:22.992026    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdxqk\" (UniqueName: \"kubernetes.io/projected/e1e132cc-95bb-466a-a573-f40cf5cf8e60-kube-api-access-jdxqk\") pod \"migrator-6865c9967-wdgll\" (UID: \"e1e132cc-95bb-466a-a573-f40cf5cf8e60\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-wdgll"
Jul 28 10:08:23.017225 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:23.017189    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-wdgll"
Jul 28 10:08:23.155444 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:23.155416    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:23.267728 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:23.267657    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:23.368101 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:23.368066    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3fd0cb5a-eb68-45ec-89f2-df8fcf761def-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mtcd6\" (UID: \"3fd0cb5a-eb68-45ec-89f2-df8fcf761def\") " pod="openshift-insights/insights-runtime-extractor-mtcd6"
Jul 28 10:08:23.368286 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:23.368171    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3fd0cb5a-eb68-45ec-89f2-df8fcf761def-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mtcd6\" (UID: \"3fd0cb5a-eb68-45ec-89f2-df8fcf761def\") " pod="openshift-insights/insights-runtime-extractor-mtcd6"
Jul 28 10:08:23.368286 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:08:23.368251    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 10:08:23.368397 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:08:23.368335    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3fd0cb5a-eb68-45ec-89f2-df8fcf761def-insights-runtime-extractor-tls podName:3fd0cb5a-eb68-45ec-89f2-df8fcf761def nodeName:}" failed. No retries permitted until 2026-07-28 10:08:24.368313097 +0000 UTC m=+38.631709421 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3fd0cb5a-eb68-45ec-89f2-df8fcf761def-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mtcd6" (UID: "3fd0cb5a-eb68-45ec-89f2-df8fcf761def") : secret "insights-runtime-extractor-tls" not found
Jul 28 10:08:23.368397 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:08:23.368259    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 10:08:23.368496 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:08:23.368441    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3fd0cb5a-eb68-45ec-89f2-df8fcf761def-kube-rbac-proxy-cm podName:3fd0cb5a-eb68-45ec-89f2-df8fcf761def nodeName:}" failed. No retries permitted until 2026-07-28 10:08:24.368421074 +0000 UTC m=+38.631817385 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3fd0cb5a-eb68-45ec-89f2-df8fcf761def-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mtcd6" (UID: "3fd0cb5a-eb68-45ec-89f2-df8fcf761def") : configmap "kube-rbac-proxy" not found
Jul 28 10:08:23.456204 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:23.456163    2579 generic.go:356] "Generic (PLEG): container finished" podID="4f0d1f6f-a98f-4c02-a856-2465e90ce76e" containerID="0ccff2a693bb4f743174d530e1263dd93597dcf6b82cb9a8a8c26acdc3a7694d" exitCode=0
Jul 28 10:08:23.456382 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:23.456249    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fftp9" event={"ID":"4f0d1f6f-a98f-4c02-a856-2465e90ce76e","Type":"ContainerDied","Data":"0ccff2a693bb4f743174d530e1263dd93597dcf6b82cb9a8a8c26acdc3a7694d"}
Jul 28 10:08:23.688287 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:23.688000    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-wdgll"]
Jul 28 10:08:23.798045 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:23.798010    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:23.901732 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:23.901686    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:24.377649 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:24.377610    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3fd0cb5a-eb68-45ec-89f2-df8fcf761def-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mtcd6\" (UID: \"3fd0cb5a-eb68-45ec-89f2-df8fcf761def\") " pod="openshift-insights/insights-runtime-extractor-mtcd6"
Jul 28 10:08:24.377789 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:24.377661    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3fd0cb5a-eb68-45ec-89f2-df8fcf761def-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mtcd6\" (UID: \"3fd0cb5a-eb68-45ec-89f2-df8fcf761def\") " pod="openshift-insights/insights-runtime-extractor-mtcd6"
Jul 28 10:08:24.377789 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:24.377702    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/69a2e04d-f363-4cfe-9dc0-61efcb33fc75-metrics-tls\") pod \"dns-default-x9gk6\" (UID: \"69a2e04d-f363-4cfe-9dc0-61efcb33fc75\") " pod="openshift-dns/dns-default-x9gk6"
Jul 28 10:08:24.377789 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:08:24.377739    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 10:08:24.377899 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:08:24.377806    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3fd0cb5a-eb68-45ec-89f2-df8fcf761def-kube-rbac-proxy-cm podName:3fd0cb5a-eb68-45ec-89f2-df8fcf761def nodeName:}" failed. No retries permitted until 2026-07-28 10:08:26.377790073 +0000 UTC m=+40.641186379 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3fd0cb5a-eb68-45ec-89f2-df8fcf761def-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mtcd6" (UID: "3fd0cb5a-eb68-45ec-89f2-df8fcf761def") : configmap "kube-rbac-proxy" not found
Jul 28 10:08:24.377899 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:08:24.377822    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 10:08:24.377899 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:08:24.377866    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69a2e04d-f363-4cfe-9dc0-61efcb33fc75-metrics-tls podName:69a2e04d-f363-4cfe-9dc0-61efcb33fc75 nodeName:}" failed. No retries permitted until 2026-07-28 10:08:28.377855099 +0000 UTC m=+42.641251406 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/69a2e04d-f363-4cfe-9dc0-61efcb33fc75-metrics-tls") pod "dns-default-x9gk6" (UID: "69a2e04d-f363-4cfe-9dc0-61efcb33fc75") : secret "dns-default-metrics-tls" not found
Jul 28 10:08:24.377899 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:08:24.377822    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 10:08:24.377899 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:08:24.377898    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3fd0cb5a-eb68-45ec-89f2-df8fcf761def-insights-runtime-extractor-tls podName:3fd0cb5a-eb68-45ec-89f2-df8fcf761def nodeName:}" failed. No retries permitted until 2026-07-28 10:08:26.377889646 +0000 UTC m=+40.641285954 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3fd0cb5a-eb68-45ec-89f2-df8fcf761def-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mtcd6" (UID: "3fd0cb5a-eb68-45ec-89f2-df8fcf761def") : secret "insights-runtime-extractor-tls" not found
Jul 28 10:08:24.463260 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:24.463226    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fftp9" event={"ID":"4f0d1f6f-a98f-4c02-a856-2465e90ce76e","Type":"ContainerStarted","Data":"bf7aa44fd5a487bd9a23243b1b28ce01c7380dca3ca825b38842dfee98e6654f"}
Jul 28 10:08:24.464811 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:24.464781    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-2qvgv" event={"ID":"985feeac-ca2d-40aa-8ee6-53797559bc72","Type":"ContainerStarted","Data":"693d227eab275bd3cbc6f559e83e2ee1092cb6753886e41e0f4b5469a86b69e2"}
Jul 28 10:08:24.465935 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:24.465906    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-wdgll" event={"ID":"e1e132cc-95bb-466a-a573-f40cf5cf8e60","Type":"ContainerStarted","Data":"b3789e2af50e11407e250b618efe461b18d61e1679f9fa7f04711469adc67b38"}
Jul 28 10:08:24.491555 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:24.491496    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-fftp9" podStartSLOduration=8.794181882 podStartE2EDuration="38.491473632s" totalImagesPullingTime="29.69729175s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 10:07:46 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:08:24.490077327 +0000 UTC m=+38.753473657" watchObservedRunningTime="2026-07-28 10:08:24.491473632 +0000 UTC m=+38.754869962"
Jul 28 10:08:24.509155 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:24.509103    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-2qvgv" podStartSLOduration=2.237868856 podStartE2EDuration="4.50908518s" totalImagesPullingTime="2.271216324s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:08:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:08:24.507969517 +0000 UTC m=+38.771365846" watchObservedRunningTime="2026-07-28 10:08:24.50908518 +0000 UTC m=+38.772481509"
Jul 28 10:08:24.666086 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:24.666014    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:24.760892 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:24.760859    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:25.271158 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:25.271129    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-2rf9h_ba296f38-5de8-4d11-b789-7a727a01b9cc/dns-node-resolver/0.log"
Jul 28 10:08:25.385706 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:25.385671    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:25.470529 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:25.470494    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-wdgll" event={"ID":"e1e132cc-95bb-466a-a573-f40cf5cf8e60","Type":"ContainerStarted","Data":"cbc5ec1071ec5ad72b761360748886fbc535c390303e01762dea4d95c8ece7eb"}
Jul 28 10:08:25.470730 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:25.470562    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-wdgll" event={"ID":"e1e132cc-95bb-466a-a573-f40cf5cf8e60","Type":"ContainerStarted","Data":"c96d1cc2cb5735cf3260b1377dab5afde5ac19179ae6048d43ae8d5606abdb46"}
Jul 28 10:08:25.494825 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:25.494775    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-wdgll" podStartSLOduration=2.471477273 podStartE2EDuration="3.494757596s" totalImagesPullingTime="1.023280323s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:08:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:08:25.49422653 +0000 UTC m=+39.757622882" watchObservedRunningTime="2026-07-28 10:08:25.494757596 +0000 UTC m=+39.758153945"
Jul 28 10:08:25.495963 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:25.495939    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:25.624728 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:25.624640    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:25.710580 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:25.710549    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:26.084318 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:26.084235    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:26.268150 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:26.268113    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:26.394913 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:26.394825    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3fd0cb5a-eb68-45ec-89f2-df8fcf761def-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mtcd6\" (UID: \"3fd0cb5a-eb68-45ec-89f2-df8fcf761def\") " pod="openshift-insights/insights-runtime-extractor-mtcd6"
Jul 28 10:08:26.394913 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:26.394878    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3fd0cb5a-eb68-45ec-89f2-df8fcf761def-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mtcd6\" (UID: \"3fd0cb5a-eb68-45ec-89f2-df8fcf761def\") " pod="openshift-insights/insights-runtime-extractor-mtcd6"
Jul 28 10:08:26.395089 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:08:26.394951    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 10:08:26.395089 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:08:26.394971    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 10:08:26.395089 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:08:26.395022    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3fd0cb5a-eb68-45ec-89f2-df8fcf761def-kube-rbac-proxy-cm podName:3fd0cb5a-eb68-45ec-89f2-df8fcf761def nodeName:}" failed. No retries permitted until 2026-07-28 10:08:30.39500618 +0000 UTC m=+44.658402487 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3fd0cb5a-eb68-45ec-89f2-df8fcf761def-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mtcd6" (UID: "3fd0cb5a-eb68-45ec-89f2-df8fcf761def") : configmap "kube-rbac-proxy" not found
Jul 28 10:08:26.395089 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:08:26.395039    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3fd0cb5a-eb68-45ec-89f2-df8fcf761def-insights-runtime-extractor-tls podName:3fd0cb5a-eb68-45ec-89f2-df8fcf761def nodeName:}" failed. No retries permitted until 2026-07-28 10:08:30.395032531 +0000 UTC m=+44.658428837 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3fd0cb5a-eb68-45ec-89f2-df8fcf761def-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mtcd6" (UID: "3fd0cb5a-eb68-45ec-89f2-df8fcf761def") : secret "insights-runtime-extractor-tls" not found
Jul 28 10:08:26.667837 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:26.667756    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-mkf4n_beb002bc-8a51-44cf-8274-fd83f8d1d237/node-ca/0.log"
Jul 28 10:08:26.771184 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:26.771150    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:27.264766 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:27.264744    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-2qvgv_985feeac-ca2d-40aa-8ee6-53797559bc72/serve-healthcheck-canary/0.log"
Jul 28 10:08:27.665467 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:27.665394    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-wdgll_e1e132cc-95bb-466a-a573-f40cf5cf8e60/migrator/0.log"
Jul 28 10:08:27.865105 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:27.865076    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-wdgll_e1e132cc-95bb-466a-a573-f40cf5cf8e60/graceful-termination/0.log"
Jul 28 10:08:28.014376 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:28.014299    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:28.410659 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:28.410570    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/69a2e04d-f363-4cfe-9dc0-61efcb33fc75-metrics-tls\") pod \"dns-default-x9gk6\" (UID: \"69a2e04d-f363-4cfe-9dc0-61efcb33fc75\") " pod="openshift-dns/dns-default-x9gk6"
Jul 28 10:08:28.411014 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:08:28.410714    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 10:08:28.411014 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:08:28.410775    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69a2e04d-f363-4cfe-9dc0-61efcb33fc75-metrics-tls podName:69a2e04d-f363-4cfe-9dc0-61efcb33fc75 nodeName:}" failed. No retries permitted until 2026-07-28 10:08:36.410760141 +0000 UTC m=+50.674156449 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/69a2e04d-f363-4cfe-9dc0-61efcb33fc75-metrics-tls") pod "dns-default-x9gk6" (UID: "69a2e04d-f363-4cfe-9dc0-61efcb33fc75") : secret "dns-default-metrics-tls" not found
Jul 28 10:08:29.513966 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:29.513927    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:29.639555 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:29.639479    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:29.740113 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:29.740078    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:29.839460 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:29.839425    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:29.941924 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:29.941892    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:30.207608 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:30.207508    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:30.425117 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:30.425076    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3fd0cb5a-eb68-45ec-89f2-df8fcf761def-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mtcd6\" (UID: \"3fd0cb5a-eb68-45ec-89f2-df8fcf761def\") " pod="openshift-insights/insights-runtime-extractor-mtcd6"
Jul 28 10:08:30.425309 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:30.425163    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3fd0cb5a-eb68-45ec-89f2-df8fcf761def-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mtcd6\" (UID: \"3fd0cb5a-eb68-45ec-89f2-df8fcf761def\") " pod="openshift-insights/insights-runtime-extractor-mtcd6"
Jul 28 10:08:30.425309 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:08:30.425232    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 10:08:30.425309 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:08:30.425295    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3fd0cb5a-eb68-45ec-89f2-df8fcf761def-insights-runtime-extractor-tls podName:3fd0cb5a-eb68-45ec-89f2-df8fcf761def nodeName:}" failed. No retries permitted until 2026-07-28 10:08:38.425279413 +0000 UTC m=+52.688675720 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3fd0cb5a-eb68-45ec-89f2-df8fcf761def-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mtcd6" (UID: "3fd0cb5a-eb68-45ec-89f2-df8fcf761def") : secret "insights-runtime-extractor-tls" not found
Jul 28 10:08:30.425712 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:30.425696    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3fd0cb5a-eb68-45ec-89f2-df8fcf761def-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mtcd6\" (UID: \"3fd0cb5a-eb68-45ec-89f2-df8fcf761def\") " pod="openshift-insights/insights-runtime-extractor-mtcd6"
Jul 28 10:08:30.600018 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:30.599984    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:31.004138 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:31.004054    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:31.200459 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:31.200429    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:31.605453 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:31.605411    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:32.002744 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:32.002666    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:32.244429 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:32.244400    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:32.340120 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:32.340085    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:32.803727 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:32.803698    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:33.203431 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:33.203360    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:33.340523 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:33.340490    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:33.443645 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:33.443609    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:33.585939 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:33.585908    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:33.682740 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:33.682706    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:33.824842 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:33.824805    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:33.901774 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:33.901693    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:34.055659 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:34.055627    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:34.230725 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:34.230646    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:34.332969 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:34.332934    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:34.455476 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:34.455443    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:34.627471 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:34.627440    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:34.718187 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:34.718152    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:34.865554 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:34.865511    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:35.061511 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:35.061484    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:35.213976 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:35.213941    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:35.311165 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:35.311133    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:35.411699 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:35.411620    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:35.629111 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:35.629076    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:35.746287 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:35.746209    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:35.863242 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:35.863207    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:36.000101 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:36.000020    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:36.470045 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:36.470014    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/69a2e04d-f363-4cfe-9dc0-61efcb33fc75-metrics-tls\") pod \"dns-default-x9gk6\" (UID: \"69a2e04d-f363-4cfe-9dc0-61efcb33fc75\") " pod="openshift-dns/dns-default-x9gk6"
Jul 28 10:08:36.473371 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:36.473340    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/69a2e04d-f363-4cfe-9dc0-61efcb33fc75-metrics-tls\") pod \"dns-default-x9gk6\" (UID: \"69a2e04d-f363-4cfe-9dc0-61efcb33fc75\") " pod="openshift-dns/dns-default-x9gk6"
Jul 28 10:08:36.513978 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:36.513945    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-x9gk6"
Jul 28 10:08:36.602993 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:36.602954    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:36.678062 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:36.678029    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-x9gk6"]
Jul 28 10:08:36.680891 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:08:36.680865    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69a2e04d_f363_4cfe_9dc0_61efcb33fc75.slice/crio-bdb927a4b70e8215be49f83d1908c8b28439f3ad9f72f7877cea25678b44cf41 WatchSource:0}: Error finding container bdb927a4b70e8215be49f83d1908c8b28439f3ad9f72f7877cea25678b44cf41: Status 404 returned error can't find the container with id bdb927a4b70e8215be49f83d1908c8b28439f3ad9f72f7877cea25678b44cf41
Jul 28 10:08:37.404590 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:37.404553    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:37.485157 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:37.485123    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:37.498608 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:37.498572    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-x9gk6" event={"ID":"69a2e04d-f363-4cfe-9dc0-61efcb33fc75","Type":"ContainerStarted","Data":"bdb927a4b70e8215be49f83d1908c8b28439f3ad9f72f7877cea25678b44cf41"}
Jul 28 10:08:37.769281 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:37.769202    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:37.840154 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:37.840121    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:38.012816 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:38.012776    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:38.396807 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:38.396778    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:38.487857 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:38.487824    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3fd0cb5a-eb68-45ec-89f2-df8fcf761def-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mtcd6\" (UID: \"3fd0cb5a-eb68-45ec-89f2-df8fcf761def\") " pod="openshift-insights/insights-runtime-extractor-mtcd6"
Jul 28 10:08:38.490250 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:38.490226    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3fd0cb5a-eb68-45ec-89f2-df8fcf761def-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mtcd6\" (UID: \"3fd0cb5a-eb68-45ec-89f2-df8fcf761def\") " pod="openshift-insights/insights-runtime-extractor-mtcd6"
Jul 28 10:08:38.568217 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:38.568189    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-mtcd6"
Jul 28 10:08:38.815119 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:38.815090    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-mtcd6"]
Jul 28 10:08:38.818056 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:08:38.818028    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fd0cb5a_eb68_45ec_89f2_df8fcf761def.slice/crio-4f716fa9cb603812a63d6ee624861e4e151a5463e96bb4e01cc8cf849a2c0e20 WatchSource:0}: Error finding container 4f716fa9cb603812a63d6ee624861e4e151a5463e96bb4e01cc8cf849a2c0e20: Status 404 returned error can't find the container with id 4f716fa9cb603812a63d6ee624861e4e151a5463e96bb4e01cc8cf849a2c0e20
Jul 28 10:08:39.205481 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:39.205443    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:39.506225 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:39.506187    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-x9gk6" event={"ID":"69a2e04d-f363-4cfe-9dc0-61efcb33fc75","Type":"ContainerStarted","Data":"d5ead4ccba86b5065dd89bd5b1fb83a55ebb59e2445d73a671cd226a25c1ca32"}
Jul 28 10:08:39.506706 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:39.506255    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-x9gk6" event={"ID":"69a2e04d-f363-4cfe-9dc0-61efcb33fc75","Type":"ContainerStarted","Data":"40c1b692e77467797289cab05418944575e23489a7f46cb9e2887767e86bb136"}
Jul 28 10:08:39.506706 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:39.506311    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-x9gk6"
Jul 28 10:08:39.507698 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:39.507668    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mtcd6" event={"ID":"3fd0cb5a-eb68-45ec-89f2-df8fcf761def","Type":"ContainerStarted","Data":"4f716fa9cb603812a63d6ee624861e4e151a5463e96bb4e01cc8cf849a2c0e20"}
Jul 28 10:08:39.525447 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:39.525295    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-x9gk6" podStartSLOduration=17.512798495 podStartE2EDuration="19.525275771s" totalImagesPullingTime="2.012477276s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:08:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:08:39.525232191 +0000 UTC m=+53.788628522" watchObservedRunningTime="2026-07-28 10:08:39.525275771 +0000 UTC m=+53.788672102"
Jul 28 10:08:39.598475 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:39.598441    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:39.881557 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:39.881508    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:40.009151 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:40.009117    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:40.394820 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:40.394785    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:40.600485 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:40.600435    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:42.283119 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:42.283034    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:44.525655 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:44.525616    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mtcd6" event={"ID":"3fd0cb5a-eb68-45ec-89f2-df8fcf761def","Type":"ContainerStarted","Data":"1da30e362d1b7e3af974bd834d01ffdff4bf7c0e917343f3648aea62da2d509e"}
Jul 28 10:08:46.754609 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:46.754548    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:46.864466 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:46.864399    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:47.318070 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:47.318035    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:47.441826 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:47.441797    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-cjlcq"
Jul 28 10:08:47.462518 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:47.462472    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:47.925128 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:47.925049    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b5b47b457-6wkt2"]
Jul 28 10:08:47.926909 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:47.926893    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5979786f78-glq6p"]
Jul 28 10:08:47.927032 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:47.927017    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b5b47b457-6wkt2"
Jul 28 10:08:47.928611 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:47.928592    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5979786f78-glq6p"
Jul 28 10:08:47.954650 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:47.954628    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 28 10:08:47.959335 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:47.959291    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 28 10:08:47.959649 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:47.959634    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 28 10:08:47.959877 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:47.959858    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 28 10:08:47.963108 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:47.963085    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b46a706b-7a33-4098-afbb-4f864a378bac-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5b5b47b457-6wkt2\" (UID: \"b46a706b-7a33-4098-afbb-4f864a378bac\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b5b47b457-6wkt2"
Jul 28 10:08:47.963194 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:47.963128    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/b46a706b-7a33-4098-afbb-4f864a378bac-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5b5b47b457-6wkt2\" (UID: \"b46a706b-7a33-4098-afbb-4f864a378bac\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b5b47b457-6wkt2"
Jul 28 10:08:47.963194 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:47.963147    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/b46a706b-7a33-4098-afbb-4f864a378bac-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5b5b47b457-6wkt2\" (UID: \"b46a706b-7a33-4098-afbb-4f864a378bac\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b5b47b457-6wkt2"
Jul 28 10:08:47.963271 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:47.963201    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/b46a706b-7a33-4098-afbb-4f864a378bac-ca\") pod \"cluster-proxy-proxy-agent-5b5b47b457-6wkt2\" (UID: \"b46a706b-7a33-4098-afbb-4f864a378bac\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b5b47b457-6wkt2"
Jul 28 10:08:47.963271 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:47.963219    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/b46a706b-7a33-4098-afbb-4f864a378bac-hub\") pod \"cluster-proxy-proxy-agent-5b5b47b457-6wkt2\" (UID: \"b46a706b-7a33-4098-afbb-4f864a378bac\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b5b47b457-6wkt2"
Jul 28 10:08:47.963271 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:47.963235    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srth9\" (UniqueName: \"kubernetes.io/projected/87291d3d-abe4-45a1-93f7-851250382566-kube-api-access-srth9\") pod \"managed-serviceaccount-addon-agent-5979786f78-glq6p\" (UID: \"87291d3d-abe4-45a1-93f7-851250382566\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5979786f78-glq6p"
Jul 28 10:08:47.963271 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:47.963262    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ptwb\" (UniqueName: \"kubernetes.io/projected/b46a706b-7a33-4098-afbb-4f864a378bac-kube-api-access-2ptwb\") pod \"cluster-proxy-proxy-agent-5b5b47b457-6wkt2\" (UID: \"b46a706b-7a33-4098-afbb-4f864a378bac\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b5b47b457-6wkt2"
Jul 28 10:08:47.963519 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:47.963280    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/87291d3d-abe4-45a1-93f7-851250382566-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5979786f78-glq6p\" (UID: \"87291d3d-abe4-45a1-93f7-851250382566\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5979786f78-glq6p"
Jul 28 10:08:47.979706 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:47.979681    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b5b47b457-6wkt2"]
Jul 28 10:08:47.997722 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:47.997697    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-nrcl5\""
Jul 28 10:08:47.997865 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:47.997788    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 28 10:08:47.999607 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:47.999590    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 28 10:08:48.000806 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:48.000784    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 28 10:08:48.003628 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:48.003601    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Jul 28 10:08:48.030249 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:48.030221    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5979786f78-glq6p"]
Jul 28 10:08:48.064049 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:48.064000    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b46a706b-7a33-4098-afbb-4f864a378bac-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5b5b47b457-6wkt2\" (UID: \"b46a706b-7a33-4098-afbb-4f864a378bac\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b5b47b457-6wkt2"
Jul 28 10:08:48.064233 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:48.064079    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/b46a706b-7a33-4098-afbb-4f864a378bac-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5b5b47b457-6wkt2\" (UID: \"b46a706b-7a33-4098-afbb-4f864a378bac\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b5b47b457-6wkt2"
Jul 28 10:08:48.064233 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:48.064107    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/b46a706b-7a33-4098-afbb-4f864a378bac-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5b5b47b457-6wkt2\" (UID: \"b46a706b-7a33-4098-afbb-4f864a378bac\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b5b47b457-6wkt2"
Jul 28 10:08:48.064354 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:48.064252    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/b46a706b-7a33-4098-afbb-4f864a378bac-ca\") pod \"cluster-proxy-proxy-agent-5b5b47b457-6wkt2\" (UID: \"b46a706b-7a33-4098-afbb-4f864a378bac\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b5b47b457-6wkt2"
Jul 28 10:08:48.064354 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:48.064280    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/b46a706b-7a33-4098-afbb-4f864a378bac-hub\") pod \"cluster-proxy-proxy-agent-5b5b47b457-6wkt2\" (UID: \"b46a706b-7a33-4098-afbb-4f864a378bac\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b5b47b457-6wkt2"
Jul 28 10:08:48.064354 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:48.064306    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srth9\" (UniqueName: \"kubernetes.io/projected/87291d3d-abe4-45a1-93f7-851250382566-kube-api-access-srth9\") pod \"managed-serviceaccount-addon-agent-5979786f78-glq6p\" (UID: \"87291d3d-abe4-45a1-93f7-851250382566\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5979786f78-glq6p"
Jul 28 10:08:48.064354 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:48.064342    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ptwb\" (UniqueName: \"kubernetes.io/projected/b46a706b-7a33-4098-afbb-4f864a378bac-kube-api-access-2ptwb\") pod \"cluster-proxy-proxy-agent-5b5b47b457-6wkt2\" (UID: \"b46a706b-7a33-4098-afbb-4f864a378bac\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b5b47b457-6wkt2"
Jul 28 10:08:48.064565 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:48.064368    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/87291d3d-abe4-45a1-93f7-851250382566-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5979786f78-glq6p\" (UID: \"87291d3d-abe4-45a1-93f7-851250382566\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5979786f78-glq6p"
Jul 28 10:08:48.065062 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:48.064995    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/b46a706b-7a33-4098-afbb-4f864a378bac-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5b5b47b457-6wkt2\" (UID: \"b46a706b-7a33-4098-afbb-4f864a378bac\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b5b47b457-6wkt2"
Jul 28 10:08:48.067865 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:48.067415    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b46a706b-7a33-4098-afbb-4f864a378bac-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5b5b47b457-6wkt2\" (UID: \"b46a706b-7a33-4098-afbb-4f864a378bac\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b5b47b457-6wkt2"
Jul 28 10:08:48.067865 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:48.067822    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/b46a706b-7a33-4098-afbb-4f864a378bac-ca\") pod \"cluster-proxy-proxy-agent-5b5b47b457-6wkt2\" (UID: \"b46a706b-7a33-4098-afbb-4f864a378bac\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b5b47b457-6wkt2"
Jul 28 10:08:48.068033 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:48.068013    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/b46a706b-7a33-4098-afbb-4f864a378bac-hub\") pod \"cluster-proxy-proxy-agent-5b5b47b457-6wkt2\" (UID: \"b46a706b-7a33-4098-afbb-4f864a378bac\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b5b47b457-6wkt2"
Jul 28 10:08:48.069402 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:48.069375    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/b46a706b-7a33-4098-afbb-4f864a378bac-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5b5b47b457-6wkt2\" (UID: \"b46a706b-7a33-4098-afbb-4f864a378bac\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b5b47b457-6wkt2"
Jul 28 10:08:48.069726 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:48.069693    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/87291d3d-abe4-45a1-93f7-851250382566-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5979786f78-glq6p\" (UID: \"87291d3d-abe4-45a1-93f7-851250382566\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5979786f78-glq6p"
Jul 28 10:08:48.085911 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:48.085862    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ptwb\" (UniqueName: \"kubernetes.io/projected/b46a706b-7a33-4098-afbb-4f864a378bac-kube-api-access-2ptwb\") pod \"cluster-proxy-proxy-agent-5b5b47b457-6wkt2\" (UID: \"b46a706b-7a33-4098-afbb-4f864a378bac\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b5b47b457-6wkt2"
Jul 28 10:08:48.095774 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:48.095746    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-srth9\" (UniqueName: \"kubernetes.io/projected/87291d3d-abe4-45a1-93f7-851250382566-kube-api-access-srth9\") pod \"managed-serviceaccount-addon-agent-5979786f78-glq6p\" (UID: \"87291d3d-abe4-45a1-93f7-851250382566\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5979786f78-glq6p"
Jul 28 10:08:48.218862 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:48.218774    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:48.247405 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:48.247354    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b5b47b457-6wkt2"
Jul 28 10:08:48.251850 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:48.251826    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5979786f78-glq6p"
Jul 28 10:08:48.425785 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:48.425742    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b5b47b457-6wkt2"]
Jul 28 10:08:48.437574 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:08:48.437530    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb46a706b_7a33_4098_afbb_4f864a378bac.slice/crio-c737e740b0958c97e9adc57ff013d089bc4f0c7c86e637b1583532c51738aaa7 WatchSource:0}: Error finding container c737e740b0958c97e9adc57ff013d089bc4f0c7c86e637b1583532c51738aaa7: Status 404 returned error can't find the container with id c737e740b0958c97e9adc57ff013d089bc4f0c7c86e637b1583532c51738aaa7
Jul 28 10:08:48.449184 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:48.449155    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:48.459197 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:48.459136    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5979786f78-glq6p"]
Jul 28 10:08:48.461666 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:08:48.461639    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod87291d3d_abe4_45a1_93f7_851250382566.slice/crio-96f26013661639c10c090a6db0bf069186dfcc1ce3aa3dcf10742ab93dd0daf2 WatchSource:0}: Error finding container 96f26013661639c10c090a6db0bf069186dfcc1ce3aa3dcf10742ab93dd0daf2: Status 404 returned error can't find the container with id 96f26013661639c10c090a6db0bf069186dfcc1ce3aa3dcf10742ab93dd0daf2
Jul 28 10:08:48.540702 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:48.540668    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5979786f78-glq6p" event={"ID":"87291d3d-abe4-45a1-93f7-851250382566","Type":"ContainerStarted","Data":"96f26013661639c10c090a6db0bf069186dfcc1ce3aa3dcf10742ab93dd0daf2"}
Jul 28 10:08:48.542718 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:48.542685    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mtcd6" event={"ID":"3fd0cb5a-eb68-45ec-89f2-df8fcf761def","Type":"ContainerStarted","Data":"de0714a8e0b1e216e0bcedb5ffa141028ce563aaff14f832e423763bfada1b0f"}
Jul 28 10:08:48.543822 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:48.543796    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b5b47b457-6wkt2" event={"ID":"b46a706b-7a33-4098-afbb-4f864a378bac","Type":"ContainerStarted","Data":"c737e740b0958c97e9adc57ff013d089bc4f0c7c86e637b1583532c51738aaa7"}
Jul 28 10:08:49.513532 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:49.513496    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-x9gk6"
Jul 28 10:08:51.484463 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:51.484421    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:51.555594 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:51.555556    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:51.896828 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:51.896788    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2s4l\" (UniqueName: \"kubernetes.io/projected/92436f2b-eb78-4e84-ab81-be7d1102f5ff-kube-api-access-h2s4l\") pod \"network-check-target-4msch\" (UID: \"92436f2b-eb78-4e84-ab81-be7d1102f5ff\") " pod="openshift-network-diagnostics/network-check-target-4msch"
Jul 28 10:08:51.897027 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:51.896848    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1fb2411c-4da6-44e6-882e-4f36fb1a26ea-original-pull-secret\") pod \"global-pull-secret-syncer-bffsx\" (UID: \"1fb2411c-4da6-44e6-882e-4f36fb1a26ea\") " pod="kube-system/global-pull-secret-syncer-bffsx"
Jul 28 10:08:51.900634 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:51.900608    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 28 10:08:51.900776 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:51.900686    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 28 10:08:51.910844 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:51.910812    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1fb2411c-4da6-44e6-882e-4f36fb1a26ea-original-pull-secret\") pod \"global-pull-secret-syncer-bffsx\" (UID: \"1fb2411c-4da6-44e6-882e-4f36fb1a26ea\") " pod="kube-system/global-pull-secret-syncer-bffsx"
Jul 28 10:08:51.912492 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:51.912472    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 28 10:08:51.920840 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:51.920807    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2s4l\" (UniqueName: \"kubernetes.io/projected/92436f2b-eb78-4e84-ab81-be7d1102f5ff-kube-api-access-h2s4l\") pod \"network-check-target-4msch\" (UID: \"92436f2b-eb78-4e84-ab81-be7d1102f5ff\") " pod="openshift-network-diagnostics/network-check-target-4msch"
Jul 28 10:08:51.997823 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:51.997786    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d67ab7ea-ff97-432b-b10a-df89bb204785-metrics-certs\") pod \"network-metrics-daemon-vtxgv\" (UID: \"d67ab7ea-ff97-432b-b10a-df89bb204785\") " pod="openshift-multus/network-metrics-daemon-vtxgv"
Jul 28 10:08:52.000857 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:52.000832    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 28 10:08:52.010843 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:52.010815    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d67ab7ea-ff97-432b-b10a-df89bb204785-metrics-certs\") pod \"network-metrics-daemon-vtxgv\" (UID: \"d67ab7ea-ff97-432b-b10a-df89bb204785\") " pod="openshift-multus/network-metrics-daemon-vtxgv"
Jul 28 10:08:52.184044 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:52.183963    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-xsq5w\""
Jul 28 10:08:52.189320 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:52.189290    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-l65kq\""
Jul 28 10:08:52.192198 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:52.192085    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4msch"
Jul 28 10:08:52.192198 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:52.192138    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bffsx"
Jul 28 10:08:52.198326 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:52.198299    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vtxgv"
Jul 28 10:08:52.644438 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:52.644400    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-bffsx"]
Jul 28 10:08:52.661765 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:52.661665    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-4msch"]
Jul 28 10:08:52.683110 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:52.683082    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-vtxgv"]
Jul 28 10:08:52.889032 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:08:52.888994    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fb2411c_4da6_44e6_882e_4f36fb1a26ea.slice/crio-4947ee05ce4d228ffc50582c37ebb5ffc59c65cd416786d05cfffc9ab57fddfb WatchSource:0}: Error finding container 4947ee05ce4d228ffc50582c37ebb5ffc59c65cd416786d05cfffc9ab57fddfb: Status 404 returned error can't find the container with id 4947ee05ce4d228ffc50582c37ebb5ffc59c65cd416786d05cfffc9ab57fddfb
Jul 28 10:08:52.889748 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:08:52.889647    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92436f2b_eb78_4e84_ab81_be7d1102f5ff.slice/crio-d74d8e6308eded30cb6b275b7fb94e720d54933fb2b9b86ed06c6312269a8f13 WatchSource:0}: Error finding container d74d8e6308eded30cb6b275b7fb94e720d54933fb2b9b86ed06c6312269a8f13: Status 404 returned error can't find the container with id d74d8e6308eded30cb6b275b7fb94e720d54933fb2b9b86ed06c6312269a8f13
Jul 28 10:08:52.890552 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:08:52.890512    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd67ab7ea_ff97_432b_b10a_df89bb204785.slice/crio-8a9705b201cf698ff4a4f34f15bf546ebfb6b5364c634c4dffbc4704ea2ac2b0 WatchSource:0}: Error finding container 8a9705b201cf698ff4a4f34f15bf546ebfb6b5364c634c4dffbc4704ea2ac2b0: Status 404 returned error can't find the container with id 8a9705b201cf698ff4a4f34f15bf546ebfb6b5364c634c4dffbc4704ea2ac2b0
Jul 28 10:08:53.348832 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:53.348794    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:53.447548 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:53.447498    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:53.532819 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:53.532782    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:53.577036 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:53.576999    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mtcd6" event={"ID":"3fd0cb5a-eb68-45ec-89f2-df8fcf761def","Type":"ContainerStarted","Data":"b89bb8318c11257411b3d616d91d2cb6ffa33216508843fd2043f0fbcd8e3640"}
Jul 28 10:08:53.578334 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:53.578305    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-4msch" event={"ID":"92436f2b-eb78-4e84-ab81-be7d1102f5ff","Type":"ContainerStarted","Data":"d74d8e6308eded30cb6b275b7fb94e720d54933fb2b9b86ed06c6312269a8f13"}
Jul 28 10:08:53.579642 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:53.579613    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vtxgv" event={"ID":"d67ab7ea-ff97-432b-b10a-df89bb204785","Type":"ContainerStarted","Data":"8a9705b201cf698ff4a4f34f15bf546ebfb6b5364c634c4dffbc4704ea2ac2b0"}
Jul 28 10:08:53.580844 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:53.580814    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-bffsx" event={"ID":"1fb2411c-4da6-44e6-882e-4f36fb1a26ea","Type":"ContainerStarted","Data":"4947ee05ce4d228ffc50582c37ebb5ffc59c65cd416786d05cfffc9ab57fddfb"}
Jul 28 10:08:53.582451 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:53.582427    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5979786f78-glq6p" event={"ID":"87291d3d-abe4-45a1-93f7-851250382566","Type":"ContainerStarted","Data":"84ad0da686cd096a28ca266f5e6ef56c693a6f137c529e94133dece07a3ba03f"}
Jul 28 10:08:53.600804 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:53.600713    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-mtcd6" podStartSLOduration=18.045586741 podStartE2EDuration="31.60069425s" totalImagesPullingTime="13.555107509s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 10:08:22 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:08:53.599518501 +0000 UTC m=+67.862914831" watchObservedRunningTime="2026-07-28 10:08:53.60069425 +0000 UTC m=+67.864090579"
Jul 28 10:08:53.620497 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:53.620446    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5979786f78-glq6p" podStartSLOduration=2.162953287 podStartE2EDuration="6.62042937s" totalImagesPullingTime="4.457476083s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:08:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:08:53.620172645 +0000 UTC m=+67.883568977" watchObservedRunningTime="2026-07-28 10:08:53.62042937 +0000 UTC m=+67.883825700"
Jul 28 10:08:53.732844 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:53.732809    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:53.834144 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:53.834114    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:53.933965 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:53.933879    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:54.282371 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:54.282296    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:55.597227 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:55.597192    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b5b47b457-6wkt2" event={"ID":"b46a706b-7a33-4098-afbb-4f864a378bac","Type":"ContainerStarted","Data":"3237dda5ecffe5da78dbffccd777524245aae6f41b38bb40a2798755b58a084f"}
Jul 28 10:08:55.597649 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:55.597247    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b5b47b457-6wkt2" event={"ID":"b46a706b-7a33-4098-afbb-4f864a378bac","Type":"ContainerStarted","Data":"ba7761de785f2294134d22191f3cc0f69dde8361c453c9242810a96d91f1b849"}
Jul 28 10:08:55.597649 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:55.597265    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b5b47b457-6wkt2" event={"ID":"b46a706b-7a33-4098-afbb-4f864a378bac","Type":"ContainerStarted","Data":"263048a10d862995a948c6f91f79bf94fd3d55ecc0f6f1373557c39f9592d968"}
Jul 28 10:08:55.621501 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:55.621436    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b5b47b457-6wkt2" podStartSLOduration=2.501555182 podStartE2EDuration="8.621412852s" totalImagesPullingTime="6.11985767s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:08:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:08:55.620332003 +0000 UTC m=+69.883728332" watchObservedRunningTime="2026-07-28 10:08:55.621412852 +0000 UTC m=+69.884809183"
Jul 28 10:08:55.900874 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:55.900764    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:55.986106 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:55.986070    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:56.423989 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:56.423954    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:56.521959 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:56.521924    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:56.603281 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:56.603241    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vtxgv" event={"ID":"d67ab7ea-ff97-432b-b10a-df89bb204785","Type":"ContainerStarted","Data":"2a4db5635d02f8f84e514243ef990f8d689b63dd0071dfd16d25c78c3233c60c"}
Jul 28 10:08:56.603759 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:56.603311    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vtxgv" event={"ID":"d67ab7ea-ff97-432b-b10a-df89bb204785","Type":"ContainerStarted","Data":"44680115e79af6202723e66f57406b703e25adf7ebe7d32713a286c907007a6a"}
Jul 28 10:08:56.755289 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:56.755205    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:56.857703 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:56.857675    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:57.106936 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:57.106889    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:57.214824 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:57.214784    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:58.548186 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:58.548152    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:58.634424 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:58.634401    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:59.330087 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:59.330055    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:59.519614 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:59.519586    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:59.619079 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:59.618991    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-4msch" event={"ID":"92436f2b-eb78-4e84-ab81-be7d1102f5ff","Type":"ContainerStarted","Data":"f7e31b30b0f509fab5bdc3833529873aee34594386a156c66abc483beb18e1b2"}
Jul 28 10:08:59.619451 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:59.619111    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-4msch"
Jul 28 10:08:59.620594 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:59.620574    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-bffsx" event={"ID":"1fb2411c-4da6-44e6-882e-4f36fb1a26ea","Type":"ContainerStarted","Data":"cdcb3c6b64a85dbc1f0f1cd4e12d2290517329fbfd57a06db9cdbfdfad59057d"}
Jul 28 10:08:59.639900 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:59.639852    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-vtxgv" podStartSLOduration=70.984445531 podStartE2EDuration="1m13.639835388s" totalImagesPullingTime="2.655389857s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:07:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:08:56.621410917 +0000 UTC m=+70.884807246" watchObservedRunningTime="2026-07-28 10:08:59.639835388 +0000 UTC m=+73.903231717"
Jul 28 10:08:59.640036 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:59.639930    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-4msch" podStartSLOduration=67.999273136 podStartE2EDuration="1m13.6399256s" totalImagesPullingTime="5.640652464s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:07:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:08:59.637454143 +0000 UTC m=+73.900850475" watchObservedRunningTime="2026-07-28 10:08:59.6399256 +0000 UTC m=+73.903321943"
Jul 28 10:08:59.657985 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:08:59.657939    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-bffsx" podStartSLOduration=68.014166373 podStartE2EDuration="1m13.657924593s" totalImagesPullingTime="5.64375822s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 10:07:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:08:59.657725484 +0000 UTC m=+73.921121812" watchObservedRunningTime="2026-07-28 10:08:59.657924593 +0000 UTC m=+73.921320967"
Jul 28 10:09:00.204561 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:09:00.204500    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:09:00.332830 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:09:00.332798    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:09:03.716559 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:09:03.716511    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:09:03.819486 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:09:03.819452    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:09:03.917127 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:09:03.917094    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:09:04.924715 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:09:04.924654    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:09:05.105182 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:09:05.105149    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:09:05.518297 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:09:05.518266    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:09:05.618129 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:09:05.618089    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:09:05.750184 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:09:05.750148    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:09:05.851415 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:09:05.851384    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:09:08.260811 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:09:08.260704    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:09:08.382386 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:09:08.382355    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:09:08.535295 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:09:08.535263    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:09:10.104663 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:09:10.104631    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:09:10.243872 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:09:10.243839    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:09:10.340131 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:09:10.340095    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:09:12.520556 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:09:12.520506    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:09:14.766553 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:09:14.766511    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:09:14.924007 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:09:14.923961    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:09:15.044309 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:09:15.044276    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:09:15.073330 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:09:15.073301    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-mttd2"]
Jul 28 10:09:15.093176 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:09:15.093136    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-mttd2"
Jul 28 10:09:15.096776 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:09:15.096753    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 28 10:09:15.096897 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:09:15.096754    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 28 10:09:15.097390 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:09:15.097372    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 28 10:09:15.097390 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:09:15.097379    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 28 10:09:15.097524 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:09:15.097389    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 28 10:09:15.097524 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:09:15.097407    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-xqkxz\""
Jul 28 10:09:15.100708 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:09:15.100692    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 28 10:09:15.161656 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:09:15.161621    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/382407f5-7905-49ae-b6df-e4458fd28bd6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mttd2\" (UID: \"382407f5-7905-49ae-b6df-e4458fd28bd6\") " pod="openshift-monitoring/node-exporter-mttd2"
Jul 28 10:09:15.161656 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:09:15.161656    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/382407f5-7905-49ae-b6df-e4458fd28bd6-node-exporter-accelerators-collector-config\") pod \"node-exporter-mttd2\" (UID: \"382407f5-7905-49ae-b6df-e4458fd28bd6\") " pod="openshift-monitoring/node-exporter-mttd2"
Jul 28 10:09:15.161930 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:09:15.161677    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/382407f5-7905-49ae-b6df-e4458fd28bd6-node-exporter-textfile\") pod \"node-exporter-mttd2\" (UID: \"382407f5-7905-49ae-b6df-e4458fd28bd6\") " pod="openshift-monitoring/node-exporter-mttd2"
Jul 28 10:09:15.161930 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:09:15.161747    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/382407f5-7905-49ae-b6df-e4458fd28bd6-sys\") pod \"node-exporter-mttd2\" (UID: \"382407f5-7905-49ae-b6df-e4458fd28bd6\") " pod="openshift-monitoring/node-exporter-mttd2"
Jul 28 10:09:15.161930 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:09:15.161797    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/382407f5-7905-49ae-b6df-e4458fd28bd6-node-exporter-tls\") pod \"node-exporter-mttd2\" (UID: \"382407f5-7905-49ae-b6df-e4458fd28bd6\") " pod="openshift-monitoring/node-exporter-mttd2"
Jul 28 10:09:15.161930 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:09:15.161870    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/382407f5-7905-49ae-b6df-e4458fd28bd6-node-exporter-wtmp\") pod \"node-exporter-mttd2\" (UID: \"382407f5-7905-49ae-b6df-e4458fd28bd6\") " pod="openshift-monitoring/node-exporter-mttd2"
Jul 28 10:09:15.161930 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:09:15.161903    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmd2n\" (UniqueName: \"kubernetes.io/projected/382407f5-7905-49ae-b6df-e4458fd28bd6-kube-api-access-pmd2n\") pod \"node-exporter-mttd2\" (UID: \"382407f5-7905-49ae-b6df-e4458fd28bd6\") " pod="openshift-monitoring/node-exporter-mttd2"
Jul 28 10:09:15.161930 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:09:15.161928    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/382407f5-7905-49ae-b6df-e4458fd28bd6-root\") pod \"node-exporter-mttd2\" (UID: \"382407f5-7905-49ae-b6df-e4458fd28bd6\") " pod="openshift-monitoring/node-exporter-mttd2"
Jul 28 10:09:15.162126 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:09:15.162016    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/382407f5-7905-49ae-b6df-e4458fd28bd6-metrics-client-ca\") pod \"node-exporter-mttd2\" (UID: \"382407f5-7905-49ae-b6df-e4458fd28bd6\") " pod="openshift-monitoring/node-exporter-mttd2"
Jul 28 10:09:15.171396 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:09:15.171346    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:09:15.263103 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:09:15.263052    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/382407f5-7905-49ae-b6df-e4458fd28bd6-root\") pod \"node-exporter-mttd2\" (UID: \"382407f5-7905-49ae-b6df-e4458fd28bd6\") " pod="openshift-monitoring/node-exporter-mttd2"
Jul 28 10:09:15.263103 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:09:15.263112    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/382407f5-7905-49ae-b6df-e4458fd28bd6-metrics-client-ca\") pod \"node-exporter-mttd2\" (UID: \"382407f5-7905-49ae-b6df-e4458fd28bd6\") " pod="openshift-monitoring/node-exporter-mttd2"
Jul 28 10:09:15.263321 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:09:15.263144    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/382407f5-7905-49ae-b6df-e4458fd28bd6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mttd2\" (UID: \"382407f5-7905-49ae-b6df-e4458fd28bd6\") " pod="openshift-monitoring/node-exporter-mttd2"
Jul 28 10:09:15.263321 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:09:15.263165    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/382407f5-7905-49ae-b6df-e4458fd28bd6-node-exporter-accelerators-collector-config\") pod \"node-exporter-mttd2\" (UID: \"382407f5-7905-49ae-b6df-e4458fd28bd6\") " pod="openshift-monitoring/node-exporter-mttd2"
Jul 28 10:09:15.263321 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:09:15.263174    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/382407f5-7905-49ae-b6df-e4458fd28bd6-root\") pod \"node-exporter-mttd2\" (UID: \"382407f5-7905-49ae-b6df-e4458fd28bd6\") " pod="openshift-monitoring/node-exporter-mttd2"
Jul 28 10:09:15.263321 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:09:15.263186    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/382407f5-7905-49ae-b6df-e4458fd28bd6-node-exporter-textfile\") pod \"node-exporter-mttd2\" (UID: \"382407f5-7905-49ae-b6df-e4458fd28bd6\") " pod="openshift-monitoring/node-exporter-mttd2"
Jul 28 10:09:15.263321 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:09:15.263247    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/382407f5-7905-49ae-b6df-e4458fd28bd6-sys\") pod \"node-exporter-mttd2\" (UID: \"382407f5-7905-49ae-b6df-e4458fd28bd6\") " pod="openshift-monitoring/node-exporter-mttd2"
Jul 28 10:09:15.263321 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:09:15.263266    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/382407f5-7905-49ae-b6df-e4458fd28bd6-node-exporter-tls\") pod \"node-exporter-mttd2\" (UID: \"382407f5-7905-49ae-b6df-e4458fd28bd6\") " pod="openshift-monitoring/node-exporter-mttd2"
Jul 28 10:09:15.263321 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:09:15.263314    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/382407f5-7905-49ae-b6df-e4458fd28bd6-node-exporter-wtmp\") pod \"node-exporter-mttd2\" (UID: \"382407f5-7905-49ae-b6df-e4458fd28bd6\") " pod="openshift-monitoring/node-exporter-mttd2"
Jul 28 10:09:15.263743 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:09:15.263339    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmd2n\" (UniqueName: \"kubernetes.io/projected/382407f5-7905-49ae-b6df-e4458fd28bd6-kube-api-access-pmd2n\") pod \"node-exporter-mttd2\" (UID: \"382407f5-7905-49ae-b6df-e4458fd28bd6\") " pod="openshift-monitoring/node-exporter-mttd2"
Jul 28 10:09:15.263743 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:09:15.263351    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/382407f5-7905-49ae-b6df-e4458fd28bd6-sys\") pod \"node-exporter-mttd2\" (UID: \"382407f5-7905-49ae-b6df-e4458fd28bd6\") " pod="openshift-monitoring/node-exporter-mttd2"
Jul 28 10:09:15.263743 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:09:15.263423    2579 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 28 10:09:15.263743 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:09:15.263477    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/382407f5-7905-49ae-b6df-e4458fd28bd6-node-exporter-wtmp\") pod \"node-exporter-mttd2\" (UID: \"382407f5-7905-49ae-b6df-e4458fd28bd6\") " pod="openshift-monitoring/node-exporter-mttd2"
Jul 28 10:09:15.263743 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:09:15.263518    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/382407f5-7905-49ae-b6df-e4458fd28bd6-node-exporter-tls podName:382407f5-7905-49ae-b6df-e4458fd28bd6 nodeName:}" failed. No retries permitted until 2026-07-28 10:09:15.763495198 +0000 UTC m=+90.026891519 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/382407f5-7905-49ae-b6df-e4458fd28bd6-node-exporter-tls") pod "node-exporter-mttd2" (UID: "382407f5-7905-49ae-b6df-e4458fd28bd6") : secret "node-exporter-tls" not found
Jul 28 10:09:15.263942 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:09:15.263746    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/382407f5-7905-49ae-b6df-e4458fd28bd6-node-exporter-textfile\") pod \"node-exporter-mttd2\" (UID: \"382407f5-7905-49ae-b6df-e4458fd28bd6\") " pod="openshift-monitoring/node-exporter-mttd2"
Jul 28 10:09:15.263942 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:09:15.263860    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/382407f5-7905-49ae-b6df-e4458fd28bd6-node-exporter-accelerators-collector-config\") pod \"node-exporter-mttd2\" (UID: \"382407f5-7905-49ae-b6df-e4458fd28bd6\") " pod="openshift-monitoring/node-exporter-mttd2"
Jul 28 10:09:15.263942 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:09:15.263877    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/382407f5-7905-49ae-b6df-e4458fd28bd6-metrics-client-ca\") pod \"node-exporter-mttd2\" (UID: \"382407f5-7905-49ae-b6df-e4458fd28bd6\") " pod="openshift-monitoring/node-exporter-mttd2"
Jul 28 10:09:15.265528 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:09:15.265509    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/382407f5-7905-49ae-b6df-e4458fd28bd6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mttd2\" (UID: \"382407f5-7905-49ae-b6df-e4458fd28bd6\") " pod="openshift-monitoring/node-exporter-mttd2"
Jul 28 10:09:15.298055 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:09:15.297972    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmd2n\" (UniqueName: \"kubernetes.io/projected/382407f5-7905-49ae-b6df-e4458fd28bd6-kube-api-access-pmd2n\") pod \"node-exporter-mttd2\" (UID: \"382407f5-7905-49ae-b6df-e4458fd28bd6\") " pod="openshift-monitoring/node-exporter-mttd2"
Jul 28 10:09:15.349012 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:09:15.348984    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:09:15.484291 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:09:15.484256    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:09:15.583671 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:09:15.583589    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:09:15.656696 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:09:15.656665    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:09:15.768250 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:09:15.768215    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/382407f5-7905-49ae-b6df-e4458fd28bd6-node-exporter-tls\") pod \"node-exporter-mttd2\" (UID: \"382407f5-7905-49ae-b6df-e4458fd28bd6\") " pod="openshift-monitoring/node-exporter-mttd2"
Jul 28 10:09:15.768670 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:09:15.768356    2579 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 28 10:09:15.768670 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:09:15.768418    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/382407f5-7905-49ae-b6df-e4458fd28bd6-node-exporter-tls podName:382407f5-7905-49ae-b6df-e4458fd28bd6 nodeName:}" failed. No retries permitted until 2026-07-28 10:09:16.768403553 +0000 UTC m=+91.031799860 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/382407f5-7905-49ae-b6df-e4458fd28bd6-node-exporter-tls") pod "node-exporter-mttd2" (UID: "382407f5-7905-49ae-b6df-e4458fd28bd6") : secret "node-exporter-tls" not found
Jul 28 10:09:15.962021 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:09:15.961939    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:09:16.064529 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:09:16.064494    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:09:16.164680 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:09:16.164652    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:09:16.373382 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:09:16.373352    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:09:16.777868 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:09:16.777773    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/382407f5-7905-49ae-b6df-e4458fd28bd6-node-exporter-tls\") pod \"node-exporter-mttd2\" (UID: \"382407f5-7905-49ae-b6df-e4458fd28bd6\") " pod="openshift-monitoring/node-exporter-mttd2"
Jul 28 10:09:16.780212 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:09:16.780183    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/382407f5-7905-49ae-b6df-e4458fd28bd6-node-exporter-tls\") pod \"node-exporter-mttd2\" (UID: \"382407f5-7905-49ae-b6df-e4458fd28bd6\") " pod="openshift-monitoring/node-exporter-mttd2"
Jul 28 10:09:16.902468 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:09:16.902414    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-mttd2"
Jul 28 10:09:16.912250 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:09:16.912226    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod382407f5_7905_49ae_b6df_e4458fd28bd6.slice/crio-872f9ab0e12e7aa563740b2c36ddec9a2d8b306b8678e6216fa74e6e0d1a540f WatchSource:0}: Error finding container 872f9ab0e12e7aa563740b2c36ddec9a2d8b306b8678e6216fa74e6e0d1a540f: Status 404 returned error can't find the container with id 872f9ab0e12e7aa563740b2c36ddec9a2d8b306b8678e6216fa74e6e0d1a540f
Jul 28 10:09:17.299036 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:09:17.298982    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:09:17.397226 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:09:17.397193    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:09:17.681879 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:09:17.681786    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mttd2" event={"ID":"382407f5-7905-49ae-b6df-e4458fd28bd6","Type":"ContainerStarted","Data":"872f9ab0e12e7aa563740b2c36ddec9a2d8b306b8678e6216fa74e6e0d1a540f"}
Jul 28 10:09:18.031423 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:09:18.031389    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:09:18.093307 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:09:18.093276    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:09:18.184494 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:09:18.184456    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:09:18.469621 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:09:18.469589    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:09:18.686896 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:09:18.686863    2579 generic.go:356] "Generic (PLEG): container finished" podID="382407f5-7905-49ae-b6df-e4458fd28bd6" containerID="3c34910954802e431bb017bc314c2bfcee0f8f72ff71f9cae6794cf43d8a811d" exitCode=0
Jul 28 10:09:18.687087 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:09:18.686949    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mttd2" event={"ID":"382407f5-7905-49ae-b6df-e4458fd28bd6","Type":"ContainerDied","Data":"3c34910954802e431bb017bc314c2bfcee0f8f72ff71f9cae6794cf43d8a811d"}
Jul 28 10:09:19.691948 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:09:19.691911    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mttd2" event={"ID":"382407f5-7905-49ae-b6df-e4458fd28bd6","Type":"ContainerStarted","Data":"40be850673c28b5f77a3031acd257e79ed0bcd09b2c9f79b20ed516fc020048f"}
Jul 28 10:09:19.692316 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:09:19.691972    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mttd2" event={"ID":"382407f5-7905-49ae-b6df-e4458fd28bd6","Type":"ContainerStarted","Data":"b5b6e2529f2715b80c8fb96d5cdcc7700aabdc0a2e06fb25e240af1d45d74f37"}
Jul 28 10:09:19.718471 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:09:19.718379    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-mttd2" podStartSLOduration=3.738005562 podStartE2EDuration="4.718358246s" totalImagesPullingTime="915.699178ms" totalInitContainerRuntime="64.653506ms" isStatefulPod=true podCreationTimestamp="2026-07-28 10:09:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:09:19.716811175 +0000 UTC m=+93.980207504" watchObservedRunningTime="2026-07-28 10:09:19.718358246 +0000 UTC m=+93.981754578"
Jul 28 10:09:19.759775 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:09:19.759737    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:09:19.850352 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:09:19.850314    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:09:19.955367 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:09:19.955292    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:09:20.056187 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:09:20.056152    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:09:20.196470 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:09:20.196437    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:09:21.806326 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:09:21.806284    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:09:30.317685 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:09:30.317653    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:09:30.417787 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:09:30.417756    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:09:30.517839 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:09:30.517801    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:09:30.627328 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:09:30.627251    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-4msch"
Jul 28 10:09:47.221834 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:09:47.221805    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:09:47.294643 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:09:47.294606    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:09:58.871452 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:09:58.871413    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:10:34.345199 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:10:34.345163    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:10:34.445231 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:10:34.445198    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:10:43.083213 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:10:43.083133    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:10:43.138147 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:10:43.138117    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:10:49.601217 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:10:49.601181    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:10:49.661860 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:10:49.661819    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:10:58.755061 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:10:58.755018    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:11:01.437980 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:11:01.437945    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:11:01.493809 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:11:01.493773    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:11:02.661968 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:11:02.661932    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:11:02.762994 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:11:02.762960    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:11:02.832559 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:11:02.832508    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:11:12.891412 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:11:12.891364    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:11:12.952755 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:11:12.952696    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:11:17.269106 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:11:17.269071    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:11:17.362028 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:11:17.361988    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:12:46.084472 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:12:46.084445    2579 kubelet.go:1648] "Image garbage collection succeeded"
Jul 28 10:14:16.020843 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:14:16.020734    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:14:17.248724 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:14:17.248683    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:14:17.395656 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:14:17.395624    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5js4z4"]
Jul 28 10:14:17.397755 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:14:17.397735    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5js4z4"
Jul 28 10:14:17.399987 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:14:17.399968    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 28 10:14:17.400100 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:14:17.400035    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-zvbm4\""
Jul 28 10:14:17.400522 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:14:17.400506    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 28 10:14:17.418721 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:14:17.418698    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5js4z4"]
Jul 28 10:14:17.467818 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:14:17.467788    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9eb573e6-abaf-4861-895a-7c678ab2715e-util\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5js4z4\" (UID: \"9eb573e6-abaf-4861-895a-7c678ab2715e\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5js4z4"
Jul 28 10:14:17.467979 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:14:17.467828    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8hg8\" (UniqueName: \"kubernetes.io/projected/9eb573e6-abaf-4861-895a-7c678ab2715e-kube-api-access-z8hg8\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5js4z4\" (UID: \"9eb573e6-abaf-4861-895a-7c678ab2715e\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5js4z4"
Jul 28 10:14:17.467979 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:14:17.467850    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9eb573e6-abaf-4861-895a-7c678ab2715e-bundle\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5js4z4\" (UID: \"9eb573e6-abaf-4861-895a-7c678ab2715e\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5js4z4"
Jul 28 10:14:17.569187 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:14:17.569149    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8hg8\" (UniqueName: \"kubernetes.io/projected/9eb573e6-abaf-4861-895a-7c678ab2715e-kube-api-access-z8hg8\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5js4z4\" (UID: \"9eb573e6-abaf-4861-895a-7c678ab2715e\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5js4z4"
Jul 28 10:14:17.569377 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:14:17.569194    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9eb573e6-abaf-4861-895a-7c678ab2715e-bundle\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5js4z4\" (UID: \"9eb573e6-abaf-4861-895a-7c678ab2715e\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5js4z4"
Jul 28 10:14:17.569377 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:14:17.569259    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9eb573e6-abaf-4861-895a-7c678ab2715e-util\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5js4z4\" (UID: \"9eb573e6-abaf-4861-895a-7c678ab2715e\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5js4z4"
Jul 28 10:14:17.569679 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:14:17.569653    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9eb573e6-abaf-4861-895a-7c678ab2715e-bundle\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5js4z4\" (UID: \"9eb573e6-abaf-4861-895a-7c678ab2715e\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5js4z4"
Jul 28 10:14:17.569762 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:14:17.569679    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9eb573e6-abaf-4861-895a-7c678ab2715e-util\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5js4z4\" (UID: \"9eb573e6-abaf-4861-895a-7c678ab2715e\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5js4z4"
Jul 28 10:14:17.578486 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:14:17.578460    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8hg8\" (UniqueName: \"kubernetes.io/projected/9eb573e6-abaf-4861-895a-7c678ab2715e-kube-api-access-z8hg8\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5js4z4\" (UID: \"9eb573e6-abaf-4861-895a-7c678ab2715e\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5js4z4"
Jul 28 10:14:17.706481 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:14:17.706437    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5js4z4"
Jul 28 10:14:17.828296 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:14:17.828215    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5js4z4"]
Jul 28 10:14:17.831347 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:14:17.831314    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9eb573e6_abaf_4861_895a_7c678ab2715e.slice/crio-b1612c8a4742ed667ea267627f344ec1e34eeac11c050f92873ac66f9a00ca93 WatchSource:0}: Error finding container b1612c8a4742ed667ea267627f344ec1e34eeac11c050f92873ac66f9a00ca93: Status 404 returned error can't find the container with id b1612c8a4742ed667ea267627f344ec1e34eeac11c050f92873ac66f9a00ca93
Jul 28 10:14:17.833258 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:14:17.833237    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 10:14:18.629524 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:14:18.629482    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5js4z4" event={"ID":"9eb573e6-abaf-4861-895a-7c678ab2715e","Type":"ContainerStarted","Data":"b1612c8a4742ed667ea267627f344ec1e34eeac11c050f92873ac66f9a00ca93"}
Jul 28 10:14:24.656190 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:14:24.656156    2579 generic.go:356] "Generic (PLEG): container finished" podID="9eb573e6-abaf-4861-895a-7c678ab2715e" containerID="ba1c6410b8578b0ce3412ddc10f9edb0a774747ef269b5fc547f32b238bf0390" exitCode=0
Jul 28 10:14:24.656593 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:14:24.656242    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5js4z4" event={"ID":"9eb573e6-abaf-4861-895a-7c678ab2715e","Type":"ContainerDied","Data":"ba1c6410b8578b0ce3412ddc10f9edb0a774747ef269b5fc547f32b238bf0390"}
Jul 28 10:14:27.668457 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:14:27.668369    2579 generic.go:356] "Generic (PLEG): container finished" podID="9eb573e6-abaf-4861-895a-7c678ab2715e" containerID="e3a56943437ff080ba06a96604cd21013e5258c25bf52609c5a9b81fcf3c30c2" exitCode=0
Jul 28 10:14:27.668457 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:14:27.668440    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5js4z4" event={"ID":"9eb573e6-abaf-4861-895a-7c678ab2715e","Type":"ContainerDied","Data":"e3a56943437ff080ba06a96604cd21013e5258c25bf52609c5a9b81fcf3c30c2"}
Jul 28 10:14:34.695801 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:14:34.695760    2579 generic.go:356] "Generic (PLEG): container finished" podID="9eb573e6-abaf-4861-895a-7c678ab2715e" containerID="b35b5e1b1c3498ab08557c215d336b4dd69823e0e54af614c9e77953d7d26687" exitCode=0
Jul 28 10:14:34.695801 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:14:34.695807    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5js4z4" event={"ID":"9eb573e6-abaf-4861-895a-7c678ab2715e","Type":"ContainerDied","Data":"b35b5e1b1c3498ab08557c215d336b4dd69823e0e54af614c9e77953d7d26687"}
Jul 28 10:14:35.819330 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:14:35.819306    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5js4z4"
Jul 28 10:14:35.909308 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:14:35.909263    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9eb573e6-abaf-4861-895a-7c678ab2715e-util\") pod \"9eb573e6-abaf-4861-895a-7c678ab2715e\" (UID: \"9eb573e6-abaf-4861-895a-7c678ab2715e\") "
Jul 28 10:14:35.909308 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:14:35.909307    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z8hg8\" (UniqueName: \"kubernetes.io/projected/9eb573e6-abaf-4861-895a-7c678ab2715e-kube-api-access-z8hg8\") pod \"9eb573e6-abaf-4861-895a-7c678ab2715e\" (UID: \"9eb573e6-abaf-4861-895a-7c678ab2715e\") "
Jul 28 10:14:35.909586 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:14:35.909345    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9eb573e6-abaf-4861-895a-7c678ab2715e-bundle\") pod \"9eb573e6-abaf-4861-895a-7c678ab2715e\" (UID: \"9eb573e6-abaf-4861-895a-7c678ab2715e\") "
Jul 28 10:14:35.910018 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:14:35.909979    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9eb573e6-abaf-4861-895a-7c678ab2715e-bundle" (OuterVolumeSpecName: "bundle") pod "9eb573e6-abaf-4861-895a-7c678ab2715e" (UID: "9eb573e6-abaf-4861-895a-7c678ab2715e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:14:35.911586 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:14:35.911564    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9eb573e6-abaf-4861-895a-7c678ab2715e-kube-api-access-z8hg8" (OuterVolumeSpecName: "kube-api-access-z8hg8") pod "9eb573e6-abaf-4861-895a-7c678ab2715e" (UID: "9eb573e6-abaf-4861-895a-7c678ab2715e"). InnerVolumeSpecName "kube-api-access-z8hg8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:14:35.914124 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:14:35.914098    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9eb573e6-abaf-4861-895a-7c678ab2715e-util" (OuterVolumeSpecName: "util") pod "9eb573e6-abaf-4861-895a-7c678ab2715e" (UID: "9eb573e6-abaf-4861-895a-7c678ab2715e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:14:36.010810 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:14:36.010727    2579 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9eb573e6-abaf-4861-895a-7c678ab2715e-util\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 10:14:36.010810 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:14:36.010755    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8hg8\" (UniqueName: \"kubernetes.io/projected/9eb573e6-abaf-4861-895a-7c678ab2715e-kube-api-access-z8hg8\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 10:14:36.010810 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:14:36.010765    2579 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9eb573e6-abaf-4861-895a-7c678ab2715e-bundle\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 10:14:36.705761 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:14:36.705732    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5js4z4" event={"ID":"9eb573e6-abaf-4861-895a-7c678ab2715e","Type":"ContainerDied","Data":"b1612c8a4742ed667ea267627f344ec1e34eeac11c050f92873ac66f9a00ca93"}
Jul 28 10:14:36.705921 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:14:36.705768    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5js4z4"
Jul 28 10:14:36.705921 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:14:36.705782    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b1612c8a4742ed667ea267627f344ec1e34eeac11c050f92873ac66f9a00ca93"
Jul 28 10:14:39.111143 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:14:39.111103    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:14:39.253337 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:14:39.253305    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:14:45.650613 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:14:45.650579    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:14:45.982313 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:14:45.982223    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:14:46.347720 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:14:46.347684    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:14:46.997285 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:14:46.997243    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:14:47.003628 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:14:47.003604    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-55557df8c-w7shf"]
Jul 28 10:14:47.003867 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:14:47.003839    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9eb573e6-abaf-4861-895a-7c678ab2715e" containerName="util"
Jul 28 10:14:47.003867 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:14:47.003852    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="9eb573e6-abaf-4861-895a-7c678ab2715e" containerName="util"
Jul 28 10:14:47.003867 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:14:47.003867    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9eb573e6-abaf-4861-895a-7c678ab2715e" containerName="pull"
Jul 28 10:14:47.004008 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:14:47.003873    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="9eb573e6-abaf-4861-895a-7c678ab2715e" containerName="pull"
Jul 28 10:14:47.004008 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:14:47.003882    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9eb573e6-abaf-4861-895a-7c678ab2715e" containerName="extract"
Jul 28 10:14:47.004008 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:14:47.003887    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="9eb573e6-abaf-4861-895a-7c678ab2715e" containerName="extract"
Jul 28 10:14:47.004008 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:14:47.003922    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="9eb573e6-abaf-4861-895a-7c678ab2715e" containerName="extract"
Jul 28 10:14:47.006796 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:14:47.006782    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-55557df8c-w7shf"
Jul 28 10:14:47.011260 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:14:47.011231    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Jul 28 10:14:47.011378 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:14:47.011236    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\""
Jul 28 10:14:47.011570 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:14:47.011553    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Jul 28 10:14:47.011896 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:14:47.011878    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Jul 28 10:14:47.012004 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:14:47.011913    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-ghrbj\""
Jul 28 10:14:47.012192 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:14:47.012177    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Jul 28 10:14:47.022715 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:14:47.022692    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-55557df8c-w7shf"]
Jul 28 10:14:47.188618 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:14:47.188585    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/885acae9-7b3d-4a9b-80dd-1bc6b5a4b0af-certificates\") pod \"keda-metrics-apiserver-55557df8c-w7shf\" (UID: \"885acae9-7b3d-4a9b-80dd-1bc6b5a4b0af\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-w7shf"
Jul 28 10:14:47.188799 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:14:47.188634    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/885acae9-7b3d-4a9b-80dd-1bc6b5a4b0af-temp-vol\") pod \"keda-metrics-apiserver-55557df8c-w7shf\" (UID: \"885acae9-7b3d-4a9b-80dd-1bc6b5a4b0af\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-w7shf"
Jul 28 10:14:47.188799 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:14:47.188655    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78wcc\" (UniqueName: \"kubernetes.io/projected/885acae9-7b3d-4a9b-80dd-1bc6b5a4b0af-kube-api-access-78wcc\") pod \"keda-metrics-apiserver-55557df8c-w7shf\" (UID: \"885acae9-7b3d-4a9b-80dd-1bc6b5a4b0af\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-w7shf"
Jul 28 10:14:47.289488 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:14:47.289461    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/885acae9-7b3d-4a9b-80dd-1bc6b5a4b0af-certificates\") pod \"keda-metrics-apiserver-55557df8c-w7shf\" (UID: \"885acae9-7b3d-4a9b-80dd-1bc6b5a4b0af\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-w7shf"
Jul 28 10:14:47.289669 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:14:47.289508    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/885acae9-7b3d-4a9b-80dd-1bc6b5a4b0af-temp-vol\") pod \"keda-metrics-apiserver-55557df8c-w7shf\" (UID: \"885acae9-7b3d-4a9b-80dd-1bc6b5a4b0af\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-w7shf"
Jul 28 10:14:47.289669 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:14:47.289528    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78wcc\" (UniqueName: \"kubernetes.io/projected/885acae9-7b3d-4a9b-80dd-1bc6b5a4b0af-kube-api-access-78wcc\") pod \"keda-metrics-apiserver-55557df8c-w7shf\" (UID: \"885acae9-7b3d-4a9b-80dd-1bc6b5a4b0af\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-w7shf"
Jul 28 10:14:47.289669 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:14:47.289645    2579 projected.go:266] Couldn't get secret openshift-keda/keda-metrics-apiserver-certs: secret "keda-metrics-apiserver-certs" not found
Jul 28 10:14:47.289793 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:14:47.289674    2579 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-55557df8c-w7shf: secret "keda-metrics-apiserver-certs" not found
Jul 28 10:14:47.289793 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:14:47.289741    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/885acae9-7b3d-4a9b-80dd-1bc6b5a4b0af-certificates podName:885acae9-7b3d-4a9b-80dd-1bc6b5a4b0af nodeName:}" failed. No retries permitted until 2026-07-28 10:14:47.789720841 +0000 UTC m=+422.053117155 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/885acae9-7b3d-4a9b-80dd-1bc6b5a4b0af-certificates") pod "keda-metrics-apiserver-55557df8c-w7shf" (UID: "885acae9-7b3d-4a9b-80dd-1bc6b5a4b0af") : secret "keda-metrics-apiserver-certs" not found
Jul 28 10:14:47.289953 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:14:47.289930    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/885acae9-7b3d-4a9b-80dd-1bc6b5a4b0af-temp-vol\") pod \"keda-metrics-apiserver-55557df8c-w7shf\" (UID: \"885acae9-7b3d-4a9b-80dd-1bc6b5a4b0af\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-w7shf"
Jul 28 10:14:47.306131 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:14:47.306095    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-78wcc\" (UniqueName: \"kubernetes.io/projected/885acae9-7b3d-4a9b-80dd-1bc6b5a4b0af-kube-api-access-78wcc\") pod \"keda-metrics-apiserver-55557df8c-w7shf\" (UID: \"885acae9-7b3d-4a9b-80dd-1bc6b5a4b0af\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-w7shf"
Jul 28 10:14:47.333336 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:14:47.333305    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-56f9798bb4-9sg4n"]
Jul 28 10:14:47.336453 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:14:47.336424    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:14:47.336602 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:14:47.336433    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-56f9798bb4-9sg4n"
Jul 28 10:14:47.339006 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:14:47.338989    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Jul 28 10:14:47.350022 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:14:47.350001    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-56f9798bb4-9sg4n"]
Jul 28 10:14:47.466928 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:14:47.466896    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:14:47.490921 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:14:47.490891    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/85da946d-72f2-4df7-adb3-31c4a6fb696e-certificates\") pod \"keda-admission-56f9798bb4-9sg4n\" (UID: \"85da946d-72f2-4df7-adb3-31c4a6fb696e\") " pod="openshift-keda/keda-admission-56f9798bb4-9sg4n"
Jul 28 10:14:47.491027 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:14:47.490948    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vvhk\" (UniqueName: \"kubernetes.io/projected/85da946d-72f2-4df7-adb3-31c4a6fb696e-kube-api-access-8vvhk\") pod \"keda-admission-56f9798bb4-9sg4n\" (UID: \"85da946d-72f2-4df7-adb3-31c4a6fb696e\") " pod="openshift-keda/keda-admission-56f9798bb4-9sg4n"
Jul 28 10:14:47.584448 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:14:47.584355    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:14:47.592318 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:14:47.592291    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vvhk\" (UniqueName: \"kubernetes.io/projected/85da946d-72f2-4df7-adb3-31c4a6fb696e-kube-api-access-8vvhk\") pod \"keda-admission-56f9798bb4-9sg4n\" (UID: \"85da946d-72f2-4df7-adb3-31c4a6fb696e\") " pod="openshift-keda/keda-admission-56f9798bb4-9sg4n"
Jul 28 10:14:47.592442 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:14:47.592348    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/85da946d-72f2-4df7-adb3-31c4a6fb696e-certificates\") pod \"keda-admission-56f9798bb4-9sg4n\" (UID: \"85da946d-72f2-4df7-adb3-31c4a6fb696e\") " pod="openshift-keda/keda-admission-56f9798bb4-9sg4n"
Jul 28 10:14:47.594800 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:14:47.594783    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/85da946d-72f2-4df7-adb3-31c4a6fb696e-certificates\") pod \"keda-admission-56f9798bb4-9sg4n\" (UID: \"85da946d-72f2-4df7-adb3-31c4a6fb696e\") " pod="openshift-keda/keda-admission-56f9798bb4-9sg4n"
Jul 28 10:14:47.601176 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:14:47.601158    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vvhk\" (UniqueName: \"kubernetes.io/projected/85da946d-72f2-4df7-adb3-31c4a6fb696e-kube-api-access-8vvhk\") pod \"keda-admission-56f9798bb4-9sg4n\" (UID: \"85da946d-72f2-4df7-adb3-31c4a6fb696e\") " pod="openshift-keda/keda-admission-56f9798bb4-9sg4n"
Jul 28 10:14:47.645863 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:14:47.645831    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-56f9798bb4-9sg4n"
Jul 28 10:14:47.775849 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:14:47.775818    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-56f9798bb4-9sg4n"]
Jul 28 10:14:47.778930 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:14:47.778906    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85da946d_72f2_4df7_adb3_31c4a6fb696e.slice/crio-7e21d163d748771ccaa6c6463bc09a2844d6809bf09bb3cd0b94ec2bce69a9b9 WatchSource:0}: Error finding container 7e21d163d748771ccaa6c6463bc09a2844d6809bf09bb3cd0b94ec2bce69a9b9: Status 404 returned error can't find the container with id 7e21d163d748771ccaa6c6463bc09a2844d6809bf09bb3cd0b94ec2bce69a9b9
Jul 28 10:14:47.779917 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:14:47.779891    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:14:47.793867 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:14:47.793848    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/885acae9-7b3d-4a9b-80dd-1bc6b5a4b0af-certificates\") pod \"keda-metrics-apiserver-55557df8c-w7shf\" (UID: \"885acae9-7b3d-4a9b-80dd-1bc6b5a4b0af\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-w7shf"
Jul 28 10:14:47.796166 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:14:47.796140    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/885acae9-7b3d-4a9b-80dd-1bc6b5a4b0af-certificates\") pod \"keda-metrics-apiserver-55557df8c-w7shf\" (UID: \"885acae9-7b3d-4a9b-80dd-1bc6b5a4b0af\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-w7shf"
Jul 28 10:14:47.916485 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:14:47.916392    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-55557df8c-w7shf"
Jul 28 10:14:48.018427 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:14:48.018389    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:14:48.061307 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:14:48.061283    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-55557df8c-w7shf"]
Jul 28 10:14:48.063724 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:14:48.063702    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod885acae9_7b3d_4a9b_80dd_1bc6b5a4b0af.slice/crio-22315170804fbc44fea73eabc99ab7e0a33b28b706d06649f6d385551a0c7516 WatchSource:0}: Error finding container 22315170804fbc44fea73eabc99ab7e0a33b28b706d06649f6d385551a0c7516: Status 404 returned error can't find the container with id 22315170804fbc44fea73eabc99ab7e0a33b28b706d06649f6d385551a0c7516
Jul 28 10:14:48.210981 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:14:48.210893    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:14:48.310199 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:14:48.310164    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:14:48.746641 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:14:48.746567    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-55557df8c-w7shf" event={"ID":"885acae9-7b3d-4a9b-80dd-1bc6b5a4b0af","Type":"ContainerStarted","Data":"22315170804fbc44fea73eabc99ab7e0a33b28b706d06649f6d385551a0c7516"}
Jul 28 10:14:48.748300 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:14:48.748273    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-56f9798bb4-9sg4n" event={"ID":"85da946d-72f2-4df7-adb3-31c4a6fb696e","Type":"ContainerStarted","Data":"7e21d163d748771ccaa6c6463bc09a2844d6809bf09bb3cd0b94ec2bce69a9b9"}
Jul 28 10:14:49.753850 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:14:49.753813    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-56f9798bb4-9sg4n" event={"ID":"85da946d-72f2-4df7-adb3-31c4a6fb696e","Type":"ContainerStarted","Data":"256a9bed41d507d9f53ba7655217880b1ede8aa1e00c438f74004632d98d48f5"}
Jul 28 10:14:49.754310 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:14:49.753963    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-56f9798bb4-9sg4n"
Jul 28 10:14:49.772616 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:14:49.772556    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-56f9798bb4-9sg4n" podStartSLOduration=0.993664693 podStartE2EDuration="2.772525855s" totalImagesPullingTime="1.778861162s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:14:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:14:49.772066224 +0000 UTC m=+424.035462568" watchObservedRunningTime="2026-07-28 10:14:49.772525855 +0000 UTC m=+424.035922183"
Jul 28 10:14:50.758883 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:14:50.758845    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-55557df8c-w7shf" event={"ID":"885acae9-7b3d-4a9b-80dd-1bc6b5a4b0af","Type":"ContainerStarted","Data":"1a36fe66339f552888aec54e2bc57674624b1bd248f817afa1394649d9003eff"}
Jul 28 10:14:50.759344 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:14:50.758951    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-55557df8c-w7shf"
Jul 28 10:14:50.787467 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:14:50.787410    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-55557df8c-w7shf" podStartSLOduration=2.310674348 podStartE2EDuration="4.787392752s" totalImagesPullingTime="2.476718404s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:14:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:14:50.785620543 +0000 UTC m=+425.049016872" watchObservedRunningTime="2026-07-28 10:14:50.787392752 +0000 UTC m=+425.050789080"
Jul 28 10:15:01.766430 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:15:01.766400    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-55557df8c-w7shf"
Jul 28 10:15:10.761232 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:15:10.761157    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-56f9798bb4-9sg4n"
Jul 28 10:15:48.264454 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:15:48.264412    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:15:48.478175 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:15:48.478137    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:15:48.597426 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:15:48.597395    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:15:48.951353 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:15:48.951258    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:15:49.142139 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:15:49.142109    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:15:56.917382 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:15:56.917344    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:15:58.788189 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:15:58.788152    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:15:58.887905 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:15:58.887873    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:15:58.948093 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:15:58.948053    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:16:03.151945 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:16:03.151911    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:16:03.223190 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:16:03.223157    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-6968749498-582ph"]
Jul 28 10:16:03.226196 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:16:03.226178    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-6968749498-582ph"
Jul 28 10:16:03.228737 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:16:03.228711    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Jul 28 10:16:03.228737 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:16:03.228725    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 28 10:16:03.229029 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:16:03.229014    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-tg8t6\""
Jul 28 10:16:03.229073 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:16:03.229028    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 28 10:16:03.238932 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:16:03.238905    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-6968749498-582ph"]
Jul 28 10:16:03.251243 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:16:03.251199    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:16:03.279122 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:16:03.279083    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-dk5fj"]
Jul 28 10:16:03.281980 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:16:03.281961    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-dk5fj"
Jul 28 10:16:03.284387 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:16:03.284369    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 28 10:16:03.284479 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:16:03.284397    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-6v48q\""
Jul 28 10:16:03.293153 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:16:03.293123    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-dk5fj"]
Jul 28 10:16:03.309091 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:16:03.309056    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blclc\" (UniqueName: \"kubernetes.io/projected/db5a09b7-b665-4729-afb4-a991f7998edf-kube-api-access-blclc\") pod \"kserve-controller-manager-6968749498-582ph\" (UID: \"db5a09b7-b665-4729-afb4-a991f7998edf\") " pod="kserve/kserve-controller-manager-6968749498-582ph"
Jul 28 10:16:03.309257 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:16:03.309118    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/db5a09b7-b665-4729-afb4-a991f7998edf-cert\") pod \"kserve-controller-manager-6968749498-582ph\" (UID: \"db5a09b7-b665-4729-afb4-a991f7998edf\") " pod="kserve/kserve-controller-manager-6968749498-582ph"
Jul 28 10:16:03.340366 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:16:03.340333    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:16:03.409799 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:16:03.409720    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hs6hf\" (UniqueName: \"kubernetes.io/projected/ec836d14-324b-4bfc-979f-dd6967b5603a-kube-api-access-hs6hf\") pod \"seaweedfs-64568bcd49-dk5fj\" (UID: \"ec836d14-324b-4bfc-979f-dd6967b5603a\") " pod="kserve/seaweedfs-64568bcd49-dk5fj"
Jul 28 10:16:03.409799 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:16:03.409766    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/db5a09b7-b665-4729-afb4-a991f7998edf-cert\") pod \"kserve-controller-manager-6968749498-582ph\" (UID: \"db5a09b7-b665-4729-afb4-a991f7998edf\") " pod="kserve/kserve-controller-manager-6968749498-582ph"
Jul 28 10:16:03.409946 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:16:03.409826    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/ec836d14-324b-4bfc-979f-dd6967b5603a-data\") pod \"seaweedfs-64568bcd49-dk5fj\" (UID: \"ec836d14-324b-4bfc-979f-dd6967b5603a\") " pod="kserve/seaweedfs-64568bcd49-dk5fj"
Jul 28 10:16:03.409946 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:16:03.409863    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blclc\" (UniqueName: \"kubernetes.io/projected/db5a09b7-b665-4729-afb4-a991f7998edf-kube-api-access-blclc\") pod \"kserve-controller-manager-6968749498-582ph\" (UID: \"db5a09b7-b665-4729-afb4-a991f7998edf\") " pod="kserve/kserve-controller-manager-6968749498-582ph"
Jul 28 10:16:03.412325 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:16:03.412299    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/db5a09b7-b665-4729-afb4-a991f7998edf-cert\") pod \"kserve-controller-manager-6968749498-582ph\" (UID: \"db5a09b7-b665-4729-afb4-a991f7998edf\") " pod="kserve/kserve-controller-manager-6968749498-582ph"
Jul 28 10:16:03.420252 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:16:03.420222    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-blclc\" (UniqueName: \"kubernetes.io/projected/db5a09b7-b665-4729-afb4-a991f7998edf-kube-api-access-blclc\") pod \"kserve-controller-manager-6968749498-582ph\" (UID: \"db5a09b7-b665-4729-afb4-a991f7998edf\") " pod="kserve/kserve-controller-manager-6968749498-582ph"
Jul 28 10:16:03.442024 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:16:03.441993    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:16:03.510688 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:16:03.510648    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hs6hf\" (UniqueName: \"kubernetes.io/projected/ec836d14-324b-4bfc-979f-dd6967b5603a-kube-api-access-hs6hf\") pod \"seaweedfs-64568bcd49-dk5fj\" (UID: \"ec836d14-324b-4bfc-979f-dd6967b5603a\") " pod="kserve/seaweedfs-64568bcd49-dk5fj"
Jul 28 10:16:03.510871 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:16:03.510719    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/ec836d14-324b-4bfc-979f-dd6967b5603a-data\") pod \"seaweedfs-64568bcd49-dk5fj\" (UID: \"ec836d14-324b-4bfc-979f-dd6967b5603a\") " pod="kserve/seaweedfs-64568bcd49-dk5fj"
Jul 28 10:16:03.511087 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:16:03.511070    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/ec836d14-324b-4bfc-979f-dd6967b5603a-data\") pod \"seaweedfs-64568bcd49-dk5fj\" (UID: \"ec836d14-324b-4bfc-979f-dd6967b5603a\") " pod="kserve/seaweedfs-64568bcd49-dk5fj"
Jul 28 10:16:03.519180 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:16:03.519146    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hs6hf\" (UniqueName: \"kubernetes.io/projected/ec836d14-324b-4bfc-979f-dd6967b5603a-kube-api-access-hs6hf\") pod \"seaweedfs-64568bcd49-dk5fj\" (UID: \"ec836d14-324b-4bfc-979f-dd6967b5603a\") " pod="kserve/seaweedfs-64568bcd49-dk5fj"
Jul 28 10:16:03.535936 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:16:03.535904    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-6968749498-582ph"
Jul 28 10:16:03.590573 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:16:03.590509    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-dk5fj"
Jul 28 10:16:03.689259 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:16:03.689225    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-6968749498-582ph"]
Jul 28 10:16:03.693098 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:16:03.693042    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb5a09b7_b665_4729_afb4_a991f7998edf.slice/crio-de3662b51d1c00fef32c256215e71b29b28af9520be488ee208c1d79097cd27d WatchSource:0}: Error finding container de3662b51d1c00fef32c256215e71b29b28af9520be488ee208c1d79097cd27d: Status 404 returned error can't find the container with id de3662b51d1c00fef32c256215e71b29b28af9520be488ee208c1d79097cd27d
Jul 28 10:16:03.730382 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:16:03.730346    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-dk5fj"]
Jul 28 10:16:03.733792 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:16:03.733761    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec836d14_324b_4bfc_979f_dd6967b5603a.slice/crio-516211ddc7f2663e03cb0cadfdb45b6cf79f11d8ab06d1057cf8efe9033bbb3d WatchSource:0}: Error finding container 516211ddc7f2663e03cb0cadfdb45b6cf79f11d8ab06d1057cf8efe9033bbb3d: Status 404 returned error can't find the container with id 516211ddc7f2663e03cb0cadfdb45b6cf79f11d8ab06d1057cf8efe9033bbb3d
Jul 28 10:16:03.935351 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:16:03.935213    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:16:03.996533 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:16:03.996477    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:16:04.006071 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:16:04.006038    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-6968749498-582ph" event={"ID":"db5a09b7-b665-4729-afb4-a991f7998edf","Type":"ContainerStarted","Data":"de3662b51d1c00fef32c256215e71b29b28af9520be488ee208c1d79097cd27d"}
Jul 28 10:16:04.007106 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:16:04.007087    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-dk5fj" event={"ID":"ec836d14-324b-4bfc-979f-dd6967b5603a","Type":"ContainerStarted","Data":"516211ddc7f2663e03cb0cadfdb45b6cf79f11d8ab06d1057cf8efe9033bbb3d"}
Jul 28 10:16:08.030219 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:16:08.030116    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-6968749498-582ph" event={"ID":"db5a09b7-b665-4729-afb4-a991f7998edf","Type":"ContainerStarted","Data":"dd4d9e4d8de9bc555d299466709d0455a7487b8eb74beafb72a475734224f4fa"}
Jul 28 10:16:08.030219 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:16:08.030211    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-6968749498-582ph"
Jul 28 10:16:08.031505 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:16:08.031484    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-dk5fj" event={"ID":"ec836d14-324b-4bfc-979f-dd6967b5603a","Type":"ContainerStarted","Data":"5d98591263595f23b3408bb1db1b6d7093e7f58f63a969ab0cf8130585bf97f5"}
Jul 28 10:16:08.031630 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:16:08.031575    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-dk5fj"
Jul 28 10:16:08.047113 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:16:08.047027    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-6968749498-582ph" podStartSLOduration=1.300613576 podStartE2EDuration="5.04700914s" totalImagesPullingTime="3.746395564s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:16:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:16:08.045706983 +0000 UTC m=+502.309103315" watchObservedRunningTime="2026-07-28 10:16:08.04700914 +0000 UTC m=+502.310405470"
Jul 28 10:16:08.061619 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:16:08.061578    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-dk5fj" podStartSLOduration=1.226256436 podStartE2EDuration="5.061563856s" totalImagesPullingTime="3.83530742s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:16:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:16:08.060619624 +0000 UTC m=+502.324015970" watchObservedRunningTime="2026-07-28 10:16:08.061563856 +0000 UTC m=+502.324960185"
Jul 28 10:16:14.037254 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:16:14.037227    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-dk5fj"
Jul 28 10:16:39.039877 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:16:39.039788    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-6968749498-582ph"
Jul 28 10:16:45.274458 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:16:45.274418    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:16:45.378782 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:16:45.378742    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:16:45.524117 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:16:45.524087    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mbzth"]
Jul 28 10:16:45.528743 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:16:45.528690    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mbzth"
Jul 28 10:16:45.530770 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:16:45.530741    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 28 10:16:45.530770 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:16:45.530765    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-zvbm4\""
Jul 28 10:16:45.530917 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:16:45.530739    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 28 10:16:45.535960 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:16:45.535578    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mbzth"]
Jul 28 10:16:45.612708 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:16:45.612668    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knx42\" (UniqueName: \"kubernetes.io/projected/f4539879-6afe-4510-afb6-250c0369ba1f-kube-api-access-knx42\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mbzth\" (UID: \"f4539879-6afe-4510-afb6-250c0369ba1f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mbzth"
Jul 28 10:16:45.612880 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:16:45.612774    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f4539879-6afe-4510-afb6-250c0369ba1f-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mbzth\" (UID: \"f4539879-6afe-4510-afb6-250c0369ba1f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mbzth"
Jul 28 10:16:45.612880 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:16:45.612809    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f4539879-6afe-4510-afb6-250c0369ba1f-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mbzth\" (UID: \"f4539879-6afe-4510-afb6-250c0369ba1f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mbzth"
Jul 28 10:16:45.713892 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:16:45.713852    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f4539879-6afe-4510-afb6-250c0369ba1f-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mbzth\" (UID: \"f4539879-6afe-4510-afb6-250c0369ba1f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mbzth"
Jul 28 10:16:45.714080 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:16:45.713903    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f4539879-6afe-4510-afb6-250c0369ba1f-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mbzth\" (UID: \"f4539879-6afe-4510-afb6-250c0369ba1f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mbzth"
Jul 28 10:16:45.714080 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:16:45.713933    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knx42\" (UniqueName: \"kubernetes.io/projected/f4539879-6afe-4510-afb6-250c0369ba1f-kube-api-access-knx42\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mbzth\" (UID: \"f4539879-6afe-4510-afb6-250c0369ba1f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mbzth"
Jul 28 10:16:45.714296 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:16:45.714273    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f4539879-6afe-4510-afb6-250c0369ba1f-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mbzth\" (UID: \"f4539879-6afe-4510-afb6-250c0369ba1f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mbzth"
Jul 28 10:16:45.714352 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:16:45.714297    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f4539879-6afe-4510-afb6-250c0369ba1f-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mbzth\" (UID: \"f4539879-6afe-4510-afb6-250c0369ba1f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mbzth"
Jul 28 10:16:45.722127 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:16:45.722094    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-knx42\" (UniqueName: \"kubernetes.io/projected/f4539879-6afe-4510-afb6-250c0369ba1f-kube-api-access-knx42\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mbzth\" (UID: \"f4539879-6afe-4510-afb6-250c0369ba1f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mbzth"
Jul 28 10:16:45.838189 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:16:45.838159    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mbzth"
Jul 28 10:16:45.962457 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:16:45.962416    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mbzth"]
Jul 28 10:16:45.966653 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:16:45.966626    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4539879_6afe_4510_afb6_250c0369ba1f.slice/crio-2dd02de92069f0c9ff6a640fa016df99822217f11921a95e48df4e359dcdb987 WatchSource:0}: Error finding container 2dd02de92069f0c9ff6a640fa016df99822217f11921a95e48df4e359dcdb987: Status 404 returned error can't find the container with id 2dd02de92069f0c9ff6a640fa016df99822217f11921a95e48df4e359dcdb987
Jul 28 10:16:46.162779 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:16:46.162690    2579 generic.go:356] "Generic (PLEG): container finished" podID="f4539879-6afe-4510-afb6-250c0369ba1f" containerID="48c8d07e212829828c42240428d778d94d2a116751628fec02f7c2caf85e78d3" exitCode=0
Jul 28 10:16:46.162779 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:16:46.162746    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mbzth" event={"ID":"f4539879-6afe-4510-afb6-250c0369ba1f","Type":"ContainerDied","Data":"48c8d07e212829828c42240428d778d94d2a116751628fec02f7c2caf85e78d3"}
Jul 28 10:16:46.162959 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:16:46.162783    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mbzth" event={"ID":"f4539879-6afe-4510-afb6-250c0369ba1f","Type":"ContainerStarted","Data":"2dd02de92069f0c9ff6a640fa016df99822217f11921a95e48df4e359dcdb987"}
Jul 28 10:16:54.192993 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:16:54.192951    2579 generic.go:356] "Generic (PLEG): container finished" podID="f4539879-6afe-4510-afb6-250c0369ba1f" containerID="026114e1289d14c515cc657adeac377715a86a220b24fad4d2f0189da59690f9" exitCode=0
Jul 28 10:16:54.193394 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:16:54.193041    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mbzth" event={"ID":"f4539879-6afe-4510-afb6-250c0369ba1f","Type":"ContainerDied","Data":"026114e1289d14c515cc657adeac377715a86a220b24fad4d2f0189da59690f9"}
Jul 28 10:16:55.198822 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:16:55.198791    2579 generic.go:356] "Generic (PLEG): container finished" podID="f4539879-6afe-4510-afb6-250c0369ba1f" containerID="f34623d227a4ff765cbf71fc956485c6ebcfe6206b5ec0c39251887d33b061be" exitCode=0
Jul 28 10:16:55.199328 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:16:55.198841    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mbzth" event={"ID":"f4539879-6afe-4510-afb6-250c0369ba1f","Type":"ContainerDied","Data":"f34623d227a4ff765cbf71fc956485c6ebcfe6206b5ec0c39251887d33b061be"}
Jul 28 10:16:56.338479 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:16:56.338451    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mbzth"
Jul 28 10:16:56.485712 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:16:56.485611    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f4539879-6afe-4510-afb6-250c0369ba1f-bundle\") pod \"f4539879-6afe-4510-afb6-250c0369ba1f\" (UID: \"f4539879-6afe-4510-afb6-250c0369ba1f\") "
Jul 28 10:16:56.485712 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:16:56.485669    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f4539879-6afe-4510-afb6-250c0369ba1f-util\") pod \"f4539879-6afe-4510-afb6-250c0369ba1f\" (UID: \"f4539879-6afe-4510-afb6-250c0369ba1f\") "
Jul 28 10:16:56.485712 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:16:56.485702    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-knx42\" (UniqueName: \"kubernetes.io/projected/f4539879-6afe-4510-afb6-250c0369ba1f-kube-api-access-knx42\") pod \"f4539879-6afe-4510-afb6-250c0369ba1f\" (UID: \"f4539879-6afe-4510-afb6-250c0369ba1f\") "
Jul 28 10:16:56.486326 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:16:56.486275    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4539879-6afe-4510-afb6-250c0369ba1f-bundle" (OuterVolumeSpecName: "bundle") pod "f4539879-6afe-4510-afb6-250c0369ba1f" (UID: "f4539879-6afe-4510-afb6-250c0369ba1f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:16:56.487731 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:16:56.487706    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4539879-6afe-4510-afb6-250c0369ba1f-kube-api-access-knx42" (OuterVolumeSpecName: "kube-api-access-knx42") pod "f4539879-6afe-4510-afb6-250c0369ba1f" (UID: "f4539879-6afe-4510-afb6-250c0369ba1f"). InnerVolumeSpecName "kube-api-access-knx42". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:16:56.492068 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:16:56.492042    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4539879-6afe-4510-afb6-250c0369ba1f-util" (OuterVolumeSpecName: "util") pod "f4539879-6afe-4510-afb6-250c0369ba1f" (UID: "f4539879-6afe-4510-afb6-250c0369ba1f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:16:56.586632 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:16:56.586592    2579 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f4539879-6afe-4510-afb6-250c0369ba1f-util\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 10:16:56.586632 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:16:56.586629    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-knx42\" (UniqueName: \"kubernetes.io/projected/f4539879-6afe-4510-afb6-250c0369ba1f-kube-api-access-knx42\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 10:16:56.586632 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:16:56.586639    2579 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f4539879-6afe-4510-afb6-250c0369ba1f-bundle\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 10:16:57.206791 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:16:57.206765    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mbzth"
Jul 28 10:16:57.206791 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:16:57.206769    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mbzth" event={"ID":"f4539879-6afe-4510-afb6-250c0369ba1f","Type":"ContainerDied","Data":"2dd02de92069f0c9ff6a640fa016df99822217f11921a95e48df4e359dcdb987"}
Jul 28 10:16:57.206995 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:16:57.206809    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2dd02de92069f0c9ff6a640fa016df99822217f11921a95e48df4e359dcdb987"
Jul 28 10:17:01.479356 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:01.479321    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:17:02.207070 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:02.207037    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:17:05.083871 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:05.083824    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:17:05.142167 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:05.142133    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:17:06.845621 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:06.845588    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:17:06.941762 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:06.941726    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:17:08.866097 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:08.866054    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:17:09.250346 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:09.250251    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:17:10.087318 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:10.087280    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:17:10.188112 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:10.188070    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:17:11.908023 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:11.907987    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:17:12.006792 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:12.006758    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:17:12.107985 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:12.107952    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:17:13.096026 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:13.095990    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnwdw4"]
Jul 28 10:17:13.096402 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:13.096235    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4539879-6afe-4510-afb6-250c0369ba1f" containerName="pull"
Jul 28 10:17:13.096402 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:13.096245    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4539879-6afe-4510-afb6-250c0369ba1f" containerName="pull"
Jul 28 10:17:13.096402 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:13.096255    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4539879-6afe-4510-afb6-250c0369ba1f" containerName="extract"
Jul 28 10:17:13.096402 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:13.096260    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4539879-6afe-4510-afb6-250c0369ba1f" containerName="extract"
Jul 28 10:17:13.096402 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:13.096273    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4539879-6afe-4510-afb6-250c0369ba1f" containerName="util"
Jul 28 10:17:13.096402 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:13.096278    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4539879-6afe-4510-afb6-250c0369ba1f" containerName="util"
Jul 28 10:17:13.096402 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:13.096324    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f4539879-6afe-4510-afb6-250c0369ba1f" containerName="extract"
Jul 28 10:17:13.099102 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:13.099085    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnwdw4"
Jul 28 10:17:13.100976 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:13.100958    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 28 10:17:13.101378 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:13.101363    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 28 10:17:13.101378 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:13.101371    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-zvbm4\""
Jul 28 10:17:13.107305 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:13.107284    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnwdw4"]
Jul 28 10:17:13.201498 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:13.201462    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8p5kt\" (UniqueName: \"kubernetes.io/projected/041f071e-3cfc-49d4-97df-e27adb78ac5b-kube-api-access-8p5kt\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnwdw4\" (UID: \"041f071e-3cfc-49d4-97df-e27adb78ac5b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnwdw4"
Jul 28 10:17:13.201498 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:13.201503    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/041f071e-3cfc-49d4-97df-e27adb78ac5b-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnwdw4\" (UID: \"041f071e-3cfc-49d4-97df-e27adb78ac5b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnwdw4"
Jul 28 10:17:13.201692 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:13.201626    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/041f071e-3cfc-49d4-97df-e27adb78ac5b-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnwdw4\" (UID: \"041f071e-3cfc-49d4-97df-e27adb78ac5b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnwdw4"
Jul 28 10:17:13.302418 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:13.302376    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/041f071e-3cfc-49d4-97df-e27adb78ac5b-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnwdw4\" (UID: \"041f071e-3cfc-49d4-97df-e27adb78ac5b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnwdw4"
Jul 28 10:17:13.302418 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:13.302421    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8p5kt\" (UniqueName: \"kubernetes.io/projected/041f071e-3cfc-49d4-97df-e27adb78ac5b-kube-api-access-8p5kt\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnwdw4\" (UID: \"041f071e-3cfc-49d4-97df-e27adb78ac5b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnwdw4"
Jul 28 10:17:13.302646 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:13.302448    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/041f071e-3cfc-49d4-97df-e27adb78ac5b-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnwdw4\" (UID: \"041f071e-3cfc-49d4-97df-e27adb78ac5b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnwdw4"
Jul 28 10:17:13.302816 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:13.302795    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/041f071e-3cfc-49d4-97df-e27adb78ac5b-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnwdw4\" (UID: \"041f071e-3cfc-49d4-97df-e27adb78ac5b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnwdw4"
Jul 28 10:17:13.302850 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:13.302813    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/041f071e-3cfc-49d4-97df-e27adb78ac5b-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnwdw4\" (UID: \"041f071e-3cfc-49d4-97df-e27adb78ac5b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnwdw4"
Jul 28 10:17:13.311381 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:13.311347    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8p5kt\" (UniqueName: \"kubernetes.io/projected/041f071e-3cfc-49d4-97df-e27adb78ac5b-kube-api-access-8p5kt\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnwdw4\" (UID: \"041f071e-3cfc-49d4-97df-e27adb78ac5b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnwdw4"
Jul 28 10:17:13.408274 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:13.408187    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnwdw4"
Jul 28 10:17:13.535197 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:13.535054    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnwdw4"]
Jul 28 10:17:13.537802 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:17:13.537771    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod041f071e_3cfc_49d4_97df_e27adb78ac5b.slice/crio-38ddfe23bee322c10344ed9c93a6d05e957c69b8d882fc50d7a9c5d582ac8c4f WatchSource:0}: Error finding container 38ddfe23bee322c10344ed9c93a6d05e957c69b8d882fc50d7a9c5d582ac8c4f: Status 404 returned error can't find the container with id 38ddfe23bee322c10344ed9c93a6d05e957c69b8d882fc50d7a9c5d582ac8c4f
Jul 28 10:17:14.270944 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:14.270910    2579 generic.go:356] "Generic (PLEG): container finished" podID="041f071e-3cfc-49d4-97df-e27adb78ac5b" containerID="35d648801ebc0374c817972703b892b707c3ee2fa1378179582dd98964bbd06f" exitCode=0
Jul 28 10:17:14.274692 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:14.274671    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnwdw4" event={"ID":"041f071e-3cfc-49d4-97df-e27adb78ac5b","Type":"ContainerDied","Data":"35d648801ebc0374c817972703b892b707c3ee2fa1378179582dd98964bbd06f"}
Jul 28 10:17:14.274758 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:14.274711    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnwdw4" event={"ID":"041f071e-3cfc-49d4-97df-e27adb78ac5b","Type":"ContainerStarted","Data":"38ddfe23bee322c10344ed9c93a6d05e957c69b8d882fc50d7a9c5d582ac8c4f"}
Jul 28 10:17:15.587375 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:15.587317    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:17:15.688340 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:15.688302    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:17:17.286405 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:17.286376    2579 generic.go:356] "Generic (PLEG): container finished" podID="041f071e-3cfc-49d4-97df-e27adb78ac5b" containerID="371c6fe430e67b60b3ed307b227101118756d6dafc126467cd2bdbd7ea3500ff" exitCode=0
Jul 28 10:17:17.286864 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:17.286448    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnwdw4" event={"ID":"041f071e-3cfc-49d4-97df-e27adb78ac5b","Type":"ContainerDied","Data":"371c6fe430e67b60b3ed307b227101118756d6dafc126467cd2bdbd7ea3500ff"}
Jul 28 10:17:17.665784 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:17.665704    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:17:18.291226 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:18.291191    2579 generic.go:356] "Generic (PLEG): container finished" podID="041f071e-3cfc-49d4-97df-e27adb78ac5b" containerID="76b756c9ea1eb1096749f6d71deabf0b4fe01281e820b3528706a3df9157b73e" exitCode=0
Jul 28 10:17:18.291643 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:18.291277    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnwdw4" event={"ID":"041f071e-3cfc-49d4-97df-e27adb78ac5b","Type":"ContainerDied","Data":"76b756c9ea1eb1096749f6d71deabf0b4fe01281e820b3528706a3df9157b73e"}
Jul 28 10:17:19.415633 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:19.415605    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnwdw4"
Jul 28 10:17:19.448829 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:19.448802    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/041f071e-3cfc-49d4-97df-e27adb78ac5b-util\") pod \"041f071e-3cfc-49d4-97df-e27adb78ac5b\" (UID: \"041f071e-3cfc-49d4-97df-e27adb78ac5b\") "
Jul 28 10:17:19.448977 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:19.448836    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/041f071e-3cfc-49d4-97df-e27adb78ac5b-bundle\") pod \"041f071e-3cfc-49d4-97df-e27adb78ac5b\" (UID: \"041f071e-3cfc-49d4-97df-e27adb78ac5b\") "
Jul 28 10:17:19.448977 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:19.448881    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8p5kt\" (UniqueName: \"kubernetes.io/projected/041f071e-3cfc-49d4-97df-e27adb78ac5b-kube-api-access-8p5kt\") pod \"041f071e-3cfc-49d4-97df-e27adb78ac5b\" (UID: \"041f071e-3cfc-49d4-97df-e27adb78ac5b\") "
Jul 28 10:17:19.449227 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:19.449210    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/041f071e-3cfc-49d4-97df-e27adb78ac5b-bundle" (OuterVolumeSpecName: "bundle") pod "041f071e-3cfc-49d4-97df-e27adb78ac5b" (UID: "041f071e-3cfc-49d4-97df-e27adb78ac5b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:17:19.450947 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:19.450920    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/041f071e-3cfc-49d4-97df-e27adb78ac5b-kube-api-access-8p5kt" (OuterVolumeSpecName: "kube-api-access-8p5kt") pod "041f071e-3cfc-49d4-97df-e27adb78ac5b" (UID: "041f071e-3cfc-49d4-97df-e27adb78ac5b"). InnerVolumeSpecName "kube-api-access-8p5kt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:17:19.456264 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:19.456238    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/041f071e-3cfc-49d4-97df-e27adb78ac5b-util" (OuterVolumeSpecName: "util") pod "041f071e-3cfc-49d4-97df-e27adb78ac5b" (UID: "041f071e-3cfc-49d4-97df-e27adb78ac5b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:17:19.550184 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:19.550102    2579 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/041f071e-3cfc-49d4-97df-e27adb78ac5b-util\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 10:17:19.550184 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:19.550143    2579 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/041f071e-3cfc-49d4-97df-e27adb78ac5b-bundle\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 10:17:19.550184 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:19.550158    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8p5kt\" (UniqueName: \"kubernetes.io/projected/041f071e-3cfc-49d4-97df-e27adb78ac5b-kube-api-access-8p5kt\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 10:17:20.300450 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:20.300421    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnwdw4"
Jul 28 10:17:20.300450 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:20.300434    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnwdw4" event={"ID":"041f071e-3cfc-49d4-97df-e27adb78ac5b","Type":"ContainerDied","Data":"38ddfe23bee322c10344ed9c93a6d05e957c69b8d882fc50d7a9c5d582ac8c4f"}
Jul 28 10:17:20.300655 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:20.300499    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="38ddfe23bee322c10344ed9c93a6d05e957c69b8d882fc50d7a9c5d582ac8c4f"
Jul 28 10:17:23.693979 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:23.693945    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:17:23.793003 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:23.792962    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:17:35.627558 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:35.627506    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:17:40.636471 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:40.636440    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-w6tmc"]
Jul 28 10:17:40.636972 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:40.636832    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="041f071e-3cfc-49d4-97df-e27adb78ac5b" containerName="pull"
Jul 28 10:17:40.636972 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:40.636850    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="041f071e-3cfc-49d4-97df-e27adb78ac5b" containerName="pull"
Jul 28 10:17:40.636972 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:40.636872    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="041f071e-3cfc-49d4-97df-e27adb78ac5b" containerName="extract"
Jul 28 10:17:40.636972 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:40.636881    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="041f071e-3cfc-49d4-97df-e27adb78ac5b" containerName="extract"
Jul 28 10:17:40.636972 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:40.636894    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="041f071e-3cfc-49d4-97df-e27adb78ac5b" containerName="util"
Jul 28 10:17:40.636972 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:40.636902    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="041f071e-3cfc-49d4-97df-e27adb78ac5b" containerName="util"
Jul 28 10:17:40.636972 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:40.636958    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="041f071e-3cfc-49d4-97df-e27adb78ac5b" containerName="extract"
Jul 28 10:17:40.639660 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:40.639640    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-w6tmc"
Jul 28 10:17:40.641878 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:40.641857    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Jul 28 10:17:40.642075 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:40.642060    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Jul 28 10:17:40.642332 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:40.642314    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 28 10:17:40.642476 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:40.642451    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 28 10:17:40.642595 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:40.642520    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Jul 28 10:17:40.642678 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:40.642659    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-tzrkf\""
Jul 28 10:17:40.643040 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:40.643022    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 28 10:17:40.653187 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:40.653163    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-w6tmc"]
Jul 28 10:17:40.700156 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:40.700123    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/348441b2-cf7f-43e6-9f8a-1d45d48301d2-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-w6tmc\" (UID: \"348441b2-cf7f-43e6-9f8a-1d45d48301d2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-w6tmc"
Jul 28 10:17:40.700310 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:40.700164    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/348441b2-cf7f-43e6-9f8a-1d45d48301d2-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-w6tmc\" (UID: \"348441b2-cf7f-43e6-9f8a-1d45d48301d2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-w6tmc"
Jul 28 10:17:40.700310 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:40.700201    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7zzh\" (UniqueName: \"kubernetes.io/projected/348441b2-cf7f-43e6-9f8a-1d45d48301d2-kube-api-access-x7zzh\") pod \"istiod-openshift-gateway-5677ccb49b-w6tmc\" (UID: \"348441b2-cf7f-43e6-9f8a-1d45d48301d2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-w6tmc"
Jul 28 10:17:40.700310 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:40.700222    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/348441b2-cf7f-43e6-9f8a-1d45d48301d2-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-w6tmc\" (UID: \"348441b2-cf7f-43e6-9f8a-1d45d48301d2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-w6tmc"
Jul 28 10:17:40.700310 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:40.700243    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/348441b2-cf7f-43e6-9f8a-1d45d48301d2-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-w6tmc\" (UID: \"348441b2-cf7f-43e6-9f8a-1d45d48301d2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-w6tmc"
Jul 28 10:17:40.700310 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:40.700289    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/348441b2-cf7f-43e6-9f8a-1d45d48301d2-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-w6tmc\" (UID: \"348441b2-cf7f-43e6-9f8a-1d45d48301d2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-w6tmc"
Jul 28 10:17:40.700471 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:40.700322    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/348441b2-cf7f-43e6-9f8a-1d45d48301d2-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-w6tmc\" (UID: \"348441b2-cf7f-43e6-9f8a-1d45d48301d2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-w6tmc"
Jul 28 10:17:40.801286 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:40.801252    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/348441b2-cf7f-43e6-9f8a-1d45d48301d2-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-w6tmc\" (UID: \"348441b2-cf7f-43e6-9f8a-1d45d48301d2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-w6tmc"
Jul 28 10:17:40.801286 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:40.801290    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/348441b2-cf7f-43e6-9f8a-1d45d48301d2-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-w6tmc\" (UID: \"348441b2-cf7f-43e6-9f8a-1d45d48301d2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-w6tmc"
Jul 28 10:17:40.801560 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:40.801313    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7zzh\" (UniqueName: \"kubernetes.io/projected/348441b2-cf7f-43e6-9f8a-1d45d48301d2-kube-api-access-x7zzh\") pod \"istiod-openshift-gateway-5677ccb49b-w6tmc\" (UID: \"348441b2-cf7f-43e6-9f8a-1d45d48301d2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-w6tmc"
Jul 28 10:17:40.801560 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:40.801332    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/348441b2-cf7f-43e6-9f8a-1d45d48301d2-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-w6tmc\" (UID: \"348441b2-cf7f-43e6-9f8a-1d45d48301d2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-w6tmc"
Jul 28 10:17:40.801560 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:40.801350    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/348441b2-cf7f-43e6-9f8a-1d45d48301d2-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-w6tmc\" (UID: \"348441b2-cf7f-43e6-9f8a-1d45d48301d2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-w6tmc"
Jul 28 10:17:40.801560 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:40.801463    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/348441b2-cf7f-43e6-9f8a-1d45d48301d2-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-w6tmc\" (UID: \"348441b2-cf7f-43e6-9f8a-1d45d48301d2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-w6tmc"
Jul 28 10:17:40.801560 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:40.801504    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/348441b2-cf7f-43e6-9f8a-1d45d48301d2-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-w6tmc\" (UID: \"348441b2-cf7f-43e6-9f8a-1d45d48301d2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-w6tmc"
Jul 28 10:17:40.802136 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:40.802090    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/348441b2-cf7f-43e6-9f8a-1d45d48301d2-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-w6tmc\" (UID: \"348441b2-cf7f-43e6-9f8a-1d45d48301d2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-w6tmc"
Jul 28 10:17:40.803564 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:40.803528    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/348441b2-cf7f-43e6-9f8a-1d45d48301d2-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-w6tmc\" (UID: \"348441b2-cf7f-43e6-9f8a-1d45d48301d2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-w6tmc"
Jul 28 10:17:40.803868 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:40.803839    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/348441b2-cf7f-43e6-9f8a-1d45d48301d2-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-w6tmc\" (UID: \"348441b2-cf7f-43e6-9f8a-1d45d48301d2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-w6tmc"
Jul 28 10:17:40.804158 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:40.804139    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/348441b2-cf7f-43e6-9f8a-1d45d48301d2-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-w6tmc\" (UID: \"348441b2-cf7f-43e6-9f8a-1d45d48301d2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-w6tmc"
Jul 28 10:17:40.804200 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:40.804139    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/348441b2-cf7f-43e6-9f8a-1d45d48301d2-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-w6tmc\" (UID: \"348441b2-cf7f-43e6-9f8a-1d45d48301d2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-w6tmc"
Jul 28 10:17:40.810197 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:40.810172    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7zzh\" (UniqueName: \"kubernetes.io/projected/348441b2-cf7f-43e6-9f8a-1d45d48301d2-kube-api-access-x7zzh\") pod \"istiod-openshift-gateway-5677ccb49b-w6tmc\" (UID: \"348441b2-cf7f-43e6-9f8a-1d45d48301d2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-w6tmc"
Jul 28 10:17:40.810305 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:40.810286    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/348441b2-cf7f-43e6-9f8a-1d45d48301d2-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-w6tmc\" (UID: \"348441b2-cf7f-43e6-9f8a-1d45d48301d2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-w6tmc"
Jul 28 10:17:40.949370 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:40.949280    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-w6tmc"
Jul 28 10:17:41.098646 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:41.098615    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-w6tmc"]
Jul 28 10:17:41.102467 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:17:41.102432    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod348441b2_cf7f_43e6_9f8a_1d45d48301d2.slice/crio-1cb75149dbf1102d5347080bca320cc5c460318c791d0b9620b8b3622ca61b01 WatchSource:0}: Error finding container 1cb75149dbf1102d5347080bca320cc5c460318c791d0b9620b8b3622ca61b01: Status 404 returned error can't find the container with id 1cb75149dbf1102d5347080bca320cc5c460318c791d0b9620b8b3622ca61b01
Jul 28 10:17:41.143948 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:41.143917    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:17:41.201998 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:41.201901    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:17:41.376351 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:41.376319    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-w6tmc" event={"ID":"348441b2-cf7f-43e6-9f8a-1d45d48301d2","Type":"ContainerStarted","Data":"1cb75149dbf1102d5347080bca320cc5c460318c791d0b9620b8b3622ca61b01"}
Jul 28 10:17:41.675577 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:41.675511    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:17:43.347527 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:43.347485    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 28 10:17:43.347823 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:43.347574    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 28 10:17:44.084462 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:44.084433    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:17:44.183943 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:44.183903    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:17:44.388584 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:44.388480    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-w6tmc" event={"ID":"348441b2-cf7f-43e6-9f8a-1d45d48301d2","Type":"ContainerStarted","Data":"d4726778a1fbd86bd8dd94a3babe297b050c51661b4492688878ad5a708bba35"}
Jul 28 10:17:44.388584 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:44.388562    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-w6tmc"
Jul 28 10:17:44.390412 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:44.390387    2579 patch_prober.go:28] interesting pod/istiod-openshift-gateway-5677ccb49b-w6tmc container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 28 10:17:44.390525 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:44.390429    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-w6tmc" podUID="348441b2-cf7f-43e6-9f8a-1d45d48301d2" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 10:17:44.410131 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:44.410083    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-w6tmc" podStartSLOduration=2.167457002 podStartE2EDuration="4.410066982s" totalImagesPullingTime="2.24260998s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:17:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:17:44.408611812 +0000 UTC m=+598.672008141" watchObservedRunningTime="2026-07-28 10:17:44.410066982 +0000 UTC m=+598.673463311"
Jul 28 10:17:45.393425 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:45.393397    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-w6tmc"
Jul 28 10:17:46.623477 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:46.623441    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:17:46.750688 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:46.750656    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:17:47.000337 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:47.000254    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-zf8mf"]
Jul 28 10:17:47.003799 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:47.003775    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-zf8mf"
Jul 28 10:17:47.006525 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:47.006503    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-default-dockercfg-sdfhm\""
Jul 28 10:17:47.041635 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:47.041602    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-zf8mf"]
Jul 28 10:17:47.054043 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:47.054011    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/490ca628-de57-478e-b4b8-af249144d0d2-workload-certs\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-zf8mf\" (UID: \"490ca628-de57-478e-b4b8-af249144d0d2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-zf8mf"
Jul 28 10:17:47.054199 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:47.054049    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/490ca628-de57-478e-b4b8-af249144d0d2-istio-envoy\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-zf8mf\" (UID: \"490ca628-de57-478e-b4b8-af249144d0d2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-zf8mf"
Jul 28 10:17:47.054199 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:47.054075    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/490ca628-de57-478e-b4b8-af249144d0d2-credential-socket\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-zf8mf\" (UID: \"490ca628-de57-478e-b4b8-af249144d0d2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-zf8mf"
Jul 28 10:17:47.054199 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:47.054156    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drwj9\" (UniqueName: \"kubernetes.io/projected/490ca628-de57-478e-b4b8-af249144d0d2-kube-api-access-drwj9\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-zf8mf\" (UID: \"490ca628-de57-478e-b4b8-af249144d0d2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-zf8mf"
Jul 28 10:17:47.054199 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:47.054197    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/490ca628-de57-478e-b4b8-af249144d0d2-istio-token\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-zf8mf\" (UID: \"490ca628-de57-478e-b4b8-af249144d0d2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-zf8mf"
Jul 28 10:17:47.054330 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:47.054225    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/490ca628-de57-478e-b4b8-af249144d0d2-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-zf8mf\" (UID: \"490ca628-de57-478e-b4b8-af249144d0d2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-zf8mf"
Jul 28 10:17:47.054330 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:47.054243    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/490ca628-de57-478e-b4b8-af249144d0d2-workload-socket\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-zf8mf\" (UID: \"490ca628-de57-478e-b4b8-af249144d0d2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-zf8mf"
Jul 28 10:17:47.054330 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:47.054264    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/490ca628-de57-478e-b4b8-af249144d0d2-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-zf8mf\" (UID: \"490ca628-de57-478e-b4b8-af249144d0d2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-zf8mf"
Jul 28 10:17:47.054330 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:47.054283    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/490ca628-de57-478e-b4b8-af249144d0d2-istio-data\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-zf8mf\" (UID: \"490ca628-de57-478e-b4b8-af249144d0d2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-zf8mf"
Jul 28 10:17:47.070599 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:47.070566    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:17:47.154758 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:47.154721    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/490ca628-de57-478e-b4b8-af249144d0d2-istio-token\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-zf8mf\" (UID: \"490ca628-de57-478e-b4b8-af249144d0d2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-zf8mf"
Jul 28 10:17:47.154956 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:47.154771    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/490ca628-de57-478e-b4b8-af249144d0d2-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-zf8mf\" (UID: \"490ca628-de57-478e-b4b8-af249144d0d2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-zf8mf"
Jul 28 10:17:47.154956 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:47.154790    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/490ca628-de57-478e-b4b8-af249144d0d2-workload-socket\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-zf8mf\" (UID: \"490ca628-de57-478e-b4b8-af249144d0d2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-zf8mf"
Jul 28 10:17:47.154956 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:47.154818    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/490ca628-de57-478e-b4b8-af249144d0d2-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-zf8mf\" (UID: \"490ca628-de57-478e-b4b8-af249144d0d2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-zf8mf"
Jul 28 10:17:47.154956 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:47.154836    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/490ca628-de57-478e-b4b8-af249144d0d2-istio-data\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-zf8mf\" (UID: \"490ca628-de57-478e-b4b8-af249144d0d2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-zf8mf"
Jul 28 10:17:47.154956 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:47.154858    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/490ca628-de57-478e-b4b8-af249144d0d2-workload-certs\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-zf8mf\" (UID: \"490ca628-de57-478e-b4b8-af249144d0d2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-zf8mf"
Jul 28 10:17:47.154956 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:47.154886    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/490ca628-de57-478e-b4b8-af249144d0d2-istio-envoy\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-zf8mf\" (UID: \"490ca628-de57-478e-b4b8-af249144d0d2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-zf8mf"
Jul 28 10:17:47.154956 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:47.154925    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/490ca628-de57-478e-b4b8-af249144d0d2-credential-socket\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-zf8mf\" (UID: \"490ca628-de57-478e-b4b8-af249144d0d2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-zf8mf"
Jul 28 10:17:47.155317 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:47.154963    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drwj9\" (UniqueName: \"kubernetes.io/projected/490ca628-de57-478e-b4b8-af249144d0d2-kube-api-access-drwj9\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-zf8mf\" (UID: \"490ca628-de57-478e-b4b8-af249144d0d2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-zf8mf"
Jul 28 10:17:47.155374 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:47.155333    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/490ca628-de57-478e-b4b8-af249144d0d2-workload-socket\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-zf8mf\" (UID: \"490ca628-de57-478e-b4b8-af249144d0d2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-zf8mf"
Jul 28 10:17:47.155426 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:47.155388    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/490ca628-de57-478e-b4b8-af249144d0d2-workload-certs\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-zf8mf\" (UID: \"490ca628-de57-478e-b4b8-af249144d0d2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-zf8mf"
Jul 28 10:17:47.155485 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:47.155452    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/490ca628-de57-478e-b4b8-af249144d0d2-credential-socket\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-zf8mf\" (UID: \"490ca628-de57-478e-b4b8-af249144d0d2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-zf8mf"
Jul 28 10:17:47.155587 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:47.155565    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/490ca628-de57-478e-b4b8-af249144d0d2-istio-data\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-zf8mf\" (UID: \"490ca628-de57-478e-b4b8-af249144d0d2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-zf8mf"
Jul 28 10:17:47.155684 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:47.155664    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/490ca628-de57-478e-b4b8-af249144d0d2-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-zf8mf\" (UID: \"490ca628-de57-478e-b4b8-af249144d0d2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-zf8mf"
Jul 28 10:17:47.157299 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:47.157272    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/490ca628-de57-478e-b4b8-af249144d0d2-istio-envoy\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-zf8mf\" (UID: \"490ca628-de57-478e-b4b8-af249144d0d2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-zf8mf"
Jul 28 10:17:47.157393 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:47.157331    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/490ca628-de57-478e-b4b8-af249144d0d2-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-zf8mf\" (UID: \"490ca628-de57-478e-b4b8-af249144d0d2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-zf8mf"
Jul 28 10:17:47.166238 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:47.166207    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:17:47.168752 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:47.168732    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/490ca628-de57-478e-b4b8-af249144d0d2-istio-token\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-zf8mf\" (UID: \"490ca628-de57-478e-b4b8-af249144d0d2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-zf8mf"
Jul 28 10:17:47.169031 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:47.169011    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-drwj9\" (UniqueName: \"kubernetes.io/projected/490ca628-de57-478e-b4b8-af249144d0d2-kube-api-access-drwj9\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-zf8mf\" (UID: \"490ca628-de57-478e-b4b8-af249144d0d2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-zf8mf"
Jul 28 10:17:47.314451 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:47.314411    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-zf8mf"
Jul 28 10:17:47.449126 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:47.449102    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-zf8mf"]
Jul 28 10:17:47.450986 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:17:47.450961    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod490ca628_de57_478e_b4b8_af249144d0d2.slice/crio-946ce3bbe0bf73e02d64d75a49c08a4b7b1e1cb558ee6fcb9f7b9f3582145db1 WatchSource:0}: Error finding container 946ce3bbe0bf73e02d64d75a49c08a4b7b1e1cb558ee6fcb9f7b9f3582145db1: Status 404 returned error can't find the container with id 946ce3bbe0bf73e02d64d75a49c08a4b7b1e1cb558ee6fcb9f7b9f3582145db1
Jul 28 10:17:48.406126 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:48.406089    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-zf8mf" event={"ID":"490ca628-de57-478e-b4b8-af249144d0d2","Type":"ContainerStarted","Data":"946ce3bbe0bf73e02d64d75a49c08a4b7b1e1cb558ee6fcb9f7b9f3582145db1"}
Jul 28 10:17:49.061338 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:49.061288    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:17:49.116690 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:49.116617    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:17:50.361887 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:50.361850    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 28 10:17:50.362171 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:50.361924    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 28 10:17:50.362171 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:50.361949    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 28 10:17:50.535777 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:50.535708    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:17:51.420816 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:51.420785    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-zf8mf" event={"ID":"490ca628-de57-478e-b4b8-af249144d0d2","Type":"ContainerStarted","Data":"3a53df2c4d7c003e6071f0f9e2b7d9823bf54b06e5b2d6a65a649efc0211cccd"}
Jul 28 10:17:51.443516 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:51.443467    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-zf8mf" podStartSLOduration=2.534714954 podStartE2EDuration="5.443449721s" totalImagesPullingTime="2.908734767s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:17:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:17:51.441630709 +0000 UTC m=+605.705027039" watchObservedRunningTime="2026-07-28 10:17:51.443449721 +0000 UTC m=+605.706846050"
Jul 28 10:17:52.315228 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:52.315191    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-zf8mf"
Jul 28 10:17:52.320096 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:52.320072    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-zf8mf"
Jul 28 10:17:52.425252 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:52.425220    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-zf8mf"
Jul 28 10:17:52.426279 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:52.426261    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-zf8mf"
Jul 28 10:17:52.613132 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:52.613043    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:17:52.713965 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:52.713928    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:17:55.184505 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:55.184470    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0l44pb"]
Jul 28 10:17:55.187898 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:55.187883    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0l44pb"
Jul 28 10:17:55.190057 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:55.190036    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 28 10:17:55.190167 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:55.190099    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 28 10:17:55.190554 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:55.190522    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-zvbm4\""
Jul 28 10:17:55.198169 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:55.198143    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0l44pb"]
Jul 28 10:17:55.272532 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:55.272502    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cm44gq"]
Jul 28 10:17:55.275695 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:55.275678    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cm44gq"
Jul 28 10:17:55.285204 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:55.285181    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cm44gq"]
Jul 28 10:17:55.322904 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:55.322867    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4r4d\" (UniqueName: \"kubernetes.io/projected/229442e4-9743-47d3-a0c2-f7884aba718a-kube-api-access-z4r4d\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0l44pb\" (UID: \"229442e4-9743-47d3-a0c2-f7884aba718a\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0l44pb"
Jul 28 10:17:55.323067 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:55.322919    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/229442e4-9743-47d3-a0c2-f7884aba718a-bundle\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0l44pb\" (UID: \"229442e4-9743-47d3-a0c2-f7884aba718a\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0l44pb"
Jul 28 10:17:55.323067 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:55.322993    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/229442e4-9743-47d3-a0c2-f7884aba718a-util\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0l44pb\" (UID: \"229442e4-9743-47d3-a0c2-f7884aba718a\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0l44pb"
Jul 28 10:17:55.370105 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:55.370071    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053b6zz7"]
Jul 28 10:17:55.373207 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:55.373192    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053b6zz7"
Jul 28 10:17:55.381284 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:55.381258    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053b6zz7"]
Jul 28 10:17:55.424160 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:55.424125    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4r4d\" (UniqueName: \"kubernetes.io/projected/229442e4-9743-47d3-a0c2-f7884aba718a-kube-api-access-z4r4d\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0l44pb\" (UID: \"229442e4-9743-47d3-a0c2-f7884aba718a\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0l44pb"
Jul 28 10:17:55.424331 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:55.424175    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8e001cac-3592-4345-9c01-71281fa40e17-util\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cm44gq\" (UID: \"8e001cac-3592-4345-9c01-71281fa40e17\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cm44gq"
Jul 28 10:17:55.424331 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:55.424210    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8e001cac-3592-4345-9c01-71281fa40e17-bundle\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cm44gq\" (UID: \"8e001cac-3592-4345-9c01-71281fa40e17\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cm44gq"
Jul 28 10:17:55.424331 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:55.424228    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/229442e4-9743-47d3-a0c2-f7884aba718a-bundle\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0l44pb\" (UID: \"229442e4-9743-47d3-a0c2-f7884aba718a\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0l44pb"
Jul 28 10:17:55.424331 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:55.424259    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/229442e4-9743-47d3-a0c2-f7884aba718a-util\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0l44pb\" (UID: \"229442e4-9743-47d3-a0c2-f7884aba718a\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0l44pb"
Jul 28 10:17:55.424485 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:55.424322    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nx8nn\" (UniqueName: \"kubernetes.io/projected/8e001cac-3592-4345-9c01-71281fa40e17-kube-api-access-nx8nn\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cm44gq\" (UID: \"8e001cac-3592-4345-9c01-71281fa40e17\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cm44gq"
Jul 28 10:17:55.424660 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:55.424637    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/229442e4-9743-47d3-a0c2-f7884aba718a-util\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0l44pb\" (UID: \"229442e4-9743-47d3-a0c2-f7884aba718a\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0l44pb"
Jul 28 10:17:55.424706 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:55.424660    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/229442e4-9743-47d3-a0c2-f7884aba718a-bundle\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0l44pb\" (UID: \"229442e4-9743-47d3-a0c2-f7884aba718a\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0l44pb"
Jul 28 10:17:55.432652 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:55.432626    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4r4d\" (UniqueName: \"kubernetes.io/projected/229442e4-9743-47d3-a0c2-f7884aba718a-kube-api-access-z4r4d\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0l44pb\" (UID: \"229442e4-9743-47d3-a0c2-f7884aba718a\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0l44pb"
Jul 28 10:17:55.473420 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:55.473348    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbxh74d"]
Jul 28 10:17:55.476728 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:55.476712    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbxh74d"
Jul 28 10:17:55.485487 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:55.485459    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbxh74d"]
Jul 28 10:17:55.497673 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:55.497648    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0l44pb"
Jul 28 10:17:55.525507 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:55.525473    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nx8nn\" (UniqueName: \"kubernetes.io/projected/8e001cac-3592-4345-9c01-71281fa40e17-kube-api-access-nx8nn\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cm44gq\" (UID: \"8e001cac-3592-4345-9c01-71281fa40e17\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cm44gq"
Jul 28 10:17:55.525687 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:55.525520    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmv4n\" (UniqueName: \"kubernetes.io/projected/04b64005-14e2-4255-b633-4b07b4e81072-kube-api-access-lmv4n\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053b6zz7\" (UID: \"04b64005-14e2-4255-b633-4b07b4e81072\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053b6zz7"
Jul 28 10:17:55.525687 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:55.525589    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8e001cac-3592-4345-9c01-71281fa40e17-util\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cm44gq\" (UID: \"8e001cac-3592-4345-9c01-71281fa40e17\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cm44gq"
Jul 28 10:17:55.525687 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:55.525609    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8e001cac-3592-4345-9c01-71281fa40e17-bundle\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cm44gq\" (UID: \"8e001cac-3592-4345-9c01-71281fa40e17\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cm44gq"
Jul 28 10:17:55.525687 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:55.525632    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/04b64005-14e2-4255-b633-4b07b4e81072-bundle\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053b6zz7\" (UID: \"04b64005-14e2-4255-b633-4b07b4e81072\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053b6zz7"
Jul 28 10:17:55.525687 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:55.525677    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/04b64005-14e2-4255-b633-4b07b4e81072-util\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053b6zz7\" (UID: \"04b64005-14e2-4255-b633-4b07b4e81072\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053b6zz7"
Jul 28 10:17:55.526028 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:55.526009    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8e001cac-3592-4345-9c01-71281fa40e17-util\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cm44gq\" (UID: \"8e001cac-3592-4345-9c01-71281fa40e17\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cm44gq"
Jul 28 10:17:55.526083 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:55.526009    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8e001cac-3592-4345-9c01-71281fa40e17-bundle\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cm44gq\" (UID: \"8e001cac-3592-4345-9c01-71281fa40e17\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cm44gq"
Jul 28 10:17:55.534868 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:55.534836    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nx8nn\" (UniqueName: \"kubernetes.io/projected/8e001cac-3592-4345-9c01-71281fa40e17-kube-api-access-nx8nn\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cm44gq\" (UID: \"8e001cac-3592-4345-9c01-71281fa40e17\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cm44gq"
Jul 28 10:17:55.584994 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:55.584955    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cm44gq"
Jul 28 10:17:55.627083 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:55.626623    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/04b64005-14e2-4255-b633-4b07b4e81072-bundle\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053b6zz7\" (UID: \"04b64005-14e2-4255-b633-4b07b4e81072\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053b6zz7"
Jul 28 10:17:55.627083 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:55.626691    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9t98\" (UniqueName: \"kubernetes.io/projected/1ebe2cb6-00f8-49de-b24f-0f1f60dcdeac-kube-api-access-s9t98\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbxh74d\" (UID: \"1ebe2cb6-00f8-49de-b24f-0f1f60dcdeac\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbxh74d"
Jul 28 10:17:55.627083 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:55.626731    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1ebe2cb6-00f8-49de-b24f-0f1f60dcdeac-util\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbxh74d\" (UID: \"1ebe2cb6-00f8-49de-b24f-0f1f60dcdeac\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbxh74d"
Jul 28 10:17:55.627083 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:55.626762    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/04b64005-14e2-4255-b633-4b07b4e81072-util\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053b6zz7\" (UID: \"04b64005-14e2-4255-b633-4b07b4e81072\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053b6zz7"
Jul 28 10:17:55.627083 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:55.626807    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1ebe2cb6-00f8-49de-b24f-0f1f60dcdeac-bundle\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbxh74d\" (UID: \"1ebe2cb6-00f8-49de-b24f-0f1f60dcdeac\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbxh74d"
Jul 28 10:17:55.627083 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:55.626837    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmv4n\" (UniqueName: \"kubernetes.io/projected/04b64005-14e2-4255-b633-4b07b4e81072-kube-api-access-lmv4n\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053b6zz7\" (UID: \"04b64005-14e2-4255-b633-4b07b4e81072\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053b6zz7"
Jul 28 10:17:55.627464 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:55.627287    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/04b64005-14e2-4255-b633-4b07b4e81072-util\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053b6zz7\" (UID: \"04b64005-14e2-4255-b633-4b07b4e81072\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053b6zz7"
Jul 28 10:17:55.627464 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:55.627431    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/04b64005-14e2-4255-b633-4b07b4e81072-bundle\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053b6zz7\" (UID: \"04b64005-14e2-4255-b633-4b07b4e81072\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053b6zz7"
Jul 28 10:17:55.632737 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:55.632710    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0l44pb"]
Jul 28 10:17:55.634605 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:17:55.634576    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod229442e4_9743_47d3_a0c2_f7884aba718a.slice/crio-b6042b5da44466a4d1fe0efc6a0c05f9708420c83e311ab19531d5eb516d1d26 WatchSource:0}: Error finding container b6042b5da44466a4d1fe0efc6a0c05f9708420c83e311ab19531d5eb516d1d26: Status 404 returned error can't find the container with id b6042b5da44466a4d1fe0efc6a0c05f9708420c83e311ab19531d5eb516d1d26
Jul 28 10:17:55.635786 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:55.635660    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmv4n\" (UniqueName: \"kubernetes.io/projected/04b64005-14e2-4255-b633-4b07b4e81072-kube-api-access-lmv4n\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053b6zz7\" (UID: \"04b64005-14e2-4255-b633-4b07b4e81072\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053b6zz7"
Jul 28 10:17:55.682275 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:55.682243    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053b6zz7"
Jul 28 10:17:55.729233 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:55.727707    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1ebe2cb6-00f8-49de-b24f-0f1f60dcdeac-bundle\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbxh74d\" (UID: \"1ebe2cb6-00f8-49de-b24f-0f1f60dcdeac\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbxh74d"
Jul 28 10:17:55.729233 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:55.727831    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9t98\" (UniqueName: \"kubernetes.io/projected/1ebe2cb6-00f8-49de-b24f-0f1f60dcdeac-kube-api-access-s9t98\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbxh74d\" (UID: \"1ebe2cb6-00f8-49de-b24f-0f1f60dcdeac\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbxh74d"
Jul 28 10:17:55.729233 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:55.727873    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1ebe2cb6-00f8-49de-b24f-0f1f60dcdeac-util\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbxh74d\" (UID: \"1ebe2cb6-00f8-49de-b24f-0f1f60dcdeac\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbxh74d"
Jul 28 10:17:55.729233 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:55.728292    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1ebe2cb6-00f8-49de-b24f-0f1f60dcdeac-util\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbxh74d\" (UID: \"1ebe2cb6-00f8-49de-b24f-0f1f60dcdeac\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbxh74d"
Jul 28 10:17:55.729233 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:55.728609    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1ebe2cb6-00f8-49de-b24f-0f1f60dcdeac-bundle\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbxh74d\" (UID: \"1ebe2cb6-00f8-49de-b24f-0f1f60dcdeac\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbxh74d"
Jul 28 10:17:55.730777 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:55.730748    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cm44gq"]
Jul 28 10:17:55.733716 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:17:55.733691    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e001cac_3592_4345_9c01_71281fa40e17.slice/crio-af3f47ab5b2782e5d4dae520bc3ceaf18c57bd293c91578532cc14708c503586 WatchSource:0}: Error finding container af3f47ab5b2782e5d4dae520bc3ceaf18c57bd293c91578532cc14708c503586: Status 404 returned error can't find the container with id af3f47ab5b2782e5d4dae520bc3ceaf18c57bd293c91578532cc14708c503586
Jul 28 10:17:55.739288 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:55.739223    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9t98\" (UniqueName: \"kubernetes.io/projected/1ebe2cb6-00f8-49de-b24f-0f1f60dcdeac-kube-api-access-s9t98\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbxh74d\" (UID: \"1ebe2cb6-00f8-49de-b24f-0f1f60dcdeac\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbxh74d"
Jul 28 10:17:55.786827 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:55.786799    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbxh74d"
Jul 28 10:17:55.823303 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:55.823179    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053b6zz7"]
Jul 28 10:17:55.830924 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:17:55.830896    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04b64005_14e2_4255_b633_4b07b4e81072.slice/crio-53fc9a057eec1f74d30d1429b03a54099c0de6e856ffc7cb5daa55b87719f258 WatchSource:0}: Error finding container 53fc9a057eec1f74d30d1429b03a54099c0de6e856ffc7cb5daa55b87719f258: Status 404 returned error can't find the container with id 53fc9a057eec1f74d30d1429b03a54099c0de6e856ffc7cb5daa55b87719f258
Jul 28 10:17:55.934529 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:55.934493    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbxh74d"]
Jul 28 10:17:56.006489 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:17:56.006414    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ebe2cb6_00f8_49de_b24f_0f1f60dcdeac.slice/crio-d68dc4230d046282f65a57d7726228ac9665b7c752c3721dddab75da7ef0a437 WatchSource:0}: Error finding container d68dc4230d046282f65a57d7726228ac9665b7c752c3721dddab75da7ef0a437: Status 404 returned error can't find the container with id d68dc4230d046282f65a57d7726228ac9665b7c752c3721dddab75da7ef0a437
Jul 28 10:17:56.442461 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:56.442422    2579 generic.go:356] "Generic (PLEG): container finished" podID="229442e4-9743-47d3-a0c2-f7884aba718a" containerID="b14bcbed74800f16294011db8d53eb813fef30189eac8bcbc2bdc50233c3ebe4" exitCode=0
Jul 28 10:17:56.442919 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:56.442506    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0l44pb" event={"ID":"229442e4-9743-47d3-a0c2-f7884aba718a","Type":"ContainerDied","Data":"b14bcbed74800f16294011db8d53eb813fef30189eac8bcbc2bdc50233c3ebe4"}
Jul 28 10:17:56.442919 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:56.442580    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0l44pb" event={"ID":"229442e4-9743-47d3-a0c2-f7884aba718a","Type":"ContainerStarted","Data":"b6042b5da44466a4d1fe0efc6a0c05f9708420c83e311ab19531d5eb516d1d26"}
Jul 28 10:17:56.444266 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:56.444104    2579 generic.go:356] "Generic (PLEG): container finished" podID="1ebe2cb6-00f8-49de-b24f-0f1f60dcdeac" containerID="de1bd8971a3675750294c4965b92bd7033266f4b20345555458fd8d874416139" exitCode=0
Jul 28 10:17:56.444266 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:56.444132    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbxh74d" event={"ID":"1ebe2cb6-00f8-49de-b24f-0f1f60dcdeac","Type":"ContainerDied","Data":"de1bd8971a3675750294c4965b92bd7033266f4b20345555458fd8d874416139"}
Jul 28 10:17:56.444266 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:56.444186    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbxh74d" event={"ID":"1ebe2cb6-00f8-49de-b24f-0f1f60dcdeac","Type":"ContainerStarted","Data":"d68dc4230d046282f65a57d7726228ac9665b7c752c3721dddab75da7ef0a437"}
Jul 28 10:17:56.446577 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:56.446480    2579 generic.go:356] "Generic (PLEG): container finished" podID="8e001cac-3592-4345-9c01-71281fa40e17" containerID="db0966252f4a43b67a53f5912a296f866147ad7234215475b22a53ac8b2b0847" exitCode=0
Jul 28 10:17:56.446687 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:56.446624    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cm44gq" event={"ID":"8e001cac-3592-4345-9c01-71281fa40e17","Type":"ContainerDied","Data":"db0966252f4a43b67a53f5912a296f866147ad7234215475b22a53ac8b2b0847"}
Jul 28 10:17:56.446687 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:56.446664    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cm44gq" event={"ID":"8e001cac-3592-4345-9c01-71281fa40e17","Type":"ContainerStarted","Data":"af3f47ab5b2782e5d4dae520bc3ceaf18c57bd293c91578532cc14708c503586"}
Jul 28 10:17:56.448396 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:56.448375    2579 generic.go:356] "Generic (PLEG): container finished" podID="04b64005-14e2-4255-b633-4b07b4e81072" containerID="0398b3093e982e6a2b68f39b7be574407bdc51bf9e90da42728ecad3352779f1" exitCode=0
Jul 28 10:17:56.448475 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:56.448411    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053b6zz7" event={"ID":"04b64005-14e2-4255-b633-4b07b4e81072","Type":"ContainerDied","Data":"0398b3093e982e6a2b68f39b7be574407bdc51bf9e90da42728ecad3352779f1"}
Jul 28 10:17:56.448475 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:56.448437    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053b6zz7" event={"ID":"04b64005-14e2-4255-b633-4b07b4e81072","Type":"ContainerStarted","Data":"53fc9a057eec1f74d30d1429b03a54099c0de6e856ffc7cb5daa55b87719f258"}
Jul 28 10:17:57.453924 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:57.453895    2579 generic.go:356] "Generic (PLEG): container finished" podID="229442e4-9743-47d3-a0c2-f7884aba718a" containerID="7f625bd3f43f9c4dbed53bf60f48e8de845ee00f7f2d52eb4e72238539ae9ca1" exitCode=0
Jul 28 10:17:57.454342 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:57.453931    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0l44pb" event={"ID":"229442e4-9743-47d3-a0c2-f7884aba718a","Type":"ContainerDied","Data":"7f625bd3f43f9c4dbed53bf60f48e8de845ee00f7f2d52eb4e72238539ae9ca1"}
Jul 28 10:17:57.455769 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:57.455746    2579 generic.go:356] "Generic (PLEG): container finished" podID="1ebe2cb6-00f8-49de-b24f-0f1f60dcdeac" containerID="5e1a277c71f053ecb3f0b5682a44cd45a0a462489163660047951785e9cda3ef" exitCode=0
Jul 28 10:17:57.455884 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:57.455800    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbxh74d" event={"ID":"1ebe2cb6-00f8-49de-b24f-0f1f60dcdeac","Type":"ContainerDied","Data":"5e1a277c71f053ecb3f0b5682a44cd45a0a462489163660047951785e9cda3ef"}
Jul 28 10:17:57.457681 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:57.457662    2579 generic.go:356] "Generic (PLEG): container finished" podID="8e001cac-3592-4345-9c01-71281fa40e17" containerID="3b3fcc23ef90c1bce12234d2858d0c1ba2e1a79fcffb58e9331ef6ffae3e4b7d" exitCode=0
Jul 28 10:17:57.457773 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:57.457689    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cm44gq" event={"ID":"8e001cac-3592-4345-9c01-71281fa40e17","Type":"ContainerDied","Data":"3b3fcc23ef90c1bce12234d2858d0c1ba2e1a79fcffb58e9331ef6ffae3e4b7d"}
Jul 28 10:17:58.464115 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:58.464073    2579 generic.go:356] "Generic (PLEG): container finished" podID="229442e4-9743-47d3-a0c2-f7884aba718a" containerID="a9201d59b64384486b0f28920e83f82c34ab451ab31be07881c221114685e617" exitCode=0
Jul 28 10:17:58.464579 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:58.464130    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0l44pb" event={"ID":"229442e4-9743-47d3-a0c2-f7884aba718a","Type":"ContainerDied","Data":"a9201d59b64384486b0f28920e83f82c34ab451ab31be07881c221114685e617"}
Jul 28 10:17:58.466103 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:58.466082    2579 generic.go:356] "Generic (PLEG): container finished" podID="1ebe2cb6-00f8-49de-b24f-0f1f60dcdeac" containerID="279b5ba04577e9a9bc518711309ccb5abe2566a91b926b60be322228a0fb2043" exitCode=0
Jul 28 10:17:58.466218 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:58.466152    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbxh74d" event={"ID":"1ebe2cb6-00f8-49de-b24f-0f1f60dcdeac","Type":"ContainerDied","Data":"279b5ba04577e9a9bc518711309ccb5abe2566a91b926b60be322228a0fb2043"}
Jul 28 10:17:58.468216 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:58.468192    2579 generic.go:356] "Generic (PLEG): container finished" podID="8e001cac-3592-4345-9c01-71281fa40e17" containerID="5d352c0c7e2478e611964fb06a5b195e773b7fe8f9002aa2d5a3ce6c08429519" exitCode=0
Jul 28 10:17:58.468310 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:58.468256    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cm44gq" event={"ID":"8e001cac-3592-4345-9c01-71281fa40e17","Type":"ContainerDied","Data":"5d352c0c7e2478e611964fb06a5b195e773b7fe8f9002aa2d5a3ce6c08429519"}
Jul 28 10:17:59.474276 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:59.474243    2579 generic.go:356] "Generic (PLEG): container finished" podID="04b64005-14e2-4255-b633-4b07b4e81072" containerID="a92514baaafe199657f2a4c2f02468c88b633883971feb3299e8a6c24fc9245b" exitCode=0
Jul 28 10:17:59.474709 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:59.474314    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053b6zz7" event={"ID":"04b64005-14e2-4255-b633-4b07b4e81072","Type":"ContainerDied","Data":"a92514baaafe199657f2a4c2f02468c88b633883971feb3299e8a6c24fc9245b"}
Jul 28 10:17:59.624150 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:59.624120    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cm44gq"
Jul 28 10:17:59.657347 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:59.657317    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbxh74d"
Jul 28 10:17:59.715493 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:59.715468    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0l44pb"
Jul 28 10:17:59.764091 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:59.764012    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1ebe2cb6-00f8-49de-b24f-0f1f60dcdeac-util\") pod \"1ebe2cb6-00f8-49de-b24f-0f1f60dcdeac\" (UID: \"1ebe2cb6-00f8-49de-b24f-0f1f60dcdeac\") "
Jul 28 10:17:59.764091 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:59.764066    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nx8nn\" (UniqueName: \"kubernetes.io/projected/8e001cac-3592-4345-9c01-71281fa40e17-kube-api-access-nx8nn\") pod \"8e001cac-3592-4345-9c01-71281fa40e17\" (UID: \"8e001cac-3592-4345-9c01-71281fa40e17\") "
Jul 28 10:17:59.764332 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:59.764100    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8e001cac-3592-4345-9c01-71281fa40e17-bundle\") pod \"8e001cac-3592-4345-9c01-71281fa40e17\" (UID: \"8e001cac-3592-4345-9c01-71281fa40e17\") "
Jul 28 10:17:59.764332 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:59.764116    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s9t98\" (UniqueName: \"kubernetes.io/projected/1ebe2cb6-00f8-49de-b24f-0f1f60dcdeac-kube-api-access-s9t98\") pod \"1ebe2cb6-00f8-49de-b24f-0f1f60dcdeac\" (UID: \"1ebe2cb6-00f8-49de-b24f-0f1f60dcdeac\") "
Jul 28 10:17:59.764332 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:59.764141    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1ebe2cb6-00f8-49de-b24f-0f1f60dcdeac-bundle\") pod \"1ebe2cb6-00f8-49de-b24f-0f1f60dcdeac\" (UID: \"1ebe2cb6-00f8-49de-b24f-0f1f60dcdeac\") "
Jul 28 10:17:59.764332 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:59.764172    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8e001cac-3592-4345-9c01-71281fa40e17-util\") pod \"8e001cac-3592-4345-9c01-71281fa40e17\" (UID: \"8e001cac-3592-4345-9c01-71281fa40e17\") "
Jul 28 10:17:59.764720 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:59.764681    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ebe2cb6-00f8-49de-b24f-0f1f60dcdeac-bundle" (OuterVolumeSpecName: "bundle") pod "1ebe2cb6-00f8-49de-b24f-0f1f60dcdeac" (UID: "1ebe2cb6-00f8-49de-b24f-0f1f60dcdeac"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:17:59.764877 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:59.764736    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e001cac-3592-4345-9c01-71281fa40e17-bundle" (OuterVolumeSpecName: "bundle") pod "8e001cac-3592-4345-9c01-71281fa40e17" (UID: "8e001cac-3592-4345-9c01-71281fa40e17"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:17:59.766244 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:59.766217    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ebe2cb6-00f8-49de-b24f-0f1f60dcdeac-kube-api-access-s9t98" (OuterVolumeSpecName: "kube-api-access-s9t98") pod "1ebe2cb6-00f8-49de-b24f-0f1f60dcdeac" (UID: "1ebe2cb6-00f8-49de-b24f-0f1f60dcdeac"). InnerVolumeSpecName "kube-api-access-s9t98". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:17:59.766972 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:59.766937    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e001cac-3592-4345-9c01-71281fa40e17-kube-api-access-nx8nn" (OuterVolumeSpecName: "kube-api-access-nx8nn") pod "8e001cac-3592-4345-9c01-71281fa40e17" (UID: "8e001cac-3592-4345-9c01-71281fa40e17"). InnerVolumeSpecName "kube-api-access-nx8nn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:17:59.770518 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:59.770083    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ebe2cb6-00f8-49de-b24f-0f1f60dcdeac-util" (OuterVolumeSpecName: "util") pod "1ebe2cb6-00f8-49de-b24f-0f1f60dcdeac" (UID: "1ebe2cb6-00f8-49de-b24f-0f1f60dcdeac"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:17:59.771046 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:59.771023    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e001cac-3592-4345-9c01-71281fa40e17-util" (OuterVolumeSpecName: "util") pod "8e001cac-3592-4345-9c01-71281fa40e17" (UID: "8e001cac-3592-4345-9c01-71281fa40e17"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:17:59.864793 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:59.864756    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/229442e4-9743-47d3-a0c2-f7884aba718a-util\") pod \"229442e4-9743-47d3-a0c2-f7884aba718a\" (UID: \"229442e4-9743-47d3-a0c2-f7884aba718a\") "
Jul 28 10:17:59.865055 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:59.864851    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z4r4d\" (UniqueName: \"kubernetes.io/projected/229442e4-9743-47d3-a0c2-f7884aba718a-kube-api-access-z4r4d\") pod \"229442e4-9743-47d3-a0c2-f7884aba718a\" (UID: \"229442e4-9743-47d3-a0c2-f7884aba718a\") "
Jul 28 10:17:59.865055 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:59.864870    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/229442e4-9743-47d3-a0c2-f7884aba718a-bundle\") pod \"229442e4-9743-47d3-a0c2-f7884aba718a\" (UID: \"229442e4-9743-47d3-a0c2-f7884aba718a\") "
Jul 28 10:17:59.865055 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:59.864978    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nx8nn\" (UniqueName: \"kubernetes.io/projected/8e001cac-3592-4345-9c01-71281fa40e17-kube-api-access-nx8nn\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 10:17:59.865055 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:59.864988    2579 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8e001cac-3592-4345-9c01-71281fa40e17-bundle\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 10:17:59.865055 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:59.864997    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9t98\" (UniqueName: \"kubernetes.io/projected/1ebe2cb6-00f8-49de-b24f-0f1f60dcdeac-kube-api-access-s9t98\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 10:17:59.865055 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:59.865016    2579 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1ebe2cb6-00f8-49de-b24f-0f1f60dcdeac-bundle\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 10:17:59.865055 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:59.865027    2579 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8e001cac-3592-4345-9c01-71281fa40e17-util\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 10:17:59.865055 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:59.865039    2579 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1ebe2cb6-00f8-49de-b24f-0f1f60dcdeac-util\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 10:17:59.865510 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:59.865426    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/229442e4-9743-47d3-a0c2-f7884aba718a-bundle" (OuterVolumeSpecName: "bundle") pod "229442e4-9743-47d3-a0c2-f7884aba718a" (UID: "229442e4-9743-47d3-a0c2-f7884aba718a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:17:59.866909 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:59.866883    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/229442e4-9743-47d3-a0c2-f7884aba718a-kube-api-access-z4r4d" (OuterVolumeSpecName: "kube-api-access-z4r4d") pod "229442e4-9743-47d3-a0c2-f7884aba718a" (UID: "229442e4-9743-47d3-a0c2-f7884aba718a"). InnerVolumeSpecName "kube-api-access-z4r4d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:17:59.869962 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:59.869927    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/229442e4-9743-47d3-a0c2-f7884aba718a-util" (OuterVolumeSpecName: "util") pod "229442e4-9743-47d3-a0c2-f7884aba718a" (UID: "229442e4-9743-47d3-a0c2-f7884aba718a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:17:59.965944 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:59.965905    2579 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/229442e4-9743-47d3-a0c2-f7884aba718a-util\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 10:17:59.965944 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:59.965936    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4r4d\" (UniqueName: \"kubernetes.io/projected/229442e4-9743-47d3-a0c2-f7884aba718a-kube-api-access-z4r4d\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 10:17:59.965944 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:17:59.965951    2579 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/229442e4-9743-47d3-a0c2-f7884aba718a-bundle\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 10:18:00.479754 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:18:00.479716    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0l44pb" event={"ID":"229442e4-9743-47d3-a0c2-f7884aba718a","Type":"ContainerDied","Data":"b6042b5da44466a4d1fe0efc6a0c05f9708420c83e311ab19531d5eb516d1d26"}
Jul 28 10:18:00.480146 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:18:00.479767    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b6042b5da44466a4d1fe0efc6a0c05f9708420c83e311ab19531d5eb516d1d26"
Jul 28 10:18:00.480146 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:18:00.479729    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0l44pb"
Jul 28 10:18:00.481797 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:18:00.481670    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbxh74d"
Jul 28 10:18:00.481797 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:18:00.481687    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbxh74d" event={"ID":"1ebe2cb6-00f8-49de-b24f-0f1f60dcdeac","Type":"ContainerDied","Data":"d68dc4230d046282f65a57d7726228ac9665b7c752c3721dddab75da7ef0a437"}
Jul 28 10:18:00.481797 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:18:00.481728    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d68dc4230d046282f65a57d7726228ac9665b7c752c3721dddab75da7ef0a437"
Jul 28 10:18:00.483970 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:18:00.483941    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cm44gq" event={"ID":"8e001cac-3592-4345-9c01-71281fa40e17","Type":"ContainerDied","Data":"af3f47ab5b2782e5d4dae520bc3ceaf18c57bd293c91578532cc14708c503586"}
Jul 28 10:18:00.484088 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:18:00.483982    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="af3f47ab5b2782e5d4dae520bc3ceaf18c57bd293c91578532cc14708c503586"
Jul 28 10:18:00.484088 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:18:00.484038    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cm44gq"
Jul 28 10:18:00.486109 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:18:00.486085    2579 generic.go:356] "Generic (PLEG): container finished" podID="04b64005-14e2-4255-b633-4b07b4e81072" containerID="dcd2f0081f95ab94879c1bb8c0ed10d1a9338e2ea05ffcb15b81edcae7126ded" exitCode=0
Jul 28 10:18:00.486226 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:18:00.486115    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053b6zz7" event={"ID":"04b64005-14e2-4255-b633-4b07b4e81072","Type":"ContainerDied","Data":"dcd2f0081f95ab94879c1bb8c0ed10d1a9338e2ea05ffcb15b81edcae7126ded"}
Jul 28 10:18:01.618444 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:18:01.618416    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053b6zz7"
Jul 28 10:18:01.782365 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:18:01.782281    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lmv4n\" (UniqueName: \"kubernetes.io/projected/04b64005-14e2-4255-b633-4b07b4e81072-kube-api-access-lmv4n\") pod \"04b64005-14e2-4255-b633-4b07b4e81072\" (UID: \"04b64005-14e2-4255-b633-4b07b4e81072\") "
Jul 28 10:18:01.782365 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:18:01.782341    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/04b64005-14e2-4255-b633-4b07b4e81072-util\") pod \"04b64005-14e2-4255-b633-4b07b4e81072\" (UID: \"04b64005-14e2-4255-b633-4b07b4e81072\") "
Jul 28 10:18:01.782581 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:18:01.782413    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/04b64005-14e2-4255-b633-4b07b4e81072-bundle\") pod \"04b64005-14e2-4255-b633-4b07b4e81072\" (UID: \"04b64005-14e2-4255-b633-4b07b4e81072\") "
Jul 28 10:18:01.783208 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:18:01.783179    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04b64005-14e2-4255-b633-4b07b4e81072-bundle" (OuterVolumeSpecName: "bundle") pod "04b64005-14e2-4255-b633-4b07b4e81072" (UID: "04b64005-14e2-4255-b633-4b07b4e81072"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:18:01.784487 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:18:01.784464    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04b64005-14e2-4255-b633-4b07b4e81072-kube-api-access-lmv4n" (OuterVolumeSpecName: "kube-api-access-lmv4n") pod "04b64005-14e2-4255-b633-4b07b4e81072" (UID: "04b64005-14e2-4255-b633-4b07b4e81072"). InnerVolumeSpecName "kube-api-access-lmv4n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:18:01.789101 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:18:01.789075    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04b64005-14e2-4255-b633-4b07b4e81072-util" (OuterVolumeSpecName: "util") pod "04b64005-14e2-4255-b633-4b07b4e81072" (UID: "04b64005-14e2-4255-b633-4b07b4e81072"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:18:01.883236 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:18:01.883203    2579 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/04b64005-14e2-4255-b633-4b07b4e81072-bundle\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 10:18:01.883236 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:18:01.883229    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmv4n\" (UniqueName: \"kubernetes.io/projected/04b64005-14e2-4255-b633-4b07b4e81072-kube-api-access-lmv4n\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 10:18:01.883236 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:18:01.883241    2579 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/04b64005-14e2-4255-b633-4b07b4e81072-util\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 10:18:02.496348 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:18:02.496311    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053b6zz7" event={"ID":"04b64005-14e2-4255-b633-4b07b4e81072","Type":"ContainerDied","Data":"53fc9a057eec1f74d30d1429b03a54099c0de6e856ffc7cb5daa55b87719f258"}
Jul 28 10:18:02.496348 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:18:02.496349    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053b6zz7"
Jul 28 10:18:02.496562 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:18:02.496358    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="53fc9a057eec1f74d30d1429b03a54099c0de6e856ffc7cb5daa55b87719f258"
Jul 28 10:18:03.826205 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:18:03.826167    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:18:04.415722 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:18:04.415661    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:18:04.653171 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:18:04.653127    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:18:07.963133 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:18:07.963056    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-56fb6f6856-tlmfp"]
Jul 28 10:18:07.963484 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:18:07.963337    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04b64005-14e2-4255-b633-4b07b4e81072" containerName="extract"
Jul 28 10:18:07.963484 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:18:07.963349    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="04b64005-14e2-4255-b633-4b07b4e81072" containerName="extract"
Jul 28 10:18:07.963484 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:18:07.963358    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="229442e4-9743-47d3-a0c2-f7884aba718a" containerName="extract"
Jul 28 10:18:07.963484 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:18:07.963364    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="229442e4-9743-47d3-a0c2-f7884aba718a" containerName="extract"
Jul 28 10:18:07.963484 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:18:07.963372    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e001cac-3592-4345-9c01-71281fa40e17" containerName="util"
Jul 28 10:18:07.963484 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:18:07.963378    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e001cac-3592-4345-9c01-71281fa40e17" containerName="util"
Jul 28 10:18:07.963484 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:18:07.963383    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="229442e4-9743-47d3-a0c2-f7884aba718a" containerName="util"
Jul 28 10:18:07.963484 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:18:07.963388    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="229442e4-9743-47d3-a0c2-f7884aba718a" containerName="util"
Jul 28 10:18:07.963484 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:18:07.963394    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ebe2cb6-00f8-49de-b24f-0f1f60dcdeac" containerName="util"
Jul 28 10:18:07.963484 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:18:07.963401    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ebe2cb6-00f8-49de-b24f-0f1f60dcdeac" containerName="util"
Jul 28 10:18:07.963484 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:18:07.963411    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ebe2cb6-00f8-49de-b24f-0f1f60dcdeac" containerName="pull"
Jul 28 10:18:07.963484 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:18:07.963417    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ebe2cb6-00f8-49de-b24f-0f1f60dcdeac" containerName="pull"
Jul 28 10:18:07.963484 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:18:07.963425    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ebe2cb6-00f8-49de-b24f-0f1f60dcdeac" containerName="extract"
Jul 28 10:18:07.963484 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:18:07.963430    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ebe2cb6-00f8-49de-b24f-0f1f60dcdeac" containerName="extract"
Jul 28 10:18:07.963484 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:18:07.963440    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04b64005-14e2-4255-b633-4b07b4e81072" containerName="pull"
Jul 28 10:18:07.963484 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:18:07.963445    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="04b64005-14e2-4255-b633-4b07b4e81072" containerName="pull"
Jul 28 10:18:07.963484 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:18:07.963454    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e001cac-3592-4345-9c01-71281fa40e17" containerName="extract"
Jul 28 10:18:07.963484 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:18:07.963458    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e001cac-3592-4345-9c01-71281fa40e17" containerName="extract"
Jul 28 10:18:07.963484 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:18:07.963465    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e001cac-3592-4345-9c01-71281fa40e17" containerName="pull"
Jul 28 10:18:07.963484 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:18:07.963470    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e001cac-3592-4345-9c01-71281fa40e17" containerName="pull"
Jul 28 10:18:07.963484 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:18:07.963476    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04b64005-14e2-4255-b633-4b07b4e81072" containerName="util"
Jul 28 10:18:07.963484 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:18:07.963483    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="04b64005-14e2-4255-b633-4b07b4e81072" containerName="util"
Jul 28 10:18:07.963484 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:18:07.963493    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="229442e4-9743-47d3-a0c2-f7884aba718a" containerName="pull"
Jul 28 10:18:07.964215 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:18:07.963498    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="229442e4-9743-47d3-a0c2-f7884aba718a" containerName="pull"
Jul 28 10:18:07.964215 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:18:07.963552    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ebe2cb6-00f8-49de-b24f-0f1f60dcdeac" containerName="extract"
Jul 28 10:18:07.964215 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:18:07.963560    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="229442e4-9743-47d3-a0c2-f7884aba718a" containerName="extract"
Jul 28 10:18:07.964215 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:18:07.963569    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e001cac-3592-4345-9c01-71281fa40e17" containerName="extract"
Jul 28 10:18:07.964215 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:18:07.963577    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="04b64005-14e2-4255-b633-4b07b4e81072" containerName="extract"
Jul 28 10:18:08.022854 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:18:08.022816    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-56fb6f6856-tlmfp"]
Jul 28 10:18:08.022854 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:18:08.022845    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-56fb6f6856-tlmfp"
Jul 28 10:18:08.025200 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:18:08.025175    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 28 10:18:08.025312 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:18:08.025176    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 28 10:18:08.025580 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:18:08.025563    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-svjq6\""
Jul 28 10:18:08.135100 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:18:08.135062    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psc2l\" (UniqueName: \"kubernetes.io/projected/f1643f06-8d61-4ace-b0b1-f3082ae86210-kube-api-access-psc2l\") pod \"authorino-operator-56fb6f6856-tlmfp\" (UID: \"f1643f06-8d61-4ace-b0b1-f3082ae86210\") " pod="kuadrant-system/authorino-operator-56fb6f6856-tlmfp"
Jul 28 10:18:08.235836 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:18:08.235737    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psc2l\" (UniqueName: \"kubernetes.io/projected/f1643f06-8d61-4ace-b0b1-f3082ae86210-kube-api-access-psc2l\") pod \"authorino-operator-56fb6f6856-tlmfp\" (UID: \"f1643f06-8d61-4ace-b0b1-f3082ae86210\") " pod="kuadrant-system/authorino-operator-56fb6f6856-tlmfp"
Jul 28 10:18:08.259463 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:18:08.259424    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-psc2l\" (UniqueName: \"kubernetes.io/projected/f1643f06-8d61-4ace-b0b1-f3082ae86210-kube-api-access-psc2l\") pod \"authorino-operator-56fb6f6856-tlmfp\" (UID: \"f1643f06-8d61-4ace-b0b1-f3082ae86210\") " pod="kuadrant-system/authorino-operator-56fb6f6856-tlmfp"
Jul 28 10:18:08.306890 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:18:08.306858    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:18:08.332861 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:18:08.332830    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-56fb6f6856-tlmfp"
Jul 28 10:18:08.465634 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:18:08.465608    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-56fb6f6856-tlmfp"]
Jul 28 10:18:08.467425 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:18:08.467389    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1643f06_8d61_4ace_b0b1_f3082ae86210.slice/crio-2429a96a8edbfedf79ce34aced2c522e8f2e53594c14cdbb9fb46cf39b9d9df1 WatchSource:0}: Error finding container 2429a96a8edbfedf79ce34aced2c522e8f2e53594c14cdbb9fb46cf39b9d9df1: Status 404 returned error can't find the container with id 2429a96a8edbfedf79ce34aced2c522e8f2e53594c14cdbb9fb46cf39b9d9df1
Jul 28 10:18:08.521792 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:18:08.521701    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-56fb6f6856-tlmfp" event={"ID":"f1643f06-8d61-4ace-b0b1-f3082ae86210","Type":"ContainerStarted","Data":"2429a96a8edbfedf79ce34aced2c522e8f2e53594c14cdbb9fb46cf39b9d9df1"}
Jul 28 10:18:09.251200 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:18:09.251153    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:18:09.885088 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:18:09.885045    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:18:10.143417 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:18:10.143316    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:18:10.811275 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:18:10.811240    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-ccbdf686-8bdq9"]
Jul 28 10:18:10.814419 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:18:10.814396    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-ccbdf686-8bdq9"
Jul 28 10:18:10.816551 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:18:10.816526    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-8fvzd\""
Jul 28 10:18:10.816757 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:18:10.816742    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Jul 28 10:18:10.834218 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:18:10.834198    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-ccbdf686-8bdq9"]
Jul 28 10:18:10.955886 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:18:10.955855    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqr7z\" (UniqueName: \"kubernetes.io/projected/6b5d8a26-4656-441f-8bfa-7579a7956ed0-kube-api-access-hqr7z\") pod \"dns-operator-controller-manager-ccbdf686-8bdq9\" (UID: \"6b5d8a26-4656-441f-8bfa-7579a7956ed0\") " pod="kuadrant-system/dns-operator-controller-manager-ccbdf686-8bdq9"
Jul 28 10:18:11.056780 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:18:11.056672    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqr7z\" (UniqueName: \"kubernetes.io/projected/6b5d8a26-4656-441f-8bfa-7579a7956ed0-kube-api-access-hqr7z\") pod \"dns-operator-controller-manager-ccbdf686-8bdq9\" (UID: \"6b5d8a26-4656-441f-8bfa-7579a7956ed0\") " pod="kuadrant-system/dns-operator-controller-manager-ccbdf686-8bdq9"
Jul 28 10:18:11.066908 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:18:11.066812    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqr7z\" (UniqueName: \"kubernetes.io/projected/6b5d8a26-4656-441f-8bfa-7579a7956ed0-kube-api-access-hqr7z\") pod \"dns-operator-controller-manager-ccbdf686-8bdq9\" (UID: \"6b5d8a26-4656-441f-8bfa-7579a7956ed0\") " pod="kuadrant-system/dns-operator-controller-manager-ccbdf686-8bdq9"
Jul 28 10:18:11.124555 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:18:11.124429    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-ccbdf686-8bdq9"
Jul 28 10:18:11.296714 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:18:11.296462    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-ccbdf686-8bdq9"]
Jul 28 10:18:11.301846 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:18:11.301801    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b5d8a26_4656_441f_8bfa_7579a7956ed0.slice/crio-abf33e62b37e6090985307a923d8e101932afa7fb4e0f1d1ddfcf46d4fd0b3f1 WatchSource:0}: Error finding container abf33e62b37e6090985307a923d8e101932afa7fb4e0f1d1ddfcf46d4fd0b3f1: Status 404 returned error can't find the container with id abf33e62b37e6090985307a923d8e101932afa7fb4e0f1d1ddfcf46d4fd0b3f1
Jul 28 10:18:11.538400 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:18:11.538358    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-ccbdf686-8bdq9" event={"ID":"6b5d8a26-4656-441f-8bfa-7579a7956ed0","Type":"ContainerStarted","Data":"abf33e62b37e6090985307a923d8e101932afa7fb4e0f1d1ddfcf46d4fd0b3f1"}
Jul 28 10:18:11.886735 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:18:11.886650    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:18:12.032273 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:18:12.032237    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:18:12.147030 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:18:12.146920    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:18:13.551332 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:18:13.551293    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-56fb6f6856-tlmfp" event={"ID":"f1643f06-8d61-4ace-b0b1-f3082ae86210","Type":"ContainerStarted","Data":"7c648e0d885f5da6359ebded300595a2ae933844677e58dc2158d39a48750987"}
Jul 28 10:18:13.569601 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:18:13.569552    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-56fb6f6856-tlmfp" podStartSLOduration=2.369234736 podStartE2EDuration="6.569523068s" totalImagesPullingTime="4.200288332s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:18:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:18:13.567207633 +0000 UTC m=+627.830603965" watchObservedRunningTime="2026-07-28 10:18:13.569523068 +0000 UTC m=+627.832919397"
Jul 28 10:18:14.556804 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:18:14.556771    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-ccbdf686-8bdq9" event={"ID":"6b5d8a26-4656-441f-8bfa-7579a7956ed0","Type":"ContainerStarted","Data":"22185891add963530bb59aea2a9b4b18647bf9aea4408f319326d0911c3987fe"}
Jul 28 10:18:14.557211 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:18:14.556847    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-ccbdf686-8bdq9"
Jul 28 10:18:14.557211 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:18:14.557010    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-56fb6f6856-tlmfp"
Jul 28 10:18:14.575242 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:18:14.575192    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-ccbdf686-8bdq9" podStartSLOduration=2.301950053 podStartE2EDuration="4.575175571s" totalImagesPullingTime="2.273225518s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:18:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:18:14.573602164 +0000 UTC m=+628.836998492" watchObservedRunningTime="2026-07-28 10:18:14.575175571 +0000 UTC m=+628.838571900"
Jul 28 10:18:14.916710 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:18:14.916620    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-2z85v"]
Jul 28 10:18:14.919910 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:18:14.919892    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-2z85v"
Jul 28 10:18:14.922127 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:18:14.922105    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-6wlkm\""
Jul 28 10:18:14.931467 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:18:14.931441    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-2z85v"]
Jul 28 10:18:14.989980 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:18:14.989945    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czp65\" (UniqueName: \"kubernetes.io/projected/523cd2de-0f16-403c-be29-876f77b2b2b1-kube-api-access-czp65\") pod \"kuadrant-operator-controller-manager-66866f5fcc-2z85v\" (UID: \"523cd2de-0f16-403c-be29-876f77b2b2b1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-2z85v"
Jul 28 10:18:14.990163 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:18:14.989995    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/523cd2de-0f16-403c-be29-876f77b2b2b1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-66866f5fcc-2z85v\" (UID: \"523cd2de-0f16-403c-be29-876f77b2b2b1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-2z85v"
Jul 28 10:18:15.091001 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:18:15.090960    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czp65\" (UniqueName: \"kubernetes.io/projected/523cd2de-0f16-403c-be29-876f77b2b2b1-kube-api-access-czp65\") pod \"kuadrant-operator-controller-manager-66866f5fcc-2z85v\" (UID: \"523cd2de-0f16-403c-be29-876f77b2b2b1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-2z85v"
Jul 28 10:18:15.091157 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:18:15.091016    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/523cd2de-0f16-403c-be29-876f77b2b2b1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-66866f5fcc-2z85v\" (UID: \"523cd2de-0f16-403c-be29-876f77b2b2b1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-2z85v"
Jul 28 10:18:15.091346 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:18:15.091329    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/523cd2de-0f16-403c-be29-876f77b2b2b1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-66866f5fcc-2z85v\" (UID: \"523cd2de-0f16-403c-be29-876f77b2b2b1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-2z85v"
Jul 28 10:18:15.100161 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:18:15.100127    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-czp65\" (UniqueName: \"kubernetes.io/projected/523cd2de-0f16-403c-be29-876f77b2b2b1-kube-api-access-czp65\") pod \"kuadrant-operator-controller-manager-66866f5fcc-2z85v\" (UID: \"523cd2de-0f16-403c-be29-876f77b2b2b1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-2z85v"
Jul 28 10:18:15.229768 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:18:15.229697    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-2z85v"
Jul 28 10:18:15.366737 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:18:15.366709    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-2z85v"]
Jul 28 10:18:15.368674 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:18:15.368644    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod523cd2de_0f16_403c_be29_876f77b2b2b1.slice/crio-b07c07764f2ef5aab79347b346aeb75eaf10feb22aea08fa2dea13f179f72143 WatchSource:0}: Error finding container b07c07764f2ef5aab79347b346aeb75eaf10feb22aea08fa2dea13f179f72143: Status 404 returned error can't find the container with id b07c07764f2ef5aab79347b346aeb75eaf10feb22aea08fa2dea13f179f72143
Jul 28 10:18:15.562367 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:18:15.562323    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-2z85v" event={"ID":"523cd2de-0f16-403c-be29-876f77b2b2b1","Type":"ContainerStarted","Data":"b07c07764f2ef5aab79347b346aeb75eaf10feb22aea08fa2dea13f179f72143"}
Jul 28 10:18:15.681769 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:18:15.681737    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:18:16.129265 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:18:16.129228    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:18:18.440800 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:18:18.440750    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:18:18.505214 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:18:18.505169    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:18:18.835335 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:18:18.835298    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:18:18.931174 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:18:18.931136    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:18:19.583997 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:18:19.583968    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-2z85v" event={"ID":"523cd2de-0f16-403c-be29-876f77b2b2b1","Type":"ContainerStarted","Data":"28fded3255d13838f10b02157fb3a398018bc1f5ba9ccaed8f2808ac74f24be3"}
Jul 28 10:18:19.584342 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:18:19.584205    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-2z85v"
Jul 28 10:18:19.605790 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:18:19.605739    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-2z85v" podStartSLOduration=1.816958485 podStartE2EDuration="5.605723622s" totalImagesPullingTime="3.788765137s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:18:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:18:19.602978604 +0000 UTC m=+633.866374944" watchObservedRunningTime="2026-07-28 10:18:19.605723622 +0000 UTC m=+633.869119950"
Jul 28 10:18:21.593094 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:18:21.593060    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:18:22.132079 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:18:22.132039    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:18:22.435925 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:18:22.435835    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:18:22.535555 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:18:22.535506    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:18:25.564383 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:18:25.564349    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-56fb6f6856-tlmfp"
Jul 28 10:18:25.564828 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:18:25.564419    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-ccbdf686-8bdq9"
Jul 28 10:18:28.227636 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:18:28.227600    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:18:28.588438 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:18:28.588402    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:18:28.649746 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:18:28.649711    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:18:30.590167 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:18:30.590132    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-2z85v"
Jul 28 10:18:33.492089 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:18:33.492052    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:18:33.559511 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:18:33.559474    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:18:35.626478 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:18:35.626441    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:18:35.691461 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:18:35.691432    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:18:39.057727 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:18:39.057688    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:18:39.125028 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:18:39.124975    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:18:39.666376 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:18:39.666334    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:18:39.727554 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:18:39.727512    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:18:42.055604 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:18:42.055573    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:18:42.156454 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:18:42.156400    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:18:48.661349 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:18:48.661312    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:18:48.762201 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:18:48.762165    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:18:49.256160 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:18:49.256121    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:18:49.318621 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:18:49.318587    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:18:49.874602 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:18:49.874565    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:18:49.947752 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:18:49.947706    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:18:56.868647 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:18:56.868600    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:18:56.969181 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:18:56.969145    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:19:05.285367 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:19:05.285329    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:19:05.297331 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:19:05.297300    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-pmv2p"]
Jul 28 10:19:05.302050 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:19:05.302031    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-pmv2p"
Jul 28 10:19:05.304332 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:19:05.304306    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 28 10:19:05.304453 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:19:05.304306    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-hvd92\""
Jul 28 10:19:05.310356 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:19:05.310334    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-pmv2p"]
Jul 28 10:19:05.390961 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:19:05.390923    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3c5f2713-9010-4625-ae17-6546784dc4f6-config-file\") pod \"limitador-limitador-859c4d96d6-pmv2p\" (UID: \"3c5f2713-9010-4625-ae17-6546784dc4f6\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-pmv2p"
Jul 28 10:19:05.390961 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:19:05.390964    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbtnt\" (UniqueName: \"kubernetes.io/projected/3c5f2713-9010-4625-ae17-6546784dc4f6-kube-api-access-sbtnt\") pod \"limitador-limitador-859c4d96d6-pmv2p\" (UID: \"3c5f2713-9010-4625-ae17-6546784dc4f6\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-pmv2p"
Jul 28 10:19:05.394373 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:19:05.394337    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-pmv2p"]
Jul 28 10:19:05.397370 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:19:05.397328    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:19:05.491992 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:19:05.491943    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3c5f2713-9010-4625-ae17-6546784dc4f6-config-file\") pod \"limitador-limitador-859c4d96d6-pmv2p\" (UID: \"3c5f2713-9010-4625-ae17-6546784dc4f6\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-pmv2p"
Jul 28 10:19:05.492171 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:19:05.492004    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbtnt\" (UniqueName: \"kubernetes.io/projected/3c5f2713-9010-4625-ae17-6546784dc4f6-kube-api-access-sbtnt\") pod \"limitador-limitador-859c4d96d6-pmv2p\" (UID: \"3c5f2713-9010-4625-ae17-6546784dc4f6\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-pmv2p"
Jul 28 10:19:05.492968 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:19:05.492938    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3c5f2713-9010-4625-ae17-6546784dc4f6-config-file\") pod \"limitador-limitador-859c4d96d6-pmv2p\" (UID: \"3c5f2713-9010-4625-ae17-6546784dc4f6\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-pmv2p"
Jul 28 10:19:05.497945 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:19:05.497860    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:19:05.500988 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:19:05.500945    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbtnt\" (UniqueName: \"kubernetes.io/projected/3c5f2713-9010-4625-ae17-6546784dc4f6-kube-api-access-sbtnt\") pod \"limitador-limitador-859c4d96d6-pmv2p\" (UID: \"3c5f2713-9010-4625-ae17-6546784dc4f6\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-pmv2p"
Jul 28 10:19:05.613427 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:19:05.613335    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-pmv2p"
Jul 28 10:19:05.754170 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:19:05.754144    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-pmv2p"]
Jul 28 10:19:05.756508 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:19:05.756479    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c5f2713_9010_4625_ae17_6546784dc4f6.slice/crio-c5b09c5240b21af483f459c54885b77e88b8a19eccc3fe0841609385a6c1be20 WatchSource:0}: Error finding container c5b09c5240b21af483f459c54885b77e88b8a19eccc3fe0841609385a6c1be20: Status 404 returned error can't find the container with id c5b09c5240b21af483f459c54885b77e88b8a19eccc3fe0841609385a6c1be20
Jul 28 10:19:05.796528 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:19:05.796485    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-pmv2p" event={"ID":"3c5f2713-9010-4625-ae17-6546784dc4f6","Type":"ContainerStarted","Data":"c5b09c5240b21af483f459c54885b77e88b8a19eccc3fe0841609385a6c1be20"}
Jul 28 10:19:06.148225 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:19:06.148194    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:19:06.249467 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:19:06.249432    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:19:07.807798 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:19:07.807761    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-pmv2p" event={"ID":"3c5f2713-9010-4625-ae17-6546784dc4f6","Type":"ContainerStarted","Data":"92f22cbb24fa3e94c11c7f8866219b65a8f07ed7a546f77c605addb604f09a6a"}
Jul 28 10:19:07.808166 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:19:07.807891    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-859c4d96d6-pmv2p"
Jul 28 10:19:07.823620 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:19:07.823558    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-859c4d96d6-pmv2p" podStartSLOduration=1.322594854 podStartE2EDuration="2.823516276s" totalImagesPullingTime="1.500921422s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:19:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:19:07.822931382 +0000 UTC m=+682.086327710" watchObservedRunningTime="2026-07-28 10:19:07.823516276 +0000 UTC m=+682.086912606"
Jul 28 10:19:08.111667 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:19:08.111520    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:19:08.211837 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:19:08.211773    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:19:18.812709 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:19:18.812674    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-859c4d96d6-pmv2p"
Jul 28 10:19:21.428819 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:19:21.428787    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-pmv2p"]
Jul 28 10:19:21.429317 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:19:21.429055    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-859c4d96d6-pmv2p" podUID="3c5f2713-9010-4625-ae17-6546784dc4f6" containerName="limitador" containerID="cri-o://92f22cbb24fa3e94c11c7f8866219b65a8f07ed7a546f77c605addb604f09a6a" gracePeriod=30
Jul 28 10:19:21.872276 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:19:21.872231    2579 generic.go:356] "Generic (PLEG): container finished" podID="3c5f2713-9010-4625-ae17-6546784dc4f6" containerID="92f22cbb24fa3e94c11c7f8866219b65a8f07ed7a546f77c605addb604f09a6a" exitCode=0
Jul 28 10:19:21.872430 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:19:21.872303    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-pmv2p" event={"ID":"3c5f2713-9010-4625-ae17-6546784dc4f6","Type":"ContainerDied","Data":"92f22cbb24fa3e94c11c7f8866219b65a8f07ed7a546f77c605addb604f09a6a"}
Jul 28 10:19:21.965095 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:19:21.965071    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-pmv2p"
Jul 28 10:19:22.020739 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:19:22.020654    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3c5f2713-9010-4625-ae17-6546784dc4f6-config-file\") pod \"3c5f2713-9010-4625-ae17-6546784dc4f6\" (UID: \"3c5f2713-9010-4625-ae17-6546784dc4f6\") "
Jul 28 10:19:22.020874 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:19:22.020750    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sbtnt\" (UniqueName: \"kubernetes.io/projected/3c5f2713-9010-4625-ae17-6546784dc4f6-kube-api-access-sbtnt\") pod \"3c5f2713-9010-4625-ae17-6546784dc4f6\" (UID: \"3c5f2713-9010-4625-ae17-6546784dc4f6\") "
Jul 28 10:19:22.021015 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:19:22.020988    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c5f2713-9010-4625-ae17-6546784dc4f6-config-file" (OuterVolumeSpecName: "config-file") pod "3c5f2713-9010-4625-ae17-6546784dc4f6" (UID: "3c5f2713-9010-4625-ae17-6546784dc4f6"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:19:22.022918 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:19:22.022896    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c5f2713-9010-4625-ae17-6546784dc4f6-kube-api-access-sbtnt" (OuterVolumeSpecName: "kube-api-access-sbtnt") pod "3c5f2713-9010-4625-ae17-6546784dc4f6" (UID: "3c5f2713-9010-4625-ae17-6546784dc4f6"). InnerVolumeSpecName "kube-api-access-sbtnt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:19:22.121260 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:19:22.121222    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbtnt\" (UniqueName: \"kubernetes.io/projected/3c5f2713-9010-4625-ae17-6546784dc4f6-kube-api-access-sbtnt\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 10:19:22.121260 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:19:22.121251    2579 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3c5f2713-9010-4625-ae17-6546784dc4f6-config-file\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 10:19:22.877879 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:19:22.877853    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-pmv2p"
Jul 28 10:19:22.878287 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:19:22.877850    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-pmv2p" event={"ID":"3c5f2713-9010-4625-ae17-6546784dc4f6","Type":"ContainerDied","Data":"c5b09c5240b21af483f459c54885b77e88b8a19eccc3fe0841609385a6c1be20"}
Jul 28 10:19:22.878287 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:19:22.878011    2579 scope.go:117] "RemoveContainer" containerID="92f22cbb24fa3e94c11c7f8866219b65a8f07ed7a546f77c605addb604f09a6a"
Jul 28 10:19:22.893918 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:19:22.893890    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-pmv2p"]
Jul 28 10:19:22.898296 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:19:22.898272    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-pmv2p"]
Jul 28 10:19:24.275702 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:19:24.275670    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3c5f2713-9010-4625-ae17-6546784dc4f6" path="/var/lib/kubelet/pods/3c5f2713-9010-4625-ae17-6546784dc4f6/volumes"
Jul 28 10:19:27.514271 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:19:27.514237    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:19:28.930765 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:19:28.930727    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:19:38.501885 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:19:38.501797    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:19:38.592429 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:19:38.592381    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:19:43.608205 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:19:43.608168    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-6968749498-582ph"]
Jul 28 10:19:43.608681 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:19:43.608355    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/kserve-controller-manager-6968749498-582ph" podUID="db5a09b7-b665-4729-afb4-a991f7998edf" containerName="manager" containerID="cri-o://dd4d9e4d8de9bc555d299466709d0455a7487b8eb74beafb72a475734224f4fa" gracePeriod=10
Jul 28 10:19:43.633847 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:19:43.633819    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-6968749498-brk56"]
Jul 28 10:19:43.634114 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:19:43.634103    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c5f2713-9010-4625-ae17-6546784dc4f6" containerName="limitador"
Jul 28 10:19:43.634160 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:19:43.634116    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c5f2713-9010-4625-ae17-6546784dc4f6" containerName="limitador"
Jul 28 10:19:43.634193 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:19:43.634161    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c5f2713-9010-4625-ae17-6546784dc4f6" containerName="limitador"
Jul 28 10:19:43.636919 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:19:43.636904    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-6968749498-brk56"
Jul 28 10:19:43.643786 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:19:43.643763    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-6968749498-brk56"]
Jul 28 10:19:43.685444 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:19:43.685413    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4l42\" (UniqueName: \"kubernetes.io/projected/6c21258e-66df-4a31-a1b4-8d10fcb6ea5d-kube-api-access-t4l42\") pod \"kserve-controller-manager-6968749498-brk56\" (UID: \"6c21258e-66df-4a31-a1b4-8d10fcb6ea5d\") " pod="kserve/kserve-controller-manager-6968749498-brk56"
Jul 28 10:19:43.685609 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:19:43.685459    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6c21258e-66df-4a31-a1b4-8d10fcb6ea5d-cert\") pod \"kserve-controller-manager-6968749498-brk56\" (UID: \"6c21258e-66df-4a31-a1b4-8d10fcb6ea5d\") " pod="kserve/kserve-controller-manager-6968749498-brk56"
Jul 28 10:19:43.786111 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:19:43.786071    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6c21258e-66df-4a31-a1b4-8d10fcb6ea5d-cert\") pod \"kserve-controller-manager-6968749498-brk56\" (UID: \"6c21258e-66df-4a31-a1b4-8d10fcb6ea5d\") " pod="kserve/kserve-controller-manager-6968749498-brk56"
Jul 28 10:19:43.786262 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:19:43.786180    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4l42\" (UniqueName: \"kubernetes.io/projected/6c21258e-66df-4a31-a1b4-8d10fcb6ea5d-kube-api-access-t4l42\") pod \"kserve-controller-manager-6968749498-brk56\" (UID: \"6c21258e-66df-4a31-a1b4-8d10fcb6ea5d\") " pod="kserve/kserve-controller-manager-6968749498-brk56"
Jul 28 10:19:43.789406 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:19:43.789377    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6c21258e-66df-4a31-a1b4-8d10fcb6ea5d-cert\") pod \"kserve-controller-manager-6968749498-brk56\" (UID: \"6c21258e-66df-4a31-a1b4-8d10fcb6ea5d\") " pod="kserve/kserve-controller-manager-6968749498-brk56"
Jul 28 10:19:43.794170 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:19:43.794145    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4l42\" (UniqueName: \"kubernetes.io/projected/6c21258e-66df-4a31-a1b4-8d10fcb6ea5d-kube-api-access-t4l42\") pod \"kserve-controller-manager-6968749498-brk56\" (UID: \"6c21258e-66df-4a31-a1b4-8d10fcb6ea5d\") " pod="kserve/kserve-controller-manager-6968749498-brk56"
Jul 28 10:19:43.856605 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:19:43.856573    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-6968749498-582ph"
Jul 28 10:19:43.886515 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:19:43.886430    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-blclc\" (UniqueName: \"kubernetes.io/projected/db5a09b7-b665-4729-afb4-a991f7998edf-kube-api-access-blclc\") pod \"db5a09b7-b665-4729-afb4-a991f7998edf\" (UID: \"db5a09b7-b665-4729-afb4-a991f7998edf\") "
Jul 28 10:19:43.886683 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:19:43.886524    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/db5a09b7-b665-4729-afb4-a991f7998edf-cert\") pod \"db5a09b7-b665-4729-afb4-a991f7998edf\" (UID: \"db5a09b7-b665-4729-afb4-a991f7998edf\") "
Jul 28 10:19:43.888701 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:19:43.888670    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db5a09b7-b665-4729-afb4-a991f7998edf-kube-api-access-blclc" (OuterVolumeSpecName: "kube-api-access-blclc") pod "db5a09b7-b665-4729-afb4-a991f7998edf" (UID: "db5a09b7-b665-4729-afb4-a991f7998edf"). InnerVolumeSpecName "kube-api-access-blclc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:19:43.888701 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:19:43.888682    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db5a09b7-b665-4729-afb4-a991f7998edf-cert" (OuterVolumeSpecName: "cert") pod "db5a09b7-b665-4729-afb4-a991f7998edf" (UID: "db5a09b7-b665-4729-afb4-a991f7998edf"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:19:43.979963 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:19:43.979929    2579 generic.go:356] "Generic (PLEG): container finished" podID="db5a09b7-b665-4729-afb4-a991f7998edf" containerID="dd4d9e4d8de9bc555d299466709d0455a7487b8eb74beafb72a475734224f4fa" exitCode=0
Jul 28 10:19:43.980135 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:19:43.979982    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-6968749498-582ph"
Jul 28 10:19:43.980135 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:19:43.980017    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-6968749498-582ph" event={"ID":"db5a09b7-b665-4729-afb4-a991f7998edf","Type":"ContainerDied","Data":"dd4d9e4d8de9bc555d299466709d0455a7487b8eb74beafb72a475734224f4fa"}
Jul 28 10:19:43.980135 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:19:43.980071    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-6968749498-582ph" event={"ID":"db5a09b7-b665-4729-afb4-a991f7998edf","Type":"ContainerDied","Data":"de3662b51d1c00fef32c256215e71b29b28af9520be488ee208c1d79097cd27d"}
Jul 28 10:19:43.980135 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:19:43.980094    2579 scope.go:117] "RemoveContainer" containerID="dd4d9e4d8de9bc555d299466709d0455a7487b8eb74beafb72a475734224f4fa"
Jul 28 10:19:43.987593 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:19:43.987570    2579 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/db5a09b7-b665-4729-afb4-a991f7998edf-cert\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 10:19:43.987593 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:19:43.987593    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-blclc\" (UniqueName: \"kubernetes.io/projected/db5a09b7-b665-4729-afb4-a991f7998edf-kube-api-access-blclc\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 10:19:43.988854 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:19:43.988836    2579 scope.go:117] "RemoveContainer" containerID="dd4d9e4d8de9bc555d299466709d0455a7487b8eb74beafb72a475734224f4fa"
Jul 28 10:19:43.989117 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:19:43.989098    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd4d9e4d8de9bc555d299466709d0455a7487b8eb74beafb72a475734224f4fa\": container with ID starting with dd4d9e4d8de9bc555d299466709d0455a7487b8eb74beafb72a475734224f4fa not found: ID does not exist" containerID="dd4d9e4d8de9bc555d299466709d0455a7487b8eb74beafb72a475734224f4fa"
Jul 28 10:19:43.989164 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:19:43.989127    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd4d9e4d8de9bc555d299466709d0455a7487b8eb74beafb72a475734224f4fa"} err="failed to get container status \"dd4d9e4d8de9bc555d299466709d0455a7487b8eb74beafb72a475734224f4fa\": rpc error: code = NotFound desc = could not find container \"dd4d9e4d8de9bc555d299466709d0455a7487b8eb74beafb72a475734224f4fa\": container with ID starting with dd4d9e4d8de9bc555d299466709d0455a7487b8eb74beafb72a475734224f4fa not found: ID does not exist"
Jul 28 10:19:43.991598 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:19:43.991582    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-6968749498-brk56"
Jul 28 10:19:44.000816 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:19:44.000762    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-6968749498-582ph"]
Jul 28 10:19:44.003277 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:19:44.003260    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-6968749498-582ph"]
Jul 28 10:19:44.121136 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:19:44.121112    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-6968749498-brk56"]
Jul 28 10:19:44.123161 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:19:44.123131    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c21258e_66df_4a31_a1b4_8d10fcb6ea5d.slice/crio-90012c642c5b513a9772ea1aa4215358a9d8f5a7c631f148fa9bf9acd40062ec WatchSource:0}: Error finding container 90012c642c5b513a9772ea1aa4215358a9d8f5a7c631f148fa9bf9acd40062ec: Status 404 returned error can't find the container with id 90012c642c5b513a9772ea1aa4215358a9d8f5a7c631f148fa9bf9acd40062ec
Jul 28 10:19:44.124465 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:19:44.124448    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 10:19:44.275895 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:19:44.275814    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="db5a09b7-b665-4729-afb4-a991f7998edf" path="/var/lib/kubelet/pods/db5a09b7-b665-4729-afb4-a991f7998edf/volumes"
Jul 28 10:19:44.985912 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:19:44.985803    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-6968749498-brk56" event={"ID":"6c21258e-66df-4a31-a1b4-8d10fcb6ea5d","Type":"ContainerStarted","Data":"3d2a4df783f8a0b1560398c6833e55e11ff615e62875e10869148de0ea842b13"}
Jul 28 10:19:44.985912 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:19:44.985874    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-6968749498-brk56" event={"ID":"6c21258e-66df-4a31-a1b4-8d10fcb6ea5d","Type":"ContainerStarted","Data":"90012c642c5b513a9772ea1aa4215358a9d8f5a7c631f148fa9bf9acd40062ec"}
Jul 28 10:19:44.985912 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:19:44.985905    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-6968749498-brk56"
Jul 28 10:19:45.003935 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:19:45.003883    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-6968749498-brk56" podStartSLOduration=1.5881687709999999 podStartE2EDuration="2.003868489s" totalImagesPullingTime="415.699718ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:19:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:19:45.001662792 +0000 UTC m=+719.265059121" watchObservedRunningTime="2026-07-28 10:19:45.003868489 +0000 UTC m=+719.267264818"
Jul 28 10:20:15.996629 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:20:15.996601    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-6968749498-brk56"
Jul 28 10:20:17.091292 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:20:17.091256    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:20:17.162765 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:20:17.162731    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-7b954ffc95-tr22l"]
Jul 28 10:20:17.163022 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:20:17.163010    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db5a09b7-b665-4729-afb4-a991f7998edf" containerName="manager"
Jul 28 10:20:17.163072 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:20:17.163024    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="db5a09b7-b665-4729-afb4-a991f7998edf" containerName="manager"
Jul 28 10:20:17.163105 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:20:17.163078    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="db5a09b7-b665-4729-afb4-a991f7998edf" containerName="manager"
Jul 28 10:20:17.165921 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:20:17.165902    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-tr22l"
Jul 28 10:20:17.169313 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:20:17.169285    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-ltgj8\""
Jul 28 10:20:17.189407 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:20:17.189383    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-tr22l"]
Jul 28 10:20:17.194820 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:20:17.194799    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Jul 28 10:20:17.233048 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:20:17.233017    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-75d9cdbddf-vgnpz"]
Jul 28 10:20:17.235966 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:20:17.235951    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-75d9cdbddf-vgnpz"
Jul 28 10:20:17.239094 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:20:17.239075    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-5td4b\""
Jul 28 10:20:17.239166 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:20:17.239113    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Jul 28 10:20:17.265656 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:20:17.265618    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41f55d40-f073-4037-bde6-fd85c2e58cf1-tls-certs\") pod \"model-serving-api-7b954ffc95-tr22l\" (UID: \"41f55d40-f073-4037-bde6-fd85c2e58cf1\") " pod="kserve/model-serving-api-7b954ffc95-tr22l"
Jul 28 10:20:17.265781 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:20:17.265659    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmnfj\" (UniqueName: \"kubernetes.io/projected/41f55d40-f073-4037-bde6-fd85c2e58cf1-kube-api-access-xmnfj\") pod \"model-serving-api-7b954ffc95-tr22l\" (UID: \"41f55d40-f073-4037-bde6-fd85c2e58cf1\") " pod="kserve/model-serving-api-7b954ffc95-tr22l"
Jul 28 10:20:17.273865 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:20:17.273843    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-75d9cdbddf-vgnpz"]
Jul 28 10:20:17.308901 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:20:17.308875    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:20:17.367071 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:20:17.366986    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41f55d40-f073-4037-bde6-fd85c2e58cf1-tls-certs\") pod \"model-serving-api-7b954ffc95-tr22l\" (UID: \"41f55d40-f073-4037-bde6-fd85c2e58cf1\") " pod="kserve/model-serving-api-7b954ffc95-tr22l"
Jul 28 10:20:17.367071 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:20:17.367025    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f4b1f552-5478-4e5e-8fd5-6659b9b61ba2-cert\") pod \"odh-model-controller-75d9cdbddf-vgnpz\" (UID: \"f4b1f552-5478-4e5e-8fd5-6659b9b61ba2\") " pod="kserve/odh-model-controller-75d9cdbddf-vgnpz"
Jul 28 10:20:17.367289 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:20:17.367141    2579 secret.go:189] Couldn't get secret kserve/model-serving-api-tls: secret "model-serving-api-tls" not found
Jul 28 10:20:17.367289 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:20:17.367155    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmnfj\" (UniqueName: \"kubernetes.io/projected/41f55d40-f073-4037-bde6-fd85c2e58cf1-kube-api-access-xmnfj\") pod \"model-serving-api-7b954ffc95-tr22l\" (UID: \"41f55d40-f073-4037-bde6-fd85c2e58cf1\") " pod="kserve/model-serving-api-7b954ffc95-tr22l"
Jul 28 10:20:17.367289 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:20:17.367195    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41f55d40-f073-4037-bde6-fd85c2e58cf1-tls-certs podName:41f55d40-f073-4037-bde6-fd85c2e58cf1 nodeName:}" failed. No retries permitted until 2026-07-28 10:20:17.867179605 +0000 UTC m=+752.130575912 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/41f55d40-f073-4037-bde6-fd85c2e58cf1-tls-certs") pod "model-serving-api-7b954ffc95-tr22l" (UID: "41f55d40-f073-4037-bde6-fd85c2e58cf1") : secret "model-serving-api-tls" not found
Jul 28 10:20:17.367289 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:20:17.367262    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dfsm\" (UniqueName: \"kubernetes.io/projected/f4b1f552-5478-4e5e-8fd5-6659b9b61ba2-kube-api-access-2dfsm\") pod \"odh-model-controller-75d9cdbddf-vgnpz\" (UID: \"f4b1f552-5478-4e5e-8fd5-6659b9b61ba2\") " pod="kserve/odh-model-controller-75d9cdbddf-vgnpz"
Jul 28 10:20:17.388326 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:20:17.388292    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmnfj\" (UniqueName: \"kubernetes.io/projected/41f55d40-f073-4037-bde6-fd85c2e58cf1-kube-api-access-xmnfj\") pod \"model-serving-api-7b954ffc95-tr22l\" (UID: \"41f55d40-f073-4037-bde6-fd85c2e58cf1\") " pod="kserve/model-serving-api-7b954ffc95-tr22l"
Jul 28 10:20:17.442260 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:20:17.442223    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:20:17.467862 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:20:17.467827    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f4b1f552-5478-4e5e-8fd5-6659b9b61ba2-cert\") pod \"odh-model-controller-75d9cdbddf-vgnpz\" (UID: \"f4b1f552-5478-4e5e-8fd5-6659b9b61ba2\") " pod="kserve/odh-model-controller-75d9cdbddf-vgnpz"
Jul 28 10:20:17.468014 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:20:17.467899    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dfsm\" (UniqueName: \"kubernetes.io/projected/f4b1f552-5478-4e5e-8fd5-6659b9b61ba2-kube-api-access-2dfsm\") pod \"odh-model-controller-75d9cdbddf-vgnpz\" (UID: \"f4b1f552-5478-4e5e-8fd5-6659b9b61ba2\") " pod="kserve/odh-model-controller-75d9cdbddf-vgnpz"
Jul 28 10:20:17.470159 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:20:17.470140    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f4b1f552-5478-4e5e-8fd5-6659b9b61ba2-cert\") pod \"odh-model-controller-75d9cdbddf-vgnpz\" (UID: \"f4b1f552-5478-4e5e-8fd5-6659b9b61ba2\") " pod="kserve/odh-model-controller-75d9cdbddf-vgnpz"
Jul 28 10:20:17.479575 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:20:17.479534    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dfsm\" (UniqueName: \"kubernetes.io/projected/f4b1f552-5478-4e5e-8fd5-6659b9b61ba2-kube-api-access-2dfsm\") pod \"odh-model-controller-75d9cdbddf-vgnpz\" (UID: \"f4b1f552-5478-4e5e-8fd5-6659b9b61ba2\") " pod="kserve/odh-model-controller-75d9cdbddf-vgnpz"
Jul 28 10:20:17.544464 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:20:17.544432    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:20:17.545234 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:20:17.545208    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-75d9cdbddf-vgnpz"
Jul 28 10:20:17.678333 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:20:17.678307    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-75d9cdbddf-vgnpz"]
Jul 28 10:20:17.681077 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:20:17.681053    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4b1f552_5478_4e5e_8fd5_6659b9b61ba2.slice/crio-4310147f06e4d6875adaf5eb7458767f75bf5ee4786eb9b046b173e8ff19fe9c WatchSource:0}: Error finding container 4310147f06e4d6875adaf5eb7458767f75bf5ee4786eb9b046b173e8ff19fe9c: Status 404 returned error can't find the container with id 4310147f06e4d6875adaf5eb7458767f75bf5ee4786eb9b046b173e8ff19fe9c
Jul 28 10:20:17.871391 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:20:17.871350    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41f55d40-f073-4037-bde6-fd85c2e58cf1-tls-certs\") pod \"model-serving-api-7b954ffc95-tr22l\" (UID: \"41f55d40-f073-4037-bde6-fd85c2e58cf1\") " pod="kserve/model-serving-api-7b954ffc95-tr22l"
Jul 28 10:20:17.873848 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:20:17.873822    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41f55d40-f073-4037-bde6-fd85c2e58cf1-tls-certs\") pod \"model-serving-api-7b954ffc95-tr22l\" (UID: \"41f55d40-f073-4037-bde6-fd85c2e58cf1\") " pod="kserve/model-serving-api-7b954ffc95-tr22l"
Jul 28 10:20:18.074203 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:20:18.074168    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:20:18.075978 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:20:18.075953    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-tr22l"
Jul 28 10:20:18.137052 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:20:18.137014    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:20:18.139186 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:20:18.139160    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-75d9cdbddf-vgnpz" event={"ID":"f4b1f552-5478-4e5e-8fd5-6659b9b61ba2","Type":"ContainerStarted","Data":"4310147f06e4d6875adaf5eb7458767f75bf5ee4786eb9b046b173e8ff19fe9c"}
Jul 28 10:20:18.210475 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:20:18.210442    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-tr22l"]
Jul 28 10:20:18.212094 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:20:18.212063    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f55d40_f073_4037_bde6_fd85c2e58cf1.slice/crio-3886148e2a56cec61203eb8b78d3730e7635df07fcc734ff76ee83d616977307 WatchSource:0}: Error finding container 3886148e2a56cec61203eb8b78d3730e7635df07fcc734ff76ee83d616977307: Status 404 returned error can't find the container with id 3886148e2a56cec61203eb8b78d3730e7635df07fcc734ff76ee83d616977307
Jul 28 10:20:19.147799 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:20:19.147760    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-tr22l" event={"ID":"41f55d40-f073-4037-bde6-fd85c2e58cf1","Type":"ContainerStarted","Data":"3886148e2a56cec61203eb8b78d3730e7635df07fcc734ff76ee83d616977307"}
Jul 28 10:20:22.164803 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:20:22.164749    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-75d9cdbddf-vgnpz" event={"ID":"f4b1f552-5478-4e5e-8fd5-6659b9b61ba2","Type":"ContainerStarted","Data":"8953c97852a862a0a9066daf7adddaea4b9696139de1dd704767b8c0268510e7"}
Jul 28 10:20:22.165290 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:20:22.164926    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-75d9cdbddf-vgnpz"
Jul 28 10:20:22.166438 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:20:22.166414    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-tr22l" event={"ID":"41f55d40-f073-4037-bde6-fd85c2e58cf1","Type":"ContainerStarted","Data":"9bc3e07a4c669f78f62545c9d4f44dd3e63fd586d9d36d94b8979de296f3b23f"}
Jul 28 10:20:22.166575 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:20:22.166559    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-7b954ffc95-tr22l"
Jul 28 10:20:22.182623 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:20:22.182576    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-75d9cdbddf-vgnpz" podStartSLOduration=1.266151238 podStartE2EDuration="5.182561664s" totalImagesPullingTime="3.916410426s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:20:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:20:22.181425918 +0000 UTC m=+756.444822259" watchObservedRunningTime="2026-07-28 10:20:22.182561664 +0000 UTC m=+756.445957989"
Jul 28 10:20:22.197136 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:20:22.197089    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-7b954ffc95-tr22l" podStartSLOduration=1.7710998820000001 podStartE2EDuration="5.197076121s" totalImagesPullingTime="3.425976239s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:20:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:20:22.195289147 +0000 UTC m=+756.458685475" watchObservedRunningTime="2026-07-28 10:20:22.197076121 +0000 UTC m=+756.460472449"
Jul 28 10:20:33.173912 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:20:33.173883    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-75d9cdbddf-vgnpz"
Jul 28 10:20:33.175965 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:20:33.175943    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-7b954ffc95-tr22l"
Jul 28 10:20:34.007210 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:20:34.007174    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-vs6wx"]
Jul 28 10:20:34.010722 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:20:34.010704    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-vs6wx"
Jul 28 10:20:34.016781 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:20:34.016755    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-vs6wx"]
Jul 28 10:20:34.088481 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:20:34.088439    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9aad1d78-7972-42e1-b53b-a7fc64944590-model-cache\") pod \"s3-init-vs6wx\" (UID: \"9aad1d78-7972-42e1-b53b-a7fc64944590\") " pod="kserve/s3-init-vs6wx"
Jul 28 10:20:34.088679 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:20:34.088506    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-br4dr\" (UniqueName: \"kubernetes.io/projected/9aad1d78-7972-42e1-b53b-a7fc64944590-kube-api-access-br4dr\") pod \"s3-init-vs6wx\" (UID: \"9aad1d78-7972-42e1-b53b-a7fc64944590\") " pod="kserve/s3-init-vs6wx"
Jul 28 10:20:34.189838 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:20:34.189805    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9aad1d78-7972-42e1-b53b-a7fc64944590-model-cache\") pod \"s3-init-vs6wx\" (UID: \"9aad1d78-7972-42e1-b53b-a7fc64944590\") " pod="kserve/s3-init-vs6wx"
Jul 28 10:20:34.190181 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:20:34.189866    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-br4dr\" (UniqueName: \"kubernetes.io/projected/9aad1d78-7972-42e1-b53b-a7fc64944590-kube-api-access-br4dr\") pod \"s3-init-vs6wx\" (UID: \"9aad1d78-7972-42e1-b53b-a7fc64944590\") " pod="kserve/s3-init-vs6wx"
Jul 28 10:20:34.190221 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:20:34.190177    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9aad1d78-7972-42e1-b53b-a7fc64944590-model-cache\") pod \"s3-init-vs6wx\" (UID: \"9aad1d78-7972-42e1-b53b-a7fc64944590\") " pod="kserve/s3-init-vs6wx"
Jul 28 10:20:34.198047 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:20:34.198025    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-br4dr\" (UniqueName: \"kubernetes.io/projected/9aad1d78-7972-42e1-b53b-a7fc64944590-kube-api-access-br4dr\") pod \"s3-init-vs6wx\" (UID: \"9aad1d78-7972-42e1-b53b-a7fc64944590\") " pod="kserve/s3-init-vs6wx"
Jul 28 10:20:34.320469 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:20:34.320427    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-vs6wx"
Jul 28 10:20:34.465602 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:20:34.465576    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-vs6wx"]
Jul 28 10:20:34.467255 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:20:34.467229    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9aad1d78_7972_42e1_b53b_a7fc64944590.slice/crio-ecbbcb76f90fe57c1c40ac0045ad2dd59820e508b68c72f27a98d1e9fcb3ce41 WatchSource:0}: Error finding container ecbbcb76f90fe57c1c40ac0045ad2dd59820e508b68c72f27a98d1e9fcb3ce41: Status 404 returned error can't find the container with id ecbbcb76f90fe57c1c40ac0045ad2dd59820e508b68c72f27a98d1e9fcb3ce41
Jul 28 10:20:35.228272 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:20:35.228232    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-vs6wx" event={"ID":"9aad1d78-7972-42e1-b53b-a7fc64944590","Type":"ContainerStarted","Data":"ecbbcb76f90fe57c1c40ac0045ad2dd59820e508b68c72f27a98d1e9fcb3ce41"}
Jul 28 10:20:38.246072 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:20:38.246031    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-vs6wx" event={"ID":"9aad1d78-7972-42e1-b53b-a7fc64944590","Type":"ContainerStarted","Data":"dfb7ac72dfa85c6e863148cf2bc4f202191d8d0b6818ede23b267d05209afe7c"}
Jul 28 10:20:40.257269 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:20:40.257234    2579 generic.go:356] "Generic (PLEG): container finished" podID="9aad1d78-7972-42e1-b53b-a7fc64944590" containerID="dfb7ac72dfa85c6e863148cf2bc4f202191d8d0b6818ede23b267d05209afe7c" exitCode=0
Jul 28 10:20:40.257661 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:20:40.257312    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-vs6wx" event={"ID":"9aad1d78-7972-42e1-b53b-a7fc64944590","Type":"ContainerDied","Data":"dfb7ac72dfa85c6e863148cf2bc4f202191d8d0b6818ede23b267d05209afe7c"}
Jul 28 10:20:45.286662 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:20:45.286624    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-vs6wx" event={"ID":"9aad1d78-7972-42e1-b53b-a7fc64944590","Type":"ContainerStarted","Data":"3c920c16f71e88e8dea21532a319ce9031c306e80194da7b44095a85fbe6b249"}
Jul 28 10:20:45.304473 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:20:45.304416    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-vs6wx" podStartSLOduration=2.983816813 podStartE2EDuration="12.304397767s" totalImagesPullingTime="7.798523403s" totalInitContainerRuntime="1.522057551s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:20:33 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:20:45.30352824 +0000 UTC m=+779.566924571" watchObservedRunningTime="2026-07-28 10:20:45.304397767 +0000 UTC m=+779.567794096"
Jul 28 10:20:48.218055 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:20:48.218006    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:21:50.593617 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:21:50.593576    2579 generic.go:356] "Generic (PLEG): container finished" podID="9aad1d78-7972-42e1-b53b-a7fc64944590" containerID="3c920c16f71e88e8dea21532a319ce9031c306e80194da7b44095a85fbe6b249" exitCode=0
Jul 28 10:21:50.594041 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:21:50.593655    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-vs6wx" event={"ID":"9aad1d78-7972-42e1-b53b-a7fc64944590","Type":"ContainerDied","Data":"3c920c16f71e88e8dea21532a319ce9031c306e80194da7b44095a85fbe6b249"}
Jul 28 10:21:51.724903 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:21:51.724877    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-vs6wx"
Jul 28 10:21:51.797432 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:21:51.797399    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-br4dr\" (UniqueName: \"kubernetes.io/projected/9aad1d78-7972-42e1-b53b-a7fc64944590-kube-api-access-br4dr\") pod \"9aad1d78-7972-42e1-b53b-a7fc64944590\" (UID: \"9aad1d78-7972-42e1-b53b-a7fc64944590\") "
Jul 28 10:21:51.797629 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:21:51.797454    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9aad1d78-7972-42e1-b53b-a7fc64944590-model-cache\") pod \"9aad1d78-7972-42e1-b53b-a7fc64944590\" (UID: \"9aad1d78-7972-42e1-b53b-a7fc64944590\") "
Jul 28 10:21:51.799625 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:21:51.799585    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9aad1d78-7972-42e1-b53b-a7fc64944590-kube-api-access-br4dr" (OuterVolumeSpecName: "kube-api-access-br4dr") pod "9aad1d78-7972-42e1-b53b-a7fc64944590" (UID: "9aad1d78-7972-42e1-b53b-a7fc64944590"). InnerVolumeSpecName "kube-api-access-br4dr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:21:51.816500 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:21:51.816473    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9aad1d78-7972-42e1-b53b-a7fc64944590-model-cache" (OuterVolumeSpecName: "model-cache") pod "9aad1d78-7972-42e1-b53b-a7fc64944590" (UID: "9aad1d78-7972-42e1-b53b-a7fc64944590"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:21:51.898677 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:21:51.898599    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-br4dr\" (UniqueName: \"kubernetes.io/projected/9aad1d78-7972-42e1-b53b-a7fc64944590-kube-api-access-br4dr\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 10:21:51.898677 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:21:51.898625    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9aad1d78-7972-42e1-b53b-a7fc64944590-model-cache\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 10:21:52.603887 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:21:52.603855    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-vs6wx"
Jul 28 10:21:52.604048 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:21:52.603892    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-vs6wx" event={"ID":"9aad1d78-7972-42e1-b53b-a7fc64944590","Type":"ContainerDied","Data":"ecbbcb76f90fe57c1c40ac0045ad2dd59820e508b68c72f27a98d1e9fcb3ce41"}
Jul 28 10:21:52.604048 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:21:52.603925    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ecbbcb76f90fe57c1c40ac0045ad2dd59820e508b68c72f27a98d1e9fcb3ce41"
Jul 28 10:21:52.986082 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:21:52.985991    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:21:53.085903 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:21:53.085870    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:22:05.542585 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:22:05.542531    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:22:05.641260 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:22:05.641219    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:22:13.292933 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:22:13.292896    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-68fdf86f947d52k"]
Jul 28 10:22:13.293391 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:22:13.293235    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9aad1d78-7972-42e1-b53b-a7fc64944590" containerName="download-hf-model"
Jul 28 10:22:13.293391 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:22:13.293250    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="9aad1d78-7972-42e1-b53b-a7fc64944590" containerName="download-hf-model"
Jul 28 10:22:13.293391 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:22:13.293259    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9aad1d78-7972-42e1-b53b-a7fc64944590" containerName="s3-init"
Jul 28 10:22:13.293391 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:22:13.293265    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="9aad1d78-7972-42e1-b53b-a7fc64944590" containerName="s3-init"
Jul 28 10:22:13.293391 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:22:13.293316    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="9aad1d78-7972-42e1-b53b-a7fc64944590" containerName="s3-init"
Jul 28 10:22:13.296334 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:22:13.296315    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-68fdf86f947d52k"
Jul 28 10:22:13.298615 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:22:13.298591    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"seaweedfs-s3-creds\""
Jul 28 10:22:13.299307 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:22:13.298609    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"default-dockercfg-vvxvq\""
Jul 28 10:22:13.299449 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:22:13.298645    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"llmisvd10b93c8eba12f87c3a350a5cae2ee0c-kserve-self-signed-certs\""
Jul 28 10:22:13.299449 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:22:13.298688    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"openshift-service-ca.crt\""
Jul 28 10:22:13.299449 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:22:13.298696    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"kube-root-ca.crt\""
Jul 28 10:22:13.306701 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:22:13.306681    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-68fdf86f947d52k"]
Jul 28 10:22:13.323929 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:22:13.323899    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:22:13.373703 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:22:13.373669    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b9557a90-dc91-4177-a83f-79a0b7526722-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-68fdf86f947d52k\" (UID: \"b9557a90-dc91-4177-a83f-79a0b7526722\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-68fdf86f947d52k"
Jul 28 10:22:13.373703 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:22:13.373708    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b9557a90-dc91-4177-a83f-79a0b7526722-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-68fdf86f947d52k\" (UID: \"b9557a90-dc91-4177-a83f-79a0b7526722\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-68fdf86f947d52k"
Jul 28 10:22:13.373948 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:22:13.373733    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9557a90-dc91-4177-a83f-79a0b7526722-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-68fdf86f947d52k\" (UID: \"b9557a90-dc91-4177-a83f-79a0b7526722\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-68fdf86f947d52k"
Jul 28 10:22:13.373948 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:22:13.373791    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9557a90-dc91-4177-a83f-79a0b7526722-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-68fdf86f947d52k\" (UID: \"b9557a90-dc91-4177-a83f-79a0b7526722\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-68fdf86f947d52k"
Jul 28 10:22:13.373948 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:22:13.373808    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzs6w\" (UniqueName: \"kubernetes.io/projected/b9557a90-dc91-4177-a83f-79a0b7526722-kube-api-access-xzs6w\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-68fdf86f947d52k\" (UID: \"b9557a90-dc91-4177-a83f-79a0b7526722\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-68fdf86f947d52k"
Jul 28 10:22:13.373948 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:22:13.373842    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9557a90-dc91-4177-a83f-79a0b7526722-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-68fdf86f947d52k\" (UID: \"b9557a90-dc91-4177-a83f-79a0b7526722\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-68fdf86f947d52k"
Jul 28 10:22:13.373948 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:22:13.373865    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9557a90-dc91-4177-a83f-79a0b7526722-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-68fdf86f947d52k\" (UID: \"b9557a90-dc91-4177-a83f-79a0b7526722\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-68fdf86f947d52k"
Jul 28 10:22:13.474399 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:22:13.474364    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9557a90-dc91-4177-a83f-79a0b7526722-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-68fdf86f947d52k\" (UID: \"b9557a90-dc91-4177-a83f-79a0b7526722\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-68fdf86f947d52k"
Jul 28 10:22:13.474399 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:22:13.474399    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9557a90-dc91-4177-a83f-79a0b7526722-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-68fdf86f947d52k\" (UID: \"b9557a90-dc91-4177-a83f-79a0b7526722\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-68fdf86f947d52k"
Jul 28 10:22:13.474691 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:22:13.474436    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b9557a90-dc91-4177-a83f-79a0b7526722-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-68fdf86f947d52k\" (UID: \"b9557a90-dc91-4177-a83f-79a0b7526722\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-68fdf86f947d52k"
Jul 28 10:22:13.474691 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:22:13.474473    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b9557a90-dc91-4177-a83f-79a0b7526722-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-68fdf86f947d52k\" (UID: \"b9557a90-dc91-4177-a83f-79a0b7526722\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-68fdf86f947d52k"
Jul 28 10:22:13.474691 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:22:13.474497    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9557a90-dc91-4177-a83f-79a0b7526722-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-68fdf86f947d52k\" (UID: \"b9557a90-dc91-4177-a83f-79a0b7526722\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-68fdf86f947d52k"
Jul 28 10:22:13.474691 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:22:13.474531    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9557a90-dc91-4177-a83f-79a0b7526722-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-68fdf86f947d52k\" (UID: \"b9557a90-dc91-4177-a83f-79a0b7526722\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-68fdf86f947d52k"
Jul 28 10:22:13.474691 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:22:13.474586    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzs6w\" (UniqueName: \"kubernetes.io/projected/b9557a90-dc91-4177-a83f-79a0b7526722-kube-api-access-xzs6w\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-68fdf86f947d52k\" (UID: \"b9557a90-dc91-4177-a83f-79a0b7526722\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-68fdf86f947d52k"
Jul 28 10:22:13.474943 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:22:13.474856    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b9557a90-dc91-4177-a83f-79a0b7526722-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-68fdf86f947d52k\" (UID: \"b9557a90-dc91-4177-a83f-79a0b7526722\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-68fdf86f947d52k"
Jul 28 10:22:13.474943 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:22:13.474877    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9557a90-dc91-4177-a83f-79a0b7526722-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-68fdf86f947d52k\" (UID: \"b9557a90-dc91-4177-a83f-79a0b7526722\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-68fdf86f947d52k"
Jul 28 10:22:13.475042 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:22:13.474940    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9557a90-dc91-4177-a83f-79a0b7526722-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-68fdf86f947d52k\" (UID: \"b9557a90-dc91-4177-a83f-79a0b7526722\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-68fdf86f947d52k"
Jul 28 10:22:13.475122 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:22:13.475105    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b9557a90-dc91-4177-a83f-79a0b7526722-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-68fdf86f947d52k\" (UID: \"b9557a90-dc91-4177-a83f-79a0b7526722\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-68fdf86f947d52k"
Jul 28 10:22:13.476776 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:22:13.476756    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9557a90-dc91-4177-a83f-79a0b7526722-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-68fdf86f947d52k\" (UID: \"b9557a90-dc91-4177-a83f-79a0b7526722\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-68fdf86f947d52k"
Jul 28 10:22:13.477143 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:22:13.477120    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9557a90-dc91-4177-a83f-79a0b7526722-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-68fdf86f947d52k\" (UID: \"b9557a90-dc91-4177-a83f-79a0b7526722\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-68fdf86f947d52k"
Jul 28 10:22:13.482868 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:22:13.482830    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzs6w\" (UniqueName: \"kubernetes.io/projected/b9557a90-dc91-4177-a83f-79a0b7526722-kube-api-access-xzs6w\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-68fdf86f947d52k\" (UID: \"b9557a90-dc91-4177-a83f-79a0b7526722\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-68fdf86f947d52k"
Jul 28 10:22:13.608264 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:22:13.608177    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-68fdf86f947d52k"
Jul 28 10:22:13.753716 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:22:13.753676    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-68fdf86f947d52k"]
Jul 28 10:22:13.755520 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:22:13.755495    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9557a90_dc91_4177_a83f_79a0b7526722.slice/crio-6d0c8f480a3472251e3edc8687ec21ca8ba7f31216b3990ec351f60322b81986 WatchSource:0}: Error finding container 6d0c8f480a3472251e3edc8687ec21ca8ba7f31216b3990ec351f60322b81986: Status 404 returned error can't find the container with id 6d0c8f480a3472251e3edc8687ec21ca8ba7f31216b3990ec351f60322b81986
Jul 28 10:22:14.711134 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:22:14.711091    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-68fdf86f947d52k" event={"ID":"b9557a90-dc91-4177-a83f-79a0b7526722","Type":"ContainerStarted","Data":"ca449dba1e7688014aa15a6568cdb54b8a5b16f4f02f8f5d5dd95ec15b691328"}
Jul 28 10:22:14.711517 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:22:14.711177    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-68fdf86f947d52k" event={"ID":"b9557a90-dc91-4177-a83f-79a0b7526722","Type":"ContainerStarted","Data":"6d0c8f480a3472251e3edc8687ec21ca8ba7f31216b3990ec351f60322b81986"}
Jul 28 10:22:17.322508 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:22:17.322472    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:22:17.596800 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:22:17.596593    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-wzs74"]
Jul 28 10:22:17.601838 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:22:17.601761    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-wzs74"
Jul 28 10:22:17.606326 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:22:17.606073    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"kube-root-ca.crt\""
Jul 28 10:22:17.606470 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:22:17.606342    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"openshift-service-ca.crt\""
Jul 28 10:22:17.607207 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:22:17.606574    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"fc-smoke-test-kserve-self-signed-certs\""
Jul 28 10:22:17.607207 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:22:17.606845    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"fc-smoke-test-epp-sa-dockercfg-5mkpj\""
Jul 28 10:22:17.607207 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:22:17.607184    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"default-dockercfg-m68p6\""
Jul 28 10:22:17.611614 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:22:17.611589    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-wzs74"]
Jul 28 10:22:17.639076 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:22:17.639046    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:22:17.712575 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:22:17.712518    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc8fbfc3-4880-45a8-ad0a-4305a9c1e12a-tls-certs\") pod \"fc-smoke-test-kserve-router-scheduler-5cfd47986-wzs74\" (UID: \"dc8fbfc3-4880-45a8-ad0a-4305a9c1e12a\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-wzs74"
Jul 28 10:22:17.712752 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:22:17.712616    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfjgv\" (UniqueName: \"kubernetes.io/projected/dc8fbfc3-4880-45a8-ad0a-4305a9c1e12a-kube-api-access-nfjgv\") pod \"fc-smoke-test-kserve-router-scheduler-5cfd47986-wzs74\" (UID: \"dc8fbfc3-4880-45a8-ad0a-4305a9c1e12a\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-wzs74"
Jul 28 10:22:17.813807 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:22:17.813772    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc8fbfc3-4880-45a8-ad0a-4305a9c1e12a-tls-certs\") pod \"fc-smoke-test-kserve-router-scheduler-5cfd47986-wzs74\" (UID: \"dc8fbfc3-4880-45a8-ad0a-4305a9c1e12a\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-wzs74"
Jul 28 10:22:17.813982 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:22:17.813830    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfjgv\" (UniqueName: \"kubernetes.io/projected/dc8fbfc3-4880-45a8-ad0a-4305a9c1e12a-kube-api-access-nfjgv\") pod \"fc-smoke-test-kserve-router-scheduler-5cfd47986-wzs74\" (UID: \"dc8fbfc3-4880-45a8-ad0a-4305a9c1e12a\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-wzs74"
Jul 28 10:22:17.817011 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:22:17.816972    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc8fbfc3-4880-45a8-ad0a-4305a9c1e12a-tls-certs\") pod \"fc-smoke-test-kserve-router-scheduler-5cfd47986-wzs74\" (UID: \"dc8fbfc3-4880-45a8-ad0a-4305a9c1e12a\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-wzs74"
Jul 28 10:22:17.823028 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:22:17.822991    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfjgv\" (UniqueName: \"kubernetes.io/projected/dc8fbfc3-4880-45a8-ad0a-4305a9c1e12a-kube-api-access-nfjgv\") pod \"fc-smoke-test-kserve-router-scheduler-5cfd47986-wzs74\" (UID: \"dc8fbfc3-4880-45a8-ad0a-4305a9c1e12a\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-wzs74"
Jul 28 10:22:17.921035 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:22:17.920927    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-wzs74"
Jul 28 10:22:18.115318 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:22:18.115289    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-wzs74"]
Jul 28 10:22:18.117202 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:22:18.117181    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc8fbfc3_4880_45a8_ad0a_4305a9c1e12a.slice/crio-854a3740866b22ab5c3519acf564b9c8b776b3096b9cd533dd7f369383826284 WatchSource:0}: Error finding container 854a3740866b22ab5c3519acf564b9c8b776b3096b9cd533dd7f369383826284: Status 404 returned error can't find the container with id 854a3740866b22ab5c3519acf564b9c8b776b3096b9cd533dd7f369383826284
Jul 28 10:22:18.733464 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:22:18.733420    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-wzs74" event={"ID":"dc8fbfc3-4880-45a8-ad0a-4305a9c1e12a","Type":"ContainerStarted","Data":"854a3740866b22ab5c3519acf564b9c8b776b3096b9cd533dd7f369383826284"}
Jul 28 10:22:18.734840 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:22:18.734815    2579 generic.go:356] "Generic (PLEG): container finished" podID="b9557a90-dc91-4177-a83f-79a0b7526722" containerID="ca449dba1e7688014aa15a6568cdb54b8a5b16f4f02f8f5d5dd95ec15b691328" exitCode=0
Jul 28 10:22:18.734970 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:22:18.734854    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-68fdf86f947d52k" event={"ID":"b9557a90-dc91-4177-a83f-79a0b7526722","Type":"ContainerDied","Data":"ca449dba1e7688014aa15a6568cdb54b8a5b16f4f02f8f5d5dd95ec15b691328"}
Jul 28 10:22:23.468136 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:22:23.467761    2579 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-58c5cd85c9-zf8mf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.20:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Jul 28 10:22:23.468136 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:22:23.467834    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-zf8mf" podUID="490ca628-de57-478e-b4b8-af249144d0d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.20:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 28 10:22:24.816120 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:22:24.816084    2579 generic.go:356] "Generic (PLEG): container finished" podID="490ca628-de57-478e-b4b8-af249144d0d2" containerID="3a53df2c4d7c003e6071f0f9e2b7d9823bf54b06e5b2d6a65a649efc0211cccd" exitCode=0
Jul 28 10:22:24.816658 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:22:24.816175    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-zf8mf" event={"ID":"490ca628-de57-478e-b4b8-af249144d0d2","Type":"ContainerDied","Data":"3a53df2c4d7c003e6071f0f9e2b7d9823bf54b06e5b2d6a65a649efc0211cccd"}
Jul 28 10:22:24.816735 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:22:24.816660    2579 scope.go:117] "RemoveContainer" containerID="3a53df2c4d7c003e6071f0f9e2b7d9823bf54b06e5b2d6a65a649efc0211cccd"
Jul 28 10:22:24.817900 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:22:24.817864    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 28 10:22:24.818008 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:22:24.817955    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 28 10:22:24.818049 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:22:24.818004    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 28 10:22:24.818597 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:22:24.818573    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-wzs74" event={"ID":"dc8fbfc3-4880-45a8-ad0a-4305a9c1e12a","Type":"ContainerStarted","Data":"397cf529ba2bdc75c223e7a938876c9c6f0c5994cab30eec805a56e4004e51ad"}
Jul 28 10:22:24.818680 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:22:24.818668    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-wzs74"
Jul 28 10:22:25.828193 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:22:25.828143    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-zf8mf" event={"ID":"490ca628-de57-478e-b4b8-af249144d0d2","Type":"ContainerStarted","Data":"a4b5d1b7a19728a0d1124427d9c45f3bfe7273a7aa2f8786e502b2e01687dd75"}
Jul 28 10:22:25.856467 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:22:25.853385    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-wzs74" podStartSLOduration=2.923796706 podStartE2EDuration="8.853362941s" totalImagesPullingTime="5.929566235s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:22:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:22:24.863818841 +0000 UTC m=+879.127215171" watchObservedRunningTime="2026-07-28 10:22:25.853362941 +0000 UTC m=+880.116759272"
Jul 28 10:22:26.315007 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:22:26.314920    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-zf8mf"
Jul 28 10:22:26.315249 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:22:26.315221    2579 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-58c5cd85c9-zf8mf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused" start-of-body=
Jul 28 10:22:26.315317 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:22:26.315281    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-zf8mf" podUID="490ca628-de57-478e-b4b8-af249144d0d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused"
Jul 28 10:22:27.315772 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:22:27.315626    2579 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-58c5cd85c9-zf8mf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused" start-of-body=
Jul 28 10:22:27.315772 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:22:27.315690    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-zf8mf" podUID="490ca628-de57-478e-b4b8-af249144d0d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused"
Jul 28 10:22:28.315743 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:22:28.315702    2579 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-58c5cd85c9-zf8mf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused" start-of-body=
Jul 28 10:22:28.315934 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:22:28.315770    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-zf8mf" podUID="490ca628-de57-478e-b4b8-af249144d0d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused"
Jul 28 10:22:29.315181 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:22:29.315103    2579 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-58c5cd85c9-zf8mf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused" start-of-body=
Jul 28 10:22:29.315369 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:22:29.315182    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-zf8mf" podUID="490ca628-de57-478e-b4b8-af249144d0d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused"
Jul 28 10:22:30.315817 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:22:30.315780    2579 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-58c5cd85c9-zf8mf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused" start-of-body=
Jul 28 10:22:30.316304 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:22:30.315875    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-zf8mf" podUID="490ca628-de57-478e-b4b8-af249144d0d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused"
Jul 28 10:22:31.428962 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:22:31.428920    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-zf8mf"
Jul 28 10:22:31.429931 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:22:31.429880    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-zf8mf"
Jul 28 10:22:31.430964 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:22:31.430916    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-zf8mf"
Jul 28 10:22:45.842902 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:22:45.842818    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:22:45.944601 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:22:45.944563    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:22:52.004599 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:22:52.004508    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-68fdf86f947d52k" event={"ID":"b9557a90-dc91-4177-a83f-79a0b7526722","Type":"ContainerStarted","Data":"2324aafa4c6be305b1ca666363c8ae73fdbe41cc0d81e11c3dc59308759befd0"}
Jul 28 10:22:52.029643 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:22:52.029585    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-68fdf86f947d52k" podStartSLOduration=1.7665027530000001 podStartE2EDuration="39.02956456s" totalImagesPullingTime="32.890246501s" totalInitContainerRuntime="4.372815306s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:22:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:22:52.027788086 +0000 UTC m=+906.291184414" watchObservedRunningTime="2026-07-28 10:22:52.02956456 +0000 UTC m=+906.292960890"
Jul 28 10:22:52.608958 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:22:52.608909    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-68fdf86f947d52k"
Jul 28 10:22:53.608463 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:22:53.608413    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-68fdf86f947d52k"
Jul 28 10:22:53.609937 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:22:53.609900    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-68fdf86f947d52k" podUID="b9557a90-dc91-4177-a83f-79a0b7526722" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/health\": dial tcp 10.134.0.33:8000: connect: connection refused"
Jul 28 10:22:55.830478 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:22:55.830449    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-wzs74"
Jul 28 10:22:59.214970 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:22:59.214929    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:23:03.608868 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:23:03.608813    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-68fdf86f947d52k" podUID="b9557a90-dc91-4177-a83f-79a0b7526722" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/health\": dial tcp 10.134.0.33:8000: connect: connection refused"
Jul 28 10:23:06.235965 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:23:06.235924    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-wzs74"]
Jul 28 10:23:06.247472 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:23:06.247423    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:23:06.312391 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:23:06.312343    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:23:13.608711 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:23:13.608661    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-68fdf86f947d52k" podUID="b9557a90-dc91-4177-a83f-79a0b7526722" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/health\": dial tcp 10.134.0.33:8000: connect: connection refused"
Jul 28 10:23:15.830096 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:23:15.830052    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-wzs74" podUID="dc8fbfc3-4880-45a8-ad0a-4305a9c1e12a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 10:23:21.256504 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:23:21.256443    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-wzs74" podUID="dc8fbfc3-4880-45a8-ad0a-4305a9c1e12a" containerName="main" containerID="cri-o://397cf529ba2bdc75c223e7a938876c9c6f0c5994cab30eec805a56e4004e51ad" gracePeriod=45
Jul 28 10:23:21.533498 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:23:21.533462    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-wzs74"
Jul 28 10:23:21.580739 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:23:21.580711    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nfjgv\" (UniqueName: \"kubernetes.io/projected/dc8fbfc3-4880-45a8-ad0a-4305a9c1e12a-kube-api-access-nfjgv\") pod \"dc8fbfc3-4880-45a8-ad0a-4305a9c1e12a\" (UID: \"dc8fbfc3-4880-45a8-ad0a-4305a9c1e12a\") "
Jul 28 10:23:21.580904 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:23:21.580756    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc8fbfc3-4880-45a8-ad0a-4305a9c1e12a-tls-certs\") pod \"dc8fbfc3-4880-45a8-ad0a-4305a9c1e12a\" (UID: \"dc8fbfc3-4880-45a8-ad0a-4305a9c1e12a\") "
Jul 28 10:23:21.582918 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:23:21.582883    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc8fbfc3-4880-45a8-ad0a-4305a9c1e12a-kube-api-access-nfjgv" (OuterVolumeSpecName: "kube-api-access-nfjgv") pod "dc8fbfc3-4880-45a8-ad0a-4305a9c1e12a" (UID: "dc8fbfc3-4880-45a8-ad0a-4305a9c1e12a"). InnerVolumeSpecName "kube-api-access-nfjgv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:23:21.583022 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:23:21.582988    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc8fbfc3-4880-45a8-ad0a-4305a9c1e12a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "dc8fbfc3-4880-45a8-ad0a-4305a9c1e12a" (UID: "dc8fbfc3-4880-45a8-ad0a-4305a9c1e12a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:23:21.681985 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:23:21.681952    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfjgv\" (UniqueName: \"kubernetes.io/projected/dc8fbfc3-4880-45a8-ad0a-4305a9c1e12a-kube-api-access-nfjgv\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 10:23:21.681985 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:23:21.681982    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc8fbfc3-4880-45a8-ad0a-4305a9c1e12a-tls-certs\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 10:23:22.158114 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:23:22.158072    2579 generic.go:356] "Generic (PLEG): container finished" podID="dc8fbfc3-4880-45a8-ad0a-4305a9c1e12a" containerID="397cf529ba2bdc75c223e7a938876c9c6f0c5994cab30eec805a56e4004e51ad" exitCode=0
Jul 28 10:23:22.158279 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:23:22.158138    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-wzs74"
Jul 28 10:23:22.158279 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:23:22.158160    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-wzs74" event={"ID":"dc8fbfc3-4880-45a8-ad0a-4305a9c1e12a","Type":"ContainerDied","Data":"397cf529ba2bdc75c223e7a938876c9c6f0c5994cab30eec805a56e4004e51ad"}
Jul 28 10:23:22.158279 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:23:22.158244    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-wzs74" event={"ID":"dc8fbfc3-4880-45a8-ad0a-4305a9c1e12a","Type":"ContainerDied","Data":"854a3740866b22ab5c3519acf564b9c8b776b3096b9cd533dd7f369383826284"}
Jul 28 10:23:22.158279 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:23:22.158271    2579 scope.go:117] "RemoveContainer" containerID="397cf529ba2bdc75c223e7a938876c9c6f0c5994cab30eec805a56e4004e51ad"
Jul 28 10:23:22.167332 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:23:22.167312    2579 scope.go:117] "RemoveContainer" containerID="397cf529ba2bdc75c223e7a938876c9c6f0c5994cab30eec805a56e4004e51ad"
Jul 28 10:23:22.167626 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:23:22.167607    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"397cf529ba2bdc75c223e7a938876c9c6f0c5994cab30eec805a56e4004e51ad\": container with ID starting with 397cf529ba2bdc75c223e7a938876c9c6f0c5994cab30eec805a56e4004e51ad not found: ID does not exist" containerID="397cf529ba2bdc75c223e7a938876c9c6f0c5994cab30eec805a56e4004e51ad"
Jul 28 10:23:22.167679 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:23:22.167634    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"397cf529ba2bdc75c223e7a938876c9c6f0c5994cab30eec805a56e4004e51ad"} err="failed to get container status \"397cf529ba2bdc75c223e7a938876c9c6f0c5994cab30eec805a56e4004e51ad\": rpc error: code = NotFound desc = could not find container \"397cf529ba2bdc75c223e7a938876c9c6f0c5994cab30eec805a56e4004e51ad\": container with ID starting with 397cf529ba2bdc75c223e7a938876c9c6f0c5994cab30eec805a56e4004e51ad not found: ID does not exist"
Jul 28 10:23:22.181397 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:23:22.181369    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-wzs74"]
Jul 28 10:23:22.187715 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:23:22.187689    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-wzs74"]
Jul 28 10:23:22.277007 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:23:22.276965    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dc8fbfc3-4880-45a8-ad0a-4305a9c1e12a" path="/var/lib/kubelet/pods/dc8fbfc3-4880-45a8-ad0a-4305a9c1e12a/volumes"
Jul 28 10:23:23.609114 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:23:23.609068    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-68fdf86f947d52k" podUID="b9557a90-dc91-4177-a83f-79a0b7526722" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/health\": dial tcp 10.134.0.33:8000: connect: connection refused"
Jul 28 10:23:26.263571 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:23:26.263515    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:23:26.362207 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:23:26.362167    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:23:32.476949 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:23:32.476916    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:23:33.609317 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:23:33.609270    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-68fdf86f947d52k" podUID="b9557a90-dc91-4177-a83f-79a0b7526722" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/health\": dial tcp 10.134.0.33:8000: connect: connection refused"
Jul 28 10:23:38.006682 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:23:38.006637    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:23:38.107499 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:23:38.107456    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:23:39.374975 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:23:39.374938    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:23:39.621866 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:23:39.621829    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-tkbpp"]
Jul 28 10:23:39.622228 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:23:39.622211    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc8fbfc3-4880-45a8-ad0a-4305a9c1e12a" containerName="main"
Jul 28 10:23:39.622310 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:23:39.622231    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc8fbfc3-4880-45a8-ad0a-4305a9c1e12a" containerName="main"
Jul 28 10:23:39.622362 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:23:39.622322    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc8fbfc3-4880-45a8-ad0a-4305a9c1e12a" containerName="main"
Jul 28 10:23:39.628599 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:23:39.628514    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-tkbpp"
Jul 28 10:23:39.631485 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:23:39.631465    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"kube-root-ca.crt\""
Jul 28 10:23:39.631628 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:23:39.631600    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"default-dockercfg-9rknv\""
Jul 28 10:23:39.631628 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:23:39.631465    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"fc-concurrency-test-kserve-self-signed-certs\""
Jul 28 10:23:39.631742 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:23:39.631465    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"openshift-service-ca.crt\""
Jul 28 10:23:39.631742 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:23:39.631524    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"fc-concurrency-test-epp-sa-dockercfg-tpmcn\""
Jul 28 10:23:39.633092 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:23:39.633070    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-tkbpp"]
Jul 28 10:23:39.646174 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:23:39.646141    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:23:39.734529 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:23:39.734489    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztbbr\" (UniqueName: \"kubernetes.io/projected/9acc8b3d-88e0-4517-9961-39ff4107b8a1-kube-api-access-ztbbr\") pod \"fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-tkbpp\" (UID: \"9acc8b3d-88e0-4517-9961-39ff4107b8a1\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-tkbpp"
Jul 28 10:23:39.734736 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:23:39.734569    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9acc8b3d-88e0-4517-9961-39ff4107b8a1-tls-certs\") pod \"fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-tkbpp\" (UID: \"9acc8b3d-88e0-4517-9961-39ff4107b8a1\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-tkbpp"
Jul 28 10:23:39.835474 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:23:39.835431    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztbbr\" (UniqueName: \"kubernetes.io/projected/9acc8b3d-88e0-4517-9961-39ff4107b8a1-kube-api-access-ztbbr\") pod \"fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-tkbpp\" (UID: \"9acc8b3d-88e0-4517-9961-39ff4107b8a1\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-tkbpp"
Jul 28 10:23:39.835681 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:23:39.835499    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9acc8b3d-88e0-4517-9961-39ff4107b8a1-tls-certs\") pod \"fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-tkbpp\" (UID: \"9acc8b3d-88e0-4517-9961-39ff4107b8a1\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-tkbpp"
Jul 28 10:23:39.837995 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:23:39.837971    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9acc8b3d-88e0-4517-9961-39ff4107b8a1-tls-certs\") pod \"fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-tkbpp\" (UID: \"9acc8b3d-88e0-4517-9961-39ff4107b8a1\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-tkbpp"
Jul 28 10:23:39.843200 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:23:39.843174    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztbbr\" (UniqueName: \"kubernetes.io/projected/9acc8b3d-88e0-4517-9961-39ff4107b8a1-kube-api-access-ztbbr\") pod \"fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-tkbpp\" (UID: \"9acc8b3d-88e0-4517-9961-39ff4107b8a1\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-tkbpp"
Jul 28 10:23:39.940225 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:23:39.940124    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-tkbpp"
Jul 28 10:23:40.085896 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:23:40.085865    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-tkbpp"]
Jul 28 10:23:40.089625 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:23:40.089594    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9acc8b3d_88e0_4517_9961_39ff4107b8a1.slice/crio-478d47281b8e68a8e824b16330e166c6a61c5421808e14986ca3e84e8a6e31ae WatchSource:0}: Error finding container 478d47281b8e68a8e824b16330e166c6a61c5421808e14986ca3e84e8a6e31ae: Status 404 returned error can't find the container with id 478d47281b8e68a8e824b16330e166c6a61c5421808e14986ca3e84e8a6e31ae
Jul 28 10:23:40.251799 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:23:40.251693    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-tkbpp" event={"ID":"9acc8b3d-88e0-4517-9961-39ff4107b8a1","Type":"ContainerStarted","Data":"999b9b9fad3f9303b31eba9ced65fd2374a96dcd501c762fd7d91b56b0fbe6cc"}
Jul 28 10:23:40.251799 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:23:40.251771    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-tkbpp" event={"ID":"9acc8b3d-88e0-4517-9961-39ff4107b8a1","Type":"ContainerStarted","Data":"478d47281b8e68a8e824b16330e166c6a61c5421808e14986ca3e84e8a6e31ae"}
Jul 28 10:23:40.251980 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:23:40.251803    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-tkbpp"
Jul 28 10:23:40.270973 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:23:40.270913    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-tkbpp" podStartSLOduration=1.270890455 podStartE2EDuration="1.270890455s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:23:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:23:40.268234651 +0000 UTC m=+954.531630980" watchObservedRunningTime="2026-07-28 10:23:40.270890455 +0000 UTC m=+954.534286786"
Jul 28 10:23:43.609132 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:23:43.609079    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-68fdf86f947d52k" podUID="b9557a90-dc91-4177-a83f-79a0b7526722" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/health\": dial tcp 10.134.0.33:8000: connect: connection refused"
Jul 28 10:23:43.750112 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:23:43.750068    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:23:53.609494 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:23:53.609448    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-68fdf86f947d52k" podUID="b9557a90-dc91-4177-a83f-79a0b7526722" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/health\": dial tcp 10.134.0.33:8000: connect: connection refused"
Jul 28 10:24:03.609442 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:03.609392    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-68fdf86f947d52k" podUID="b9557a90-dc91-4177-a83f-79a0b7526722" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/health\": dial tcp 10.134.0.33:8000: connect: connection refused"
Jul 28 10:24:09.566219 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:09.566126    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:24:11.260556 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:11.260511    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-tkbpp"
Jul 28 10:24:13.609688 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:13.609640    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-68fdf86f947d52k" podUID="b9557a90-dc91-4177-a83f-79a0b7526722" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/health\": dial tcp 10.134.0.33:8000: connect: connection refused"
Jul 28 10:24:17.408211 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:17.408172    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-tkbpp"]
Jul 28 10:24:17.417612 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:17.417576    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:24:17.495666 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:17.495626    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:24:21.260600 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:21.260563    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-tkbpp" podUID="9acc8b3d-88e0-4517-9961-39ff4107b8a1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 10:24:23.608799 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:23.608754    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-68fdf86f947d52k" podUID="b9557a90-dc91-4177-a83f-79a0b7526722" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/health\": dial tcp 10.134.0.33:8000: connect: connection refused"
Jul 28 10:24:31.259816 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:31.259775    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-tkbpp" podUID="9acc8b3d-88e0-4517-9961-39ff4107b8a1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 10:24:32.429516 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:32.429472    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-tkbpp" podUID="9acc8b3d-88e0-4517-9961-39ff4107b8a1" containerName="main" containerID="cri-o://999b9b9fad3f9303b31eba9ced65fd2374a96dcd501c762fd7d91b56b0fbe6cc" gracePeriod=45
Jul 28 10:24:32.672334 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:32.672305    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-tkbpp"
Jul 28 10:24:32.794509 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:32.794468    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9acc8b3d-88e0-4517-9961-39ff4107b8a1-tls-certs\") pod \"9acc8b3d-88e0-4517-9961-39ff4107b8a1\" (UID: \"9acc8b3d-88e0-4517-9961-39ff4107b8a1\") "
Jul 28 10:24:32.794721 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:32.794592    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ztbbr\" (UniqueName: \"kubernetes.io/projected/9acc8b3d-88e0-4517-9961-39ff4107b8a1-kube-api-access-ztbbr\") pod \"9acc8b3d-88e0-4517-9961-39ff4107b8a1\" (UID: \"9acc8b3d-88e0-4517-9961-39ff4107b8a1\") "
Jul 28 10:24:32.796759 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:32.796727    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9acc8b3d-88e0-4517-9961-39ff4107b8a1-kube-api-access-ztbbr" (OuterVolumeSpecName: "kube-api-access-ztbbr") pod "9acc8b3d-88e0-4517-9961-39ff4107b8a1" (UID: "9acc8b3d-88e0-4517-9961-39ff4107b8a1"). InnerVolumeSpecName "kube-api-access-ztbbr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:24:32.796874 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:32.796841    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9acc8b3d-88e0-4517-9961-39ff4107b8a1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9acc8b3d-88e0-4517-9961-39ff4107b8a1" (UID: "9acc8b3d-88e0-4517-9961-39ff4107b8a1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:24:32.895777 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:32.895730    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9acc8b3d-88e0-4517-9961-39ff4107b8a1-tls-certs\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 10:24:32.895777 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:32.895775    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztbbr\" (UniqueName: \"kubernetes.io/projected/9acc8b3d-88e0-4517-9961-39ff4107b8a1-kube-api-access-ztbbr\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 10:24:33.546364 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:33.546329    2579 generic.go:356] "Generic (PLEG): container finished" podID="9acc8b3d-88e0-4517-9961-39ff4107b8a1" containerID="999b9b9fad3f9303b31eba9ced65fd2374a96dcd501c762fd7d91b56b0fbe6cc" exitCode=0
Jul 28 10:24:33.546796 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:33.546369    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-tkbpp" event={"ID":"9acc8b3d-88e0-4517-9961-39ff4107b8a1","Type":"ContainerDied","Data":"999b9b9fad3f9303b31eba9ced65fd2374a96dcd501c762fd7d91b56b0fbe6cc"}
Jul 28 10:24:33.546796 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:33.546376    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-tkbpp"
Jul 28 10:24:33.546796 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:33.546415    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-tkbpp" event={"ID":"9acc8b3d-88e0-4517-9961-39ff4107b8a1","Type":"ContainerDied","Data":"478d47281b8e68a8e824b16330e166c6a61c5421808e14986ca3e84e8a6e31ae"}
Jul 28 10:24:33.546796 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:33.546437    2579 scope.go:117] "RemoveContainer" containerID="999b9b9fad3f9303b31eba9ced65fd2374a96dcd501c762fd7d91b56b0fbe6cc"
Jul 28 10:24:33.556605 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:33.556584    2579 scope.go:117] "RemoveContainer" containerID="999b9b9fad3f9303b31eba9ced65fd2374a96dcd501c762fd7d91b56b0fbe6cc"
Jul 28 10:24:33.556890 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:24:33.556871    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"999b9b9fad3f9303b31eba9ced65fd2374a96dcd501c762fd7d91b56b0fbe6cc\": container with ID starting with 999b9b9fad3f9303b31eba9ced65fd2374a96dcd501c762fd7d91b56b0fbe6cc not found: ID does not exist" containerID="999b9b9fad3f9303b31eba9ced65fd2374a96dcd501c762fd7d91b56b0fbe6cc"
Jul 28 10:24:33.556954 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:33.556901    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"999b9b9fad3f9303b31eba9ced65fd2374a96dcd501c762fd7d91b56b0fbe6cc"} err="failed to get container status \"999b9b9fad3f9303b31eba9ced65fd2374a96dcd501c762fd7d91b56b0fbe6cc\": rpc error: code = NotFound desc = could not find container \"999b9b9fad3f9303b31eba9ced65fd2374a96dcd501c762fd7d91b56b0fbe6cc\": container with ID starting with 999b9b9fad3f9303b31eba9ced65fd2374a96dcd501c762fd7d91b56b0fbe6cc not found: ID does not exist"
Jul 28 10:24:33.567945 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:33.567915    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-tkbpp"]
Jul 28 10:24:33.571825 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:33.571797    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-tkbpp"]
Jul 28 10:24:33.609125 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:33.609086    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-68fdf86f947d52k" podUID="b9557a90-dc91-4177-a83f-79a0b7526722" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/health\": dial tcp 10.134.0.33:8000: connect: connection refused"
Jul 28 10:24:34.276009 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:34.275980    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9acc8b3d-88e0-4517-9961-39ff4107b8a1" path="/var/lib/kubelet/pods/9acc8b3d-88e0-4517-9961-39ff4107b8a1/volumes"
Jul 28 10:24:36.326272 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:36.326233    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:24:37.575216 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:37.575166    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:24:37.647282 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:37.647247    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-nnhnk"]
Jul 28 10:24:37.647745 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:37.647728    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9acc8b3d-88e0-4517-9961-39ff4107b8a1" containerName="main"
Jul 28 10:24:37.647815 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:37.647748    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="9acc8b3d-88e0-4517-9961-39ff4107b8a1" containerName="main"
Jul 28 10:24:37.647855 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:37.647827    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="9acc8b3d-88e0-4517-9961-39ff4107b8a1" containerName="main"
Jul 28 10:24:37.650935 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:37.650917    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-nnhnk"
Jul 28 10:24:37.653756 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:37.653731    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"openshift-gw-ca-root-cert\""
Jul 28 10:24:37.654414 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:37.654392    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"router-gateway-1-openshift-default-dockercfg-rb9l9\""
Jul 28 10:24:37.654520 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:37.654412    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"openshift-service-ca.crt\""
Jul 28 10:24:37.654520 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:37.654416    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"kube-root-ca.crt\""
Jul 28 10:24:37.661650 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:37.661615    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-nnhnk"]
Jul 28 10:24:37.691725 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:37.691692    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:24:37.734120 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:37.734075    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebd51311-0cf5-4c8c-9cca-343bfb33ff44-istio-podinfo\") pod \"router-gateway-1-openshift-default-6b46477c5b-nnhnk\" (UID: \"ebd51311-0cf5-4c8c-9cca-343bfb33ff44\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-nnhnk"
Jul 28 10:24:37.734314 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:37.734134    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebd51311-0cf5-4c8c-9cca-343bfb33ff44-workload-certs\") pod \"router-gateway-1-openshift-default-6b46477c5b-nnhnk\" (UID: \"ebd51311-0cf5-4c8c-9cca-343bfb33ff44\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-nnhnk"
Jul 28 10:24:37.734314 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:37.734165    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebd51311-0cf5-4c8c-9cca-343bfb33ff44-istio-envoy\") pod \"router-gateway-1-openshift-default-6b46477c5b-nnhnk\" (UID: \"ebd51311-0cf5-4c8c-9cca-343bfb33ff44\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-nnhnk"
Jul 28 10:24:37.734314 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:37.734191    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebd51311-0cf5-4c8c-9cca-343bfb33ff44-istio-token\") pod \"router-gateway-1-openshift-default-6b46477c5b-nnhnk\" (UID: \"ebd51311-0cf5-4c8c-9cca-343bfb33ff44\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-nnhnk"
Jul 28 10:24:37.734314 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:37.734256    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebd51311-0cf5-4c8c-9cca-343bfb33ff44-workload-socket\") pod \"router-gateway-1-openshift-default-6b46477c5b-nnhnk\" (UID: \"ebd51311-0cf5-4c8c-9cca-343bfb33ff44\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-nnhnk"
Jul 28 10:24:37.734314 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:37.734281    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebd51311-0cf5-4c8c-9cca-343bfb33ff44-istio-data\") pod \"router-gateway-1-openshift-default-6b46477c5b-nnhnk\" (UID: \"ebd51311-0cf5-4c8c-9cca-343bfb33ff44\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-nnhnk"
Jul 28 10:24:37.734314 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:37.734310    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebd51311-0cf5-4c8c-9cca-343bfb33ff44-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-6b46477c5b-nnhnk\" (UID: \"ebd51311-0cf5-4c8c-9cca-343bfb33ff44\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-nnhnk"
Jul 28 10:24:37.734565 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:37.734343    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gx442\" (UniqueName: \"kubernetes.io/projected/ebd51311-0cf5-4c8c-9cca-343bfb33ff44-kube-api-access-gx442\") pod \"router-gateway-1-openshift-default-6b46477c5b-nnhnk\" (UID: \"ebd51311-0cf5-4c8c-9cca-343bfb33ff44\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-nnhnk"
Jul 28 10:24:37.734565 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:37.734396    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebd51311-0cf5-4c8c-9cca-343bfb33ff44-credential-socket\") pod \"router-gateway-1-openshift-default-6b46477c5b-nnhnk\" (UID: \"ebd51311-0cf5-4c8c-9cca-343bfb33ff44\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-nnhnk"
Jul 28 10:24:37.791332 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:37.791301    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:24:37.835430 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:37.835334    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebd51311-0cf5-4c8c-9cca-343bfb33ff44-istio-envoy\") pod \"router-gateway-1-openshift-default-6b46477c5b-nnhnk\" (UID: \"ebd51311-0cf5-4c8c-9cca-343bfb33ff44\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-nnhnk"
Jul 28 10:24:37.835430 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:37.835377    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebd51311-0cf5-4c8c-9cca-343bfb33ff44-istio-token\") pod \"router-gateway-1-openshift-default-6b46477c5b-nnhnk\" (UID: \"ebd51311-0cf5-4c8c-9cca-343bfb33ff44\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-nnhnk"
Jul 28 10:24:37.835430 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:37.835413    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebd51311-0cf5-4c8c-9cca-343bfb33ff44-workload-socket\") pod \"router-gateway-1-openshift-default-6b46477c5b-nnhnk\" (UID: \"ebd51311-0cf5-4c8c-9cca-343bfb33ff44\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-nnhnk"
Jul 28 10:24:37.835430 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:37.835433    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebd51311-0cf5-4c8c-9cca-343bfb33ff44-istio-data\") pod \"router-gateway-1-openshift-default-6b46477c5b-nnhnk\" (UID: \"ebd51311-0cf5-4c8c-9cca-343bfb33ff44\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-nnhnk"
Jul 28 10:24:37.835794 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:37.835463    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebd51311-0cf5-4c8c-9cca-343bfb33ff44-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-6b46477c5b-nnhnk\" (UID: \"ebd51311-0cf5-4c8c-9cca-343bfb33ff44\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-nnhnk"
Jul 28 10:24:37.835794 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:37.835490    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gx442\" (UniqueName: \"kubernetes.io/projected/ebd51311-0cf5-4c8c-9cca-343bfb33ff44-kube-api-access-gx442\") pod \"router-gateway-1-openshift-default-6b46477c5b-nnhnk\" (UID: \"ebd51311-0cf5-4c8c-9cca-343bfb33ff44\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-nnhnk"
Jul 28 10:24:37.835794 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:37.835716    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebd51311-0cf5-4c8c-9cca-343bfb33ff44-credential-socket\") pod \"router-gateway-1-openshift-default-6b46477c5b-nnhnk\" (UID: \"ebd51311-0cf5-4c8c-9cca-343bfb33ff44\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-nnhnk"
Jul 28 10:24:37.835951 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:37.835807    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebd51311-0cf5-4c8c-9cca-343bfb33ff44-istio-podinfo\") pod \"router-gateway-1-openshift-default-6b46477c5b-nnhnk\" (UID: \"ebd51311-0cf5-4c8c-9cca-343bfb33ff44\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-nnhnk"
Jul 28 10:24:37.836061 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:37.835949    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebd51311-0cf5-4c8c-9cca-343bfb33ff44-workload-socket\") pod \"router-gateway-1-openshift-default-6b46477c5b-nnhnk\" (UID: \"ebd51311-0cf5-4c8c-9cca-343bfb33ff44\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-nnhnk"
Jul 28 10:24:37.836061 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:37.835984    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebd51311-0cf5-4c8c-9cca-343bfb33ff44-credential-socket\") pod \"router-gateway-1-openshift-default-6b46477c5b-nnhnk\" (UID: \"ebd51311-0cf5-4c8c-9cca-343bfb33ff44\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-nnhnk"
Jul 28 10:24:37.836294 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:37.836268    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebd51311-0cf5-4c8c-9cca-343bfb33ff44-workload-certs\") pod \"router-gateway-1-openshift-default-6b46477c5b-nnhnk\" (UID: \"ebd51311-0cf5-4c8c-9cca-343bfb33ff44\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-nnhnk"
Jul 28 10:24:37.836294 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:37.836274    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebd51311-0cf5-4c8c-9cca-343bfb33ff44-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-6b46477c5b-nnhnk\" (UID: \"ebd51311-0cf5-4c8c-9cca-343bfb33ff44\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-nnhnk"
Jul 28 10:24:37.836495 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:37.836425    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebd51311-0cf5-4c8c-9cca-343bfb33ff44-istio-data\") pod \"router-gateway-1-openshift-default-6b46477c5b-nnhnk\" (UID: \"ebd51311-0cf5-4c8c-9cca-343bfb33ff44\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-nnhnk"
Jul 28 10:24:37.836625 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:37.836602    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebd51311-0cf5-4c8c-9cca-343bfb33ff44-workload-certs\") pod \"router-gateway-1-openshift-default-6b46477c5b-nnhnk\" (UID: \"ebd51311-0cf5-4c8c-9cca-343bfb33ff44\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-nnhnk"
Jul 28 10:24:37.837750 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:37.837731    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebd51311-0cf5-4c8c-9cca-343bfb33ff44-istio-envoy\") pod \"router-gateway-1-openshift-default-6b46477c5b-nnhnk\" (UID: \"ebd51311-0cf5-4c8c-9cca-343bfb33ff44\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-nnhnk"
Jul 28 10:24:37.838139 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:37.838122    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebd51311-0cf5-4c8c-9cca-343bfb33ff44-istio-podinfo\") pod \"router-gateway-1-openshift-default-6b46477c5b-nnhnk\" (UID: \"ebd51311-0cf5-4c8c-9cca-343bfb33ff44\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-nnhnk"
Jul 28 10:24:37.843016 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:37.842992    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebd51311-0cf5-4c8c-9cca-343bfb33ff44-istio-token\") pod \"router-gateway-1-openshift-default-6b46477c5b-nnhnk\" (UID: \"ebd51311-0cf5-4c8c-9cca-343bfb33ff44\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-nnhnk"
Jul 28 10:24:37.843151 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:37.843134    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gx442\" (UniqueName: \"kubernetes.io/projected/ebd51311-0cf5-4c8c-9cca-343bfb33ff44-kube-api-access-gx442\") pod \"router-gateway-1-openshift-default-6b46477c5b-nnhnk\" (UID: \"ebd51311-0cf5-4c8c-9cca-343bfb33ff44\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-nnhnk"
Jul 28 10:24:37.963484 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:37.963448    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-nnhnk"
Jul 28 10:24:38.111610 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:38.111577    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-nnhnk"]
Jul 28 10:24:38.113657 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:24:38.113626    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebd51311_0cf5_4c8c_9cca_343bfb33ff44.slice/crio-64840e9155611b004df32b1607a84c1dbd59e043d8ec1f1f35d08c07007a8ec4 WatchSource:0}: Error finding container 64840e9155611b004df32b1607a84c1dbd59e043d8ec1f1f35d08c07007a8ec4: Status 404 returned error can't find the container with id 64840e9155611b004df32b1607a84c1dbd59e043d8ec1f1f35d08c07007a8ec4
Jul 28 10:24:38.116228 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:38.116189    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 28 10:24:38.116351 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:38.116273    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 28 10:24:38.116351 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:38.116317    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 28 10:24:38.575680 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:38.575645    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-nnhnk" event={"ID":"ebd51311-0cf5-4c8c-9cca-343bfb33ff44","Type":"ContainerStarted","Data":"679518a0c570e77a7ebbe453c517a91251dcf325dc93ea4cd9d7bcae613070d6"}
Jul 28 10:24:38.576120 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:38.575697    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-nnhnk" event={"ID":"ebd51311-0cf5-4c8c-9cca-343bfb33ff44","Type":"ContainerStarted","Data":"64840e9155611b004df32b1607a84c1dbd59e043d8ec1f1f35d08c07007a8ec4"}
Jul 28 10:24:38.598108 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:38.598048    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-nnhnk" podStartSLOduration=1.598022864 podStartE2EDuration="1.598022864s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:24:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:24:38.597770788 +0000 UTC m=+1012.861167118" watchObservedRunningTime="2026-07-28 10:24:38.598022864 +0000 UTC m=+1012.861419198"
Jul 28 10:24:38.963898 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:38.963811    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-nnhnk"
Jul 28 10:24:39.970905 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:39.970871    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-nnhnk"
Jul 28 10:24:40.586149 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:40.586122    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-nnhnk"
Jul 28 10:24:40.587123 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:40.587098    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-nnhnk"
Jul 28 10:24:41.005357 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:41.005316    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:24:43.518898 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:43.518860    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:24:43.609135 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:43.609097    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-68fdf86f947d52k" podUID="b9557a90-dc91-4177-a83f-79a0b7526722" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/health\": dial tcp 10.134.0.33:8000: connect: connection refused"
Jul 28 10:24:49.350129 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:49.350097    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:24:49.451081 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:49.451047    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:24:49.625449 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:49.625363    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:24:51.511852 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:51.511814    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:24:51.622104 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:51.622068    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:24:51.723166 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:51.723129    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:24:51.823860 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:51.823829    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-zgkf6"]
Jul 28 10:24:51.827457 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:51.827434    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-zgkf6"
Jul 28 10:24:51.831021 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:51.830999    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"openshift-gw-ca-root-cert\""
Jul 28 10:24:51.831151 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:51.831048    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"kube-root-ca.crt\""
Jul 28 10:24:51.831387 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:51.831368    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"openshift-service-ca.crt\""
Jul 28 10:24:51.832633 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:51.832614    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"router-gateway-1-openshift-default-dockercfg-4996p\""
Jul 28 10:24:51.836944 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:51.836920    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:24:51.860561 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:51.845748    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-zgkf6"]
Jul 28 10:24:51.860561 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:51.859297    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cc65021-8b4a-44cd-95c5-bcc7d70f7f0a-istio-envoy\") pod \"router-gateway-1-openshift-default-7dcbd489bd-zgkf6\" (UID: \"0cc65021-8b4a-44cd-95c5-bcc7d70f7f0a\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-zgkf6"
Jul 28 10:24:51.860561 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:51.859378    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cc65021-8b4a-44cd-95c5-bcc7d70f7f0a-workload-certs\") pod \"router-gateway-1-openshift-default-7dcbd489bd-zgkf6\" (UID: \"0cc65021-8b4a-44cd-95c5-bcc7d70f7f0a\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-zgkf6"
Jul 28 10:24:51.860561 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:51.859406    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mt4mh\" (UniqueName: \"kubernetes.io/projected/0cc65021-8b4a-44cd-95c5-bcc7d70f7f0a-kube-api-access-mt4mh\") pod \"router-gateway-1-openshift-default-7dcbd489bd-zgkf6\" (UID: \"0cc65021-8b4a-44cd-95c5-bcc7d70f7f0a\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-zgkf6"
Jul 28 10:24:51.860561 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:51.859446    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cc65021-8b4a-44cd-95c5-bcc7d70f7f0a-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-7dcbd489bd-zgkf6\" (UID: \"0cc65021-8b4a-44cd-95c5-bcc7d70f7f0a\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-zgkf6"
Jul 28 10:24:51.860561 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:51.859499    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cc65021-8b4a-44cd-95c5-bcc7d70f7f0a-credential-socket\") pod \"router-gateway-1-openshift-default-7dcbd489bd-zgkf6\" (UID: \"0cc65021-8b4a-44cd-95c5-bcc7d70f7f0a\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-zgkf6"
Jul 28 10:24:51.860561 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:51.859524    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cc65021-8b4a-44cd-95c5-bcc7d70f7f0a-istio-podinfo\") pod \"router-gateway-1-openshift-default-7dcbd489bd-zgkf6\" (UID: \"0cc65021-8b4a-44cd-95c5-bcc7d70f7f0a\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-zgkf6"
Jul 28 10:24:51.860561 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:51.859568    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cc65021-8b4a-44cd-95c5-bcc7d70f7f0a-istio-token\") pod \"router-gateway-1-openshift-default-7dcbd489bd-zgkf6\" (UID: \"0cc65021-8b4a-44cd-95c5-bcc7d70f7f0a\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-zgkf6"
Jul 28 10:24:51.860561 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:51.859599    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cc65021-8b4a-44cd-95c5-bcc7d70f7f0a-istio-data\") pod \"router-gateway-1-openshift-default-7dcbd489bd-zgkf6\" (UID: \"0cc65021-8b4a-44cd-95c5-bcc7d70f7f0a\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-zgkf6"
Jul 28 10:24:51.860561 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:51.859629    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cc65021-8b4a-44cd-95c5-bcc7d70f7f0a-workload-socket\") pod \"router-gateway-1-openshift-default-7dcbd489bd-zgkf6\" (UID: \"0cc65021-8b4a-44cd-95c5-bcc7d70f7f0a\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-zgkf6"
Jul 28 10:24:51.907929 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:51.907892    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:24:51.960561 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:51.960324    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cc65021-8b4a-44cd-95c5-bcc7d70f7f0a-credential-socket\") pod \"router-gateway-1-openshift-default-7dcbd489bd-zgkf6\" (UID: \"0cc65021-8b4a-44cd-95c5-bcc7d70f7f0a\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-zgkf6"
Jul 28 10:24:51.960561 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:51.960390    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cc65021-8b4a-44cd-95c5-bcc7d70f7f0a-istio-podinfo\") pod \"router-gateway-1-openshift-default-7dcbd489bd-zgkf6\" (UID: \"0cc65021-8b4a-44cd-95c5-bcc7d70f7f0a\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-zgkf6"
Jul 28 10:24:51.960561 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:51.960420    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cc65021-8b4a-44cd-95c5-bcc7d70f7f0a-istio-token\") pod \"router-gateway-1-openshift-default-7dcbd489bd-zgkf6\" (UID: \"0cc65021-8b4a-44cd-95c5-bcc7d70f7f0a\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-zgkf6"
Jul 28 10:24:51.960561 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:51.960464    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cc65021-8b4a-44cd-95c5-bcc7d70f7f0a-istio-data\") pod \"router-gateway-1-openshift-default-7dcbd489bd-zgkf6\" (UID: \"0cc65021-8b4a-44cd-95c5-bcc7d70f7f0a\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-zgkf6"
Jul 28 10:24:51.960561 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:51.960497    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cc65021-8b4a-44cd-95c5-bcc7d70f7f0a-workload-socket\") pod \"router-gateway-1-openshift-default-7dcbd489bd-zgkf6\" (UID: \"0cc65021-8b4a-44cd-95c5-bcc7d70f7f0a\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-zgkf6"
Jul 28 10:24:51.960561 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:51.960572    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cc65021-8b4a-44cd-95c5-bcc7d70f7f0a-istio-envoy\") pod \"router-gateway-1-openshift-default-7dcbd489bd-zgkf6\" (UID: \"0cc65021-8b4a-44cd-95c5-bcc7d70f7f0a\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-zgkf6"
Jul 28 10:24:51.961052 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:51.960658    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cc65021-8b4a-44cd-95c5-bcc7d70f7f0a-workload-certs\") pod \"router-gateway-1-openshift-default-7dcbd489bd-zgkf6\" (UID: \"0cc65021-8b4a-44cd-95c5-bcc7d70f7f0a\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-zgkf6"
Jul 28 10:24:51.961052 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:51.960683    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mt4mh\" (UniqueName: \"kubernetes.io/projected/0cc65021-8b4a-44cd-95c5-bcc7d70f7f0a-kube-api-access-mt4mh\") pod \"router-gateway-1-openshift-default-7dcbd489bd-zgkf6\" (UID: \"0cc65021-8b4a-44cd-95c5-bcc7d70f7f0a\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-zgkf6"
Jul 28 10:24:51.961052 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:51.960742    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cc65021-8b4a-44cd-95c5-bcc7d70f7f0a-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-7dcbd489bd-zgkf6\" (UID: \"0cc65021-8b4a-44cd-95c5-bcc7d70f7f0a\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-zgkf6"
Jul 28 10:24:51.961563 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:51.961330    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cc65021-8b4a-44cd-95c5-bcc7d70f7f0a-credential-socket\") pod \"router-gateway-1-openshift-default-7dcbd489bd-zgkf6\" (UID: \"0cc65021-8b4a-44cd-95c5-bcc7d70f7f0a\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-zgkf6"
Jul 28 10:24:51.961650 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:51.961623    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cc65021-8b4a-44cd-95c5-bcc7d70f7f0a-workload-socket\") pod \"router-gateway-1-openshift-default-7dcbd489bd-zgkf6\" (UID: \"0cc65021-8b4a-44cd-95c5-bcc7d70f7f0a\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-zgkf6"
Jul 28 10:24:51.962272 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:51.961745    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cc65021-8b4a-44cd-95c5-bcc7d70f7f0a-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-7dcbd489bd-zgkf6\" (UID: \"0cc65021-8b4a-44cd-95c5-bcc7d70f7f0a\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-zgkf6"
Jul 28 10:24:51.962272 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:51.961929    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cc65021-8b4a-44cd-95c5-bcc7d70f7f0a-workload-certs\") pod \"router-gateway-1-openshift-default-7dcbd489bd-zgkf6\" (UID: \"0cc65021-8b4a-44cd-95c5-bcc7d70f7f0a\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-zgkf6"
Jul 28 10:24:51.962272 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:51.962203    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cc65021-8b4a-44cd-95c5-bcc7d70f7f0a-istio-data\") pod \"router-gateway-1-openshift-default-7dcbd489bd-zgkf6\" (UID: \"0cc65021-8b4a-44cd-95c5-bcc7d70f7f0a\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-zgkf6"
Jul 28 10:24:51.968940 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:51.968885    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cc65021-8b4a-44cd-95c5-bcc7d70f7f0a-istio-podinfo\") pod \"router-gateway-1-openshift-default-7dcbd489bd-zgkf6\" (UID: \"0cc65021-8b4a-44cd-95c5-bcc7d70f7f0a\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-zgkf6"
Jul 28 10:24:51.972461 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:51.972437    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cc65021-8b4a-44cd-95c5-bcc7d70f7f0a-istio-envoy\") pod \"router-gateway-1-openshift-default-7dcbd489bd-zgkf6\" (UID: \"0cc65021-8b4a-44cd-95c5-bcc7d70f7f0a\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-zgkf6"
Jul 28 10:24:51.975461 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:51.975432    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cc65021-8b4a-44cd-95c5-bcc7d70f7f0a-istio-token\") pod \"router-gateway-1-openshift-default-7dcbd489bd-zgkf6\" (UID: \"0cc65021-8b4a-44cd-95c5-bcc7d70f7f0a\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-zgkf6"
Jul 28 10:24:51.976328 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:51.976300    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mt4mh\" (UniqueName: \"kubernetes.io/projected/0cc65021-8b4a-44cd-95c5-bcc7d70f7f0a-kube-api-access-mt4mh\") pod \"router-gateway-1-openshift-default-7dcbd489bd-zgkf6\" (UID: \"0cc65021-8b4a-44cd-95c5-bcc7d70f7f0a\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-zgkf6"
Jul 28 10:24:52.139219 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:52.139120    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-zgkf6"
Jul 28 10:24:52.331133 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:52.331103    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-zgkf6"]
Jul 28 10:24:52.332943 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:24:52.332915    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cc65021_8b4a_44cd_95c5_bcc7d70f7f0a.slice/crio-461d3f0c9805f1fd70ccf2cad50d0859a28d4f316cc125d90a033e99136dcb2f WatchSource:0}: Error finding container 461d3f0c9805f1fd70ccf2cad50d0859a28d4f316cc125d90a033e99136dcb2f: Status 404 returned error can't find the container with id 461d3f0c9805f1fd70ccf2cad50d0859a28d4f316cc125d90a033e99136dcb2f
Jul 28 10:24:52.334979 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:52.334945    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 28 10:24:52.335062 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:52.335013    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 28 10:24:52.335062 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:52.335039    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 28 10:24:52.648445 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:52.648347    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-zgkf6" event={"ID":"0cc65021-8b4a-44cd-95c5-bcc7d70f7f0a","Type":"ContainerStarted","Data":"be7e4e89101e6926c32fbdc1b5a7552479fdfda7006e72198dfa23fa5177f4e5"}
Jul 28 10:24:52.649035 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:52.649008    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-zgkf6" event={"ID":"0cc65021-8b4a-44cd-95c5-bcc7d70f7f0a","Type":"ContainerStarted","Data":"461d3f0c9805f1fd70ccf2cad50d0859a28d4f316cc125d90a033e99136dcb2f"}
Jul 28 10:24:52.669742 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:52.669675    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-zgkf6" podStartSLOduration=1.669654318 podStartE2EDuration="1.669654318s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:24:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:24:52.668229816 +0000 UTC m=+1026.931626156" watchObservedRunningTime="2026-07-28 10:24:52.669654318 +0000 UTC m=+1026.933050649"
Jul 28 10:24:53.140345 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:53.140309    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-zgkf6"
Jul 28 10:24:53.160967 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:53.160927    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-zgkf6"
Jul 28 10:24:53.618918 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:53.618885    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-68fdf86f947d52k"
Jul 28 10:24:53.627033 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:53.627006    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-68fdf86f947d52k"
Jul 28 10:24:53.657373 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:53.657335    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-zgkf6"
Jul 28 10:24:53.666397 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:53.666361    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-zgkf6"
Jul 28 10:24:55.175230 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:55.175141    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:24:56.495335 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:56.495296    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-nnhnk"]
Jul 28 10:24:56.495767 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:56.495533    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-nnhnk" podUID="ebd51311-0cf5-4c8c-9cca-343bfb33ff44" containerName="istio-proxy" containerID="cri-o://679518a0c570e77a7ebbe453c517a91251dcf325dc93ea4cd9d7bcae613070d6" gracePeriod=30
Jul 28 10:24:56.591089 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:56.591048    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:24:58.996311 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:58.996278    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:24:59.698753 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:59.698726    2579 generic.go:356] "Generic (PLEG): container finished" podID="ebd51311-0cf5-4c8c-9cca-343bfb33ff44" containerID="679518a0c570e77a7ebbe453c517a91251dcf325dc93ea4cd9d7bcae613070d6" exitCode=0
Jul 28 10:24:59.698943 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:59.698806    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-nnhnk" event={"ID":"ebd51311-0cf5-4c8c-9cca-343bfb33ff44","Type":"ContainerDied","Data":"679518a0c570e77a7ebbe453c517a91251dcf325dc93ea4cd9d7bcae613070d6"}
Jul 28 10:24:59.759589 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:59.759559    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-nnhnk"
Jul 28 10:24:59.836036 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:59.835959    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebd51311-0cf5-4c8c-9cca-343bfb33ff44-istio-data\") pod \"ebd51311-0cf5-4c8c-9cca-343bfb33ff44\" (UID: \"ebd51311-0cf5-4c8c-9cca-343bfb33ff44\") "
Jul 28 10:24:59.836179 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:59.836035    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebd51311-0cf5-4c8c-9cca-343bfb33ff44-istio-token\") pod \"ebd51311-0cf5-4c8c-9cca-343bfb33ff44\" (UID: \"ebd51311-0cf5-4c8c-9cca-343bfb33ff44\") "
Jul 28 10:24:59.836179 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:59.836074    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebd51311-0cf5-4c8c-9cca-343bfb33ff44-istiod-ca-cert\") pod \"ebd51311-0cf5-4c8c-9cca-343bfb33ff44\" (UID: \"ebd51311-0cf5-4c8c-9cca-343bfb33ff44\") "
Jul 28 10:24:59.836179 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:59.836109    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebd51311-0cf5-4c8c-9cca-343bfb33ff44-workload-socket\") pod \"ebd51311-0cf5-4c8c-9cca-343bfb33ff44\" (UID: \"ebd51311-0cf5-4c8c-9cca-343bfb33ff44\") "
Jul 28 10:24:59.836179 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:59.836146    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebd51311-0cf5-4c8c-9cca-343bfb33ff44-istio-podinfo\") pod \"ebd51311-0cf5-4c8c-9cca-343bfb33ff44\" (UID: \"ebd51311-0cf5-4c8c-9cca-343bfb33ff44\") "
Jul 28 10:24:59.836359 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:59.836210    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebd51311-0cf5-4c8c-9cca-343bfb33ff44-istio-envoy\") pod \"ebd51311-0cf5-4c8c-9cca-343bfb33ff44\" (UID: \"ebd51311-0cf5-4c8c-9cca-343bfb33ff44\") "
Jul 28 10:24:59.836359 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:59.836234    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebd51311-0cf5-4c8c-9cca-343bfb33ff44-istio-data" (OuterVolumeSpecName: "istio-data") pod "ebd51311-0cf5-4c8c-9cca-343bfb33ff44" (UID: "ebd51311-0cf5-4c8c-9cca-343bfb33ff44"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:24:59.836359 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:59.836254    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebd51311-0cf5-4c8c-9cca-343bfb33ff44-workload-certs\") pod \"ebd51311-0cf5-4c8c-9cca-343bfb33ff44\" (UID: \"ebd51311-0cf5-4c8c-9cca-343bfb33ff44\") "
Jul 28 10:24:59.836359 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:59.836286    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gx442\" (UniqueName: \"kubernetes.io/projected/ebd51311-0cf5-4c8c-9cca-343bfb33ff44-kube-api-access-gx442\") pod \"ebd51311-0cf5-4c8c-9cca-343bfb33ff44\" (UID: \"ebd51311-0cf5-4c8c-9cca-343bfb33ff44\") "
Jul 28 10:24:59.836359 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:59.836320    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebd51311-0cf5-4c8c-9cca-343bfb33ff44-credential-socket\") pod \"ebd51311-0cf5-4c8c-9cca-343bfb33ff44\" (UID: \"ebd51311-0cf5-4c8c-9cca-343bfb33ff44\") "
Jul 28 10:24:59.836587 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:59.836517    2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebd51311-0cf5-4c8c-9cca-343bfb33ff44-istio-data\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 10:24:59.836645 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:59.836597    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebd51311-0cf5-4c8c-9cca-343bfb33ff44-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ebd51311-0cf5-4c8c-9cca-343bfb33ff44" (UID: "ebd51311-0cf5-4c8c-9cca-343bfb33ff44"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:24:59.836830 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:59.836766    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebd51311-0cf5-4c8c-9cca-343bfb33ff44-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ebd51311-0cf5-4c8c-9cca-343bfb33ff44" (UID: "ebd51311-0cf5-4c8c-9cca-343bfb33ff44"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:24:59.836830 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:59.836749    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebd51311-0cf5-4c8c-9cca-343bfb33ff44-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ebd51311-0cf5-4c8c-9cca-343bfb33ff44" (UID: "ebd51311-0cf5-4c8c-9cca-343bfb33ff44"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:24:59.837009 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:59.836839    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebd51311-0cf5-4c8c-9cca-343bfb33ff44-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ebd51311-0cf5-4c8c-9cca-343bfb33ff44" (UID: "ebd51311-0cf5-4c8c-9cca-343bfb33ff44"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:24:59.838818 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:59.838775    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebd51311-0cf5-4c8c-9cca-343bfb33ff44-istio-token" (OuterVolumeSpecName: "istio-token") pod "ebd51311-0cf5-4c8c-9cca-343bfb33ff44" (UID: "ebd51311-0cf5-4c8c-9cca-343bfb33ff44"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:24:59.838936 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:59.838861    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebd51311-0cf5-4c8c-9cca-343bfb33ff44-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ebd51311-0cf5-4c8c-9cca-343bfb33ff44" (UID: "ebd51311-0cf5-4c8c-9cca-343bfb33ff44"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:24:59.838936 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:59.838881    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebd51311-0cf5-4c8c-9cca-343bfb33ff44-kube-api-access-gx442" (OuterVolumeSpecName: "kube-api-access-gx442") pod "ebd51311-0cf5-4c8c-9cca-343bfb33ff44" (UID: "ebd51311-0cf5-4c8c-9cca-343bfb33ff44"). InnerVolumeSpecName "kube-api-access-gx442". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:24:59.839050 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:59.838944    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ebd51311-0cf5-4c8c-9cca-343bfb33ff44-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ebd51311-0cf5-4c8c-9cca-343bfb33ff44" (UID: "ebd51311-0cf5-4c8c-9cca-343bfb33ff44"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 28 10:24:59.937589 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:59.937533    2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebd51311-0cf5-4c8c-9cca-343bfb33ff44-istio-envoy\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 10:24:59.937589 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:59.937582    2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebd51311-0cf5-4c8c-9cca-343bfb33ff44-workload-certs\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 10:24:59.937589 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:59.937592    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gx442\" (UniqueName: \"kubernetes.io/projected/ebd51311-0cf5-4c8c-9cca-343bfb33ff44-kube-api-access-gx442\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 10:24:59.937819 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:59.937603    2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebd51311-0cf5-4c8c-9cca-343bfb33ff44-credential-socket\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 10:24:59.937819 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:59.937612    2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebd51311-0cf5-4c8c-9cca-343bfb33ff44-istio-token\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 10:24:59.937819 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:59.937620    2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebd51311-0cf5-4c8c-9cca-343bfb33ff44-istiod-ca-cert\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 10:24:59.937819 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:59.937627    2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebd51311-0cf5-4c8c-9cca-343bfb33ff44-workload-socket\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 10:24:59.937819 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:24:59.937634    2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebd51311-0cf5-4c8c-9cca-343bfb33ff44-istio-podinfo\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 10:25:00.705450 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:25:00.705359    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-nnhnk" event={"ID":"ebd51311-0cf5-4c8c-9cca-343bfb33ff44","Type":"ContainerDied","Data":"64840e9155611b004df32b1607a84c1dbd59e043d8ec1f1f35d08c07007a8ec4"}
Jul 28 10:25:00.705450 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:25:00.705406    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-nnhnk"
Jul 28 10:25:00.705450 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:25:00.705420    2579 scope.go:117] "RemoveContainer" containerID="679518a0c570e77a7ebbe453c517a91251dcf325dc93ea4cd9d7bcae613070d6"
Jul 28 10:25:00.721771 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:25:00.721747    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-nnhnk"]
Jul 28 10:25:00.726628 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:25:00.726606    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-nnhnk"]
Jul 28 10:25:01.422746 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:25:01.422712    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:25:01.480642 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:25:01.480597    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-68fdf86f947d52k"]
Jul 28 10:25:01.507383 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:25:01.507352    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:25:01.648583 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:25:01.648533    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:25:01.998277 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:25:01.998243    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:25:02.275749 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:25:02.275672    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ebd51311-0cf5-4c8c-9cca-343bfb33ff44" path="/var/lib/kubelet/pods/ebd51311-0cf5-4c8c-9cca-343bfb33ff44/volumes"
Jul 28 10:25:02.333239 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:25:02.333204    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:25:03.459562 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:25:03.458405    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:25:03.533620 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:25:03.533566    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:25:03.599671 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:25:03.599586    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:25:06.393413 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:25:06.393330    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-zgkf6"]
Jul 28 10:25:06.393854 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:25:06.393553    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-zgkf6" podUID="0cc65021-8b4a-44cd-95c5-bcc7d70f7f0a" containerName="istio-proxy" containerID="cri-o://be7e4e89101e6926c32fbdc1b5a7552479fdfda7006e72198dfa23fa5177f4e5" gracePeriod=30
Jul 28 10:25:06.435146 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:25:06.435108    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:25:06.493273 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:25:06.493232    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:25:08.810677 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:25:08.810616    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-zgkf6" podUID="0cc65021-8b4a-44cd-95c5-bcc7d70f7f0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": read tcp 10.134.0.2:47110->10.134.0.37:15021: read: connection reset by peer"
Jul 28 10:25:09.044672 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:25:09.044647    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-zgkf6"
Jul 28 10:25:09.136029 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:25:09.136004    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cc65021-8b4a-44cd-95c5-bcc7d70f7f0a-istio-token\") pod \"0cc65021-8b4a-44cd-95c5-bcc7d70f7f0a\" (UID: \"0cc65021-8b4a-44cd-95c5-bcc7d70f7f0a\") "
Jul 28 10:25:09.136149 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:25:09.136046    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cc65021-8b4a-44cd-95c5-bcc7d70f7f0a-istio-envoy\") pod \"0cc65021-8b4a-44cd-95c5-bcc7d70f7f0a\" (UID: \"0cc65021-8b4a-44cd-95c5-bcc7d70f7f0a\") "
Jul 28 10:25:09.136149 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:25:09.136079    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mt4mh\" (UniqueName: \"kubernetes.io/projected/0cc65021-8b4a-44cd-95c5-bcc7d70f7f0a-kube-api-access-mt4mh\") pod \"0cc65021-8b4a-44cd-95c5-bcc7d70f7f0a\" (UID: \"0cc65021-8b4a-44cd-95c5-bcc7d70f7f0a\") "
Jul 28 10:25:09.136149 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:25:09.136098    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cc65021-8b4a-44cd-95c5-bcc7d70f7f0a-istio-data\") pod \"0cc65021-8b4a-44cd-95c5-bcc7d70f7f0a\" (UID: \"0cc65021-8b4a-44cd-95c5-bcc7d70f7f0a\") "
Jul 28 10:25:09.136298 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:25:09.136169    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cc65021-8b4a-44cd-95c5-bcc7d70f7f0a-credential-socket\") pod \"0cc65021-8b4a-44cd-95c5-bcc7d70f7f0a\" (UID: \"0cc65021-8b4a-44cd-95c5-bcc7d70f7f0a\") "
Jul 28 10:25:09.136298 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:25:09.136219    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cc65021-8b4a-44cd-95c5-bcc7d70f7f0a-workload-certs\") pod \"0cc65021-8b4a-44cd-95c5-bcc7d70f7f0a\" (UID: \"0cc65021-8b4a-44cd-95c5-bcc7d70f7f0a\") "
Jul 28 10:25:09.136298 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:25:09.136259    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cc65021-8b4a-44cd-95c5-bcc7d70f7f0a-istiod-ca-cert\") pod \"0cc65021-8b4a-44cd-95c5-bcc7d70f7f0a\" (UID: \"0cc65021-8b4a-44cd-95c5-bcc7d70f7f0a\") "
Jul 28 10:25:09.136298 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:25:09.136283    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cc65021-8b4a-44cd-95c5-bcc7d70f7f0a-istio-podinfo\") pod \"0cc65021-8b4a-44cd-95c5-bcc7d70f7f0a\" (UID: \"0cc65021-8b4a-44cd-95c5-bcc7d70f7f0a\") "
Jul 28 10:25:09.136487 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:25:09.136308    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cc65021-8b4a-44cd-95c5-bcc7d70f7f0a-workload-socket\") pod \"0cc65021-8b4a-44cd-95c5-bcc7d70f7f0a\" (UID: \"0cc65021-8b4a-44cd-95c5-bcc7d70f7f0a\") "
Jul 28 10:25:09.136487 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:25:09.136442    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cc65021-8b4a-44cd-95c5-bcc7d70f7f0a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0cc65021-8b4a-44cd-95c5-bcc7d70f7f0a" (UID: "0cc65021-8b4a-44cd-95c5-bcc7d70f7f0a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:25:09.136487 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:25:09.136475    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cc65021-8b4a-44cd-95c5-bcc7d70f7f0a-istio-data" (OuterVolumeSpecName: "istio-data") pod "0cc65021-8b4a-44cd-95c5-bcc7d70f7f0a" (UID: "0cc65021-8b4a-44cd-95c5-bcc7d70f7f0a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:25:09.136653 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:25:09.136641    2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cc65021-8b4a-44cd-95c5-bcc7d70f7f0a-istio-data\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 10:25:09.136708 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:25:09.136662    2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cc65021-8b4a-44cd-95c5-bcc7d70f7f0a-credential-socket\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 10:25:09.136763 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:25:09.136715    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cc65021-8b4a-44cd-95c5-bcc7d70f7f0a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0cc65021-8b4a-44cd-95c5-bcc7d70f7f0a" (UID: "0cc65021-8b4a-44cd-95c5-bcc7d70f7f0a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:25:09.136763 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:25:09.136745    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cc65021-8b4a-44cd-95c5-bcc7d70f7f0a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0cc65021-8b4a-44cd-95c5-bcc7d70f7f0a" (UID: "0cc65021-8b4a-44cd-95c5-bcc7d70f7f0a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:25:09.136884 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:25:09.136857    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cc65021-8b4a-44cd-95c5-bcc7d70f7f0a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0cc65021-8b4a-44cd-95c5-bcc7d70f7f0a" (UID: "0cc65021-8b4a-44cd-95c5-bcc7d70f7f0a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:25:09.138434 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:25:09.138407    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cc65021-8b4a-44cd-95c5-bcc7d70f7f0a-kube-api-access-mt4mh" (OuterVolumeSpecName: "kube-api-access-mt4mh") pod "0cc65021-8b4a-44cd-95c5-bcc7d70f7f0a" (UID: "0cc65021-8b4a-44cd-95c5-bcc7d70f7f0a"). InnerVolumeSpecName "kube-api-access-mt4mh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:25:09.138564 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:25:09.138499    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cc65021-8b4a-44cd-95c5-bcc7d70f7f0a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0cc65021-8b4a-44cd-95c5-bcc7d70f7f0a" (UID: "0cc65021-8b4a-44cd-95c5-bcc7d70f7f0a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:25:09.138651 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:25:09.138623    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0cc65021-8b4a-44cd-95c5-bcc7d70f7f0a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0cc65021-8b4a-44cd-95c5-bcc7d70f7f0a" (UID: "0cc65021-8b4a-44cd-95c5-bcc7d70f7f0a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 28 10:25:09.138693 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:25:09.138629    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cc65021-8b4a-44cd-95c5-bcc7d70f7f0a-istio-token" (OuterVolumeSpecName: "istio-token") pod "0cc65021-8b4a-44cd-95c5-bcc7d70f7f0a" (UID: "0cc65021-8b4a-44cd-95c5-bcc7d70f7f0a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:25:09.237245 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:25:09.237201    2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cc65021-8b4a-44cd-95c5-bcc7d70f7f0a-istiod-ca-cert\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 10:25:09.237245 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:25:09.237236    2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cc65021-8b4a-44cd-95c5-bcc7d70f7f0a-istio-podinfo\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 10:25:09.237245 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:25:09.237245    2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cc65021-8b4a-44cd-95c5-bcc7d70f7f0a-workload-socket\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 10:25:09.237245 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:25:09.237253    2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cc65021-8b4a-44cd-95c5-bcc7d70f7f0a-istio-token\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 10:25:09.237785 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:25:09.237262    2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cc65021-8b4a-44cd-95c5-bcc7d70f7f0a-istio-envoy\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 10:25:09.237785 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:25:09.237272    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mt4mh\" (UniqueName: \"kubernetes.io/projected/0cc65021-8b4a-44cd-95c5-bcc7d70f7f0a-kube-api-access-mt4mh\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 10:25:09.237785 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:25:09.237283    2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cc65021-8b4a-44cd-95c5-bcc7d70f7f0a-workload-certs\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 10:25:09.753166 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:25:09.753126    2579 generic.go:356] "Generic (PLEG): container finished" podID="0cc65021-8b4a-44cd-95c5-bcc7d70f7f0a" containerID="be7e4e89101e6926c32fbdc1b5a7552479fdfda7006e72198dfa23fa5177f4e5" exitCode=0
Jul 28 10:25:09.753166 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:25:09.753170    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-zgkf6"
Jul 28 10:25:09.753396 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:25:09.753208    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-zgkf6" event={"ID":"0cc65021-8b4a-44cd-95c5-bcc7d70f7f0a","Type":"ContainerDied","Data":"be7e4e89101e6926c32fbdc1b5a7552479fdfda7006e72198dfa23fa5177f4e5"}
Jul 28 10:25:09.753396 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:25:09.753259    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-zgkf6" event={"ID":"0cc65021-8b4a-44cd-95c5-bcc7d70f7f0a","Type":"ContainerDied","Data":"461d3f0c9805f1fd70ccf2cad50d0859a28d4f316cc125d90a033e99136dcb2f"}
Jul 28 10:25:09.753396 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:25:09.753281    2579 scope.go:117] "RemoveContainer" containerID="be7e4e89101e6926c32fbdc1b5a7552479fdfda7006e72198dfa23fa5177f4e5"
Jul 28 10:25:09.764327 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:25:09.763736    2579 scope.go:117] "RemoveContainer" containerID="be7e4e89101e6926c32fbdc1b5a7552479fdfda7006e72198dfa23fa5177f4e5"
Jul 28 10:25:09.764631 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:25:09.764608    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be7e4e89101e6926c32fbdc1b5a7552479fdfda7006e72198dfa23fa5177f4e5\": container with ID starting with be7e4e89101e6926c32fbdc1b5a7552479fdfda7006e72198dfa23fa5177f4e5 not found: ID does not exist" containerID="be7e4e89101e6926c32fbdc1b5a7552479fdfda7006e72198dfa23fa5177f4e5"
Jul 28 10:25:09.764728 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:25:09.764639    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be7e4e89101e6926c32fbdc1b5a7552479fdfda7006e72198dfa23fa5177f4e5"} err="failed to get container status \"be7e4e89101e6926c32fbdc1b5a7552479fdfda7006e72198dfa23fa5177f4e5\": rpc error: code = NotFound desc = could not find container \"be7e4e89101e6926c32fbdc1b5a7552479fdfda7006e72198dfa23fa5177f4e5\": container with ID starting with be7e4e89101e6926c32fbdc1b5a7552479fdfda7006e72198dfa23fa5177f4e5 not found: ID does not exist"
Jul 28 10:25:09.775369 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:25:09.775347    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-zgkf6"]
Jul 28 10:25:09.781825 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:25:09.781803    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-zgkf6"]
Jul 28 10:25:10.276455 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:25:10.276420    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0cc65021-8b4a-44cd-95c5-bcc7d70f7f0a" path="/var/lib/kubelet/pods/0cc65021-8b4a-44cd-95c5-bcc7d70f7f0a/volumes"
Jul 28 10:25:11.514112 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:25:11.514079    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:25:12.372208 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:25:12.372170    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:25:13.946423 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:25:13.946388    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:25:14.515500 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:25:14.515452    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:25:14.754719 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:25:14.754680    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-5nmb8"]
Jul 28 10:25:14.755085 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:25:14.755066    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebd51311-0cf5-4c8c-9cca-343bfb33ff44" containerName="istio-proxy"
Jul 28 10:25:14.755172 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:25:14.755086    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebd51311-0cf5-4c8c-9cca-343bfb33ff44" containerName="istio-proxy"
Jul 28 10:25:14.755172 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:25:14.755129    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0cc65021-8b4a-44cd-95c5-bcc7d70f7f0a" containerName="istio-proxy"
Jul 28 10:25:14.755172 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:25:14.755139    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="0cc65021-8b4a-44cd-95c5-bcc7d70f7f0a" containerName="istio-proxy"
Jul 28 10:25:14.755322 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:25:14.755214    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ebd51311-0cf5-4c8c-9cca-343bfb33ff44" containerName="istio-proxy"
Jul 28 10:25:14.755322 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:25:14.755227    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="0cc65021-8b4a-44cd-95c5-bcc7d70f7f0a" containerName="istio-proxy"
Jul 28 10:25:14.757250 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:25:14.757228    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-5nmb8"
Jul 28 10:25:14.760012 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:25:14.759989    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"openshift-service-ca.crt\""
Jul 28 10:25:14.760177 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:25:14.760031    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"auth-enabled-test-epp-sa-dockercfg-z9qld\""
Jul 28 10:25:14.760177 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:25:14.760031    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"auth-enabled-test-kserve-self-signed-certs\""
Jul 28 10:25:14.760177 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:25:14.760063    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"kube-root-ca.crt\""
Jul 28 10:25:14.760177 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:25:14.760083    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"default-dockercfg-977dm\""
Jul 28 10:25:14.766211 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:25:14.766156    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-5nmb8"]
Jul 28 10:25:14.871553 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:25:14.871504    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:25:14.885514 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:25:14.885481    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9p7j\" (UniqueName: \"kubernetes.io/projected/9bbe32a0-9b8e-4546-b3c1-d2e8ea4ec8f7-kube-api-access-q9p7j\") pod \"auth-enabled-test-kserve-router-scheduler-5dbb5856d8-5nmb8\" (UID: \"9bbe32a0-9b8e-4546-b3c1-d2e8ea4ec8f7\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-5nmb8"
Jul 28 10:25:14.885753 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:25:14.885533    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9bbe32a0-9b8e-4546-b3c1-d2e8ea4ec8f7-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-5dbb5856d8-5nmb8\" (UID: \"9bbe32a0-9b8e-4546-b3c1-d2e8ea4ec8f7\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-5nmb8"
Jul 28 10:25:14.986880 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:25:14.986837    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9p7j\" (UniqueName: \"kubernetes.io/projected/9bbe32a0-9b8e-4546-b3c1-d2e8ea4ec8f7-kube-api-access-q9p7j\") pod \"auth-enabled-test-kserve-router-scheduler-5dbb5856d8-5nmb8\" (UID: \"9bbe32a0-9b8e-4546-b3c1-d2e8ea4ec8f7\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-5nmb8"
Jul 28 10:25:14.987317 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:25:14.986895    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9bbe32a0-9b8e-4546-b3c1-d2e8ea4ec8f7-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-5dbb5856d8-5nmb8\" (UID: \"9bbe32a0-9b8e-4546-b3c1-d2e8ea4ec8f7\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-5nmb8"
Jul 28 10:25:14.989522 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:25:14.989497    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9bbe32a0-9b8e-4546-b3c1-d2e8ea4ec8f7-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-5dbb5856d8-5nmb8\" (UID: \"9bbe32a0-9b8e-4546-b3c1-d2e8ea4ec8f7\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-5nmb8"
Jul 28 10:25:14.995489 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:25:14.995464    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9p7j\" (UniqueName: \"kubernetes.io/projected/9bbe32a0-9b8e-4546-b3c1-d2e8ea4ec8f7-kube-api-access-q9p7j\") pod \"auth-enabled-test-kserve-router-scheduler-5dbb5856d8-5nmb8\" (UID: \"9bbe32a0-9b8e-4546-b3c1-d2e8ea4ec8f7\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-5nmb8"
Jul 28 10:25:15.068675 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:25:15.068638    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-5nmb8"
Jul 28 10:25:15.205060 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:25:15.205031    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-5nmb8"]
Jul 28 10:25:15.206958 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:25:15.206926    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bbe32a0_9b8e_4546_b3c1_d2e8ea4ec8f7.slice/crio-1a6446c79d78f0d372d5ff5aa7d07e2d4c18e1a0b9e24c4b1fbeb0d614761d29 WatchSource:0}: Error finding container 1a6446c79d78f0d372d5ff5aa7d07e2d4c18e1a0b9e24c4b1fbeb0d614761d29: Status 404 returned error can't find the container with id 1a6446c79d78f0d372d5ff5aa7d07e2d4c18e1a0b9e24c4b1fbeb0d614761d29
Jul 28 10:25:15.785735 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:25:15.785702    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-5nmb8" event={"ID":"9bbe32a0-9b8e-4546-b3c1-d2e8ea4ec8f7","Type":"ContainerStarted","Data":"fdabfeb18e06aa0df1f750ab76c873e8704e4512b284cbb84b0443a06acb9b2e"}
Jul 28 10:25:15.785922 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:25:15.785756    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-5nmb8" event={"ID":"9bbe32a0-9b8e-4546-b3c1-d2e8ea4ec8f7","Type":"ContainerStarted","Data":"1a6446c79d78f0d372d5ff5aa7d07e2d4c18e1a0b9e24c4b1fbeb0d614761d29"}
Jul 28 10:25:15.785922 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:25:15.785783    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-5nmb8"
Jul 28 10:25:15.802130 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:25:15.802074    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-5nmb8" podStartSLOduration=1.8020582950000001 podStartE2EDuration="1.802058295s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:25:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:25:15.80046483 +0000 UTC m=+1050.063861159" watchObservedRunningTime="2026-07-28 10:25:15.802058295 +0000 UTC m=+1050.065454624"
Jul 28 10:25:16.502790 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:25:16.502745    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-68fdf86f947d52k" podUID="b9557a90-dc91-4177-a83f-79a0b7526722" containerName="main" containerID="cri-o://2324aafa4c6be305b1ca666363c8ae73fdbe41cc0d81e11c3dc59308759befd0" gracePeriod=45
Jul 28 10:25:16.619845 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:25:16.619800    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-68fdf86f947d52k" podUID="b9557a90-dc91-4177-a83f-79a0b7526722" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/health\": dial tcp 10.134.0.33:8000: connect: connection refused"
Jul 28 10:25:17.256344 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:25:17.256319    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-68fdf86f947d52k"
Jul 28 10:25:17.407069 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:25:17.407017    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xzs6w\" (UniqueName: \"kubernetes.io/projected/b9557a90-dc91-4177-a83f-79a0b7526722-kube-api-access-xzs6w\") pod \"b9557a90-dc91-4177-a83f-79a0b7526722\" (UID: \"b9557a90-dc91-4177-a83f-79a0b7526722\") "
Jul 28 10:25:17.407069 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:25:17.407076    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b9557a90-dc91-4177-a83f-79a0b7526722-kserve-provision-location\") pod \"b9557a90-dc91-4177-a83f-79a0b7526722\" (UID: \"b9557a90-dc91-4177-a83f-79a0b7526722\") "
Jul 28 10:25:17.407321 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:25:17.407104    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9557a90-dc91-4177-a83f-79a0b7526722-model-cache\") pod \"b9557a90-dc91-4177-a83f-79a0b7526722\" (UID: \"b9557a90-dc91-4177-a83f-79a0b7526722\") "
Jul 28 10:25:17.407321 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:25:17.407158    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9557a90-dc91-4177-a83f-79a0b7526722-tls-certs\") pod \"b9557a90-dc91-4177-a83f-79a0b7526722\" (UID: \"b9557a90-dc91-4177-a83f-79a0b7526722\") "
Jul 28 10:25:17.407321 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:25:17.407195    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9557a90-dc91-4177-a83f-79a0b7526722-tmp-dir\") pod \"b9557a90-dc91-4177-a83f-79a0b7526722\" (UID: \"b9557a90-dc91-4177-a83f-79a0b7526722\") "
Jul 28 10:25:17.407321 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:25:17.407233    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9557a90-dc91-4177-a83f-79a0b7526722-dshm\") pod \"b9557a90-dc91-4177-a83f-79a0b7526722\" (UID: \"b9557a90-dc91-4177-a83f-79a0b7526722\") "
Jul 28 10:25:17.407321 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:25:17.407294    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b9557a90-dc91-4177-a83f-79a0b7526722-home\") pod \"b9557a90-dc91-4177-a83f-79a0b7526722\" (UID: \"b9557a90-dc91-4177-a83f-79a0b7526722\") "
Jul 28 10:25:17.407628 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:25:17.407432    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9557a90-dc91-4177-a83f-79a0b7526722-model-cache" (OuterVolumeSpecName: "model-cache") pod "b9557a90-dc91-4177-a83f-79a0b7526722" (UID: "b9557a90-dc91-4177-a83f-79a0b7526722"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:25:17.407628 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:25:17.407607    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9557a90-dc91-4177-a83f-79a0b7526722-model-cache\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 10:25:17.408050 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:25:17.408025    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9557a90-dc91-4177-a83f-79a0b7526722-home" (OuterVolumeSpecName: "home") pod "b9557a90-dc91-4177-a83f-79a0b7526722" (UID: "b9557a90-dc91-4177-a83f-79a0b7526722"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:25:17.409458 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:25:17.409428    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9557a90-dc91-4177-a83f-79a0b7526722-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b9557a90-dc91-4177-a83f-79a0b7526722" (UID: "b9557a90-dc91-4177-a83f-79a0b7526722"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:25:17.409657 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:25:17.409640    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9557a90-dc91-4177-a83f-79a0b7526722-dshm" (OuterVolumeSpecName: "dshm") pod "b9557a90-dc91-4177-a83f-79a0b7526722" (UID: "b9557a90-dc91-4177-a83f-79a0b7526722"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:25:17.409765 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:25:17.409748    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9557a90-dc91-4177-a83f-79a0b7526722-kube-api-access-xzs6w" (OuterVolumeSpecName: "kube-api-access-xzs6w") pod "b9557a90-dc91-4177-a83f-79a0b7526722" (UID: "b9557a90-dc91-4177-a83f-79a0b7526722"). InnerVolumeSpecName "kube-api-access-xzs6w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:25:17.418880 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:25:17.418853    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9557a90-dc91-4177-a83f-79a0b7526722-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b9557a90-dc91-4177-a83f-79a0b7526722" (UID: "b9557a90-dc91-4177-a83f-79a0b7526722"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:25:17.470386 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:25:17.470332    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9557a90-dc91-4177-a83f-79a0b7526722-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b9557a90-dc91-4177-a83f-79a0b7526722" (UID: "b9557a90-dc91-4177-a83f-79a0b7526722"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:25:17.508131 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:25:17.508094    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b9557a90-dc91-4177-a83f-79a0b7526722-home\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 10:25:17.508131 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:25:17.508130    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xzs6w\" (UniqueName: \"kubernetes.io/projected/b9557a90-dc91-4177-a83f-79a0b7526722-kube-api-access-xzs6w\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 10:25:17.508513 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:25:17.508140    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b9557a90-dc91-4177-a83f-79a0b7526722-kserve-provision-location\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 10:25:17.508513 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:25:17.508150    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9557a90-dc91-4177-a83f-79a0b7526722-tls-certs\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 10:25:17.508513 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:25:17.508159    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9557a90-dc91-4177-a83f-79a0b7526722-tmp-dir\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 10:25:17.508513 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:25:17.508166    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9557a90-dc91-4177-a83f-79a0b7526722-dshm\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 10:25:17.797759 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:25:17.797724    2579 generic.go:356] "Generic (PLEG): container finished" podID="b9557a90-dc91-4177-a83f-79a0b7526722" containerID="2324aafa4c6be305b1ca666363c8ae73fdbe41cc0d81e11c3dc59308759befd0" exitCode=0
Jul 28 10:25:17.797958 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:25:17.797781    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-68fdf86f947d52k" event={"ID":"b9557a90-dc91-4177-a83f-79a0b7526722","Type":"ContainerDied","Data":"2324aafa4c6be305b1ca666363c8ae73fdbe41cc0d81e11c3dc59308759befd0"}
Jul 28 10:25:17.797958 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:25:17.797798    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-68fdf86f947d52k"
Jul 28 10:25:17.797958 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:25:17.797867    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-68fdf86f947d52k" event={"ID":"b9557a90-dc91-4177-a83f-79a0b7526722","Type":"ContainerDied","Data":"6d0c8f480a3472251e3edc8687ec21ca8ba7f31216b3990ec351f60322b81986"}
Jul 28 10:25:17.797958 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:25:17.797900    2579 scope.go:117] "RemoveContainer" containerID="2324aafa4c6be305b1ca666363c8ae73fdbe41cc0d81e11c3dc59308759befd0"
Jul 28 10:25:17.807457 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:25:17.807440    2579 scope.go:117] "RemoveContainer" containerID="ca449dba1e7688014aa15a6568cdb54b8a5b16f4f02f8f5d5dd95ec15b691328"
Jul 28 10:25:17.815684 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:25:17.815663    2579 scope.go:117] "RemoveContainer" containerID="2324aafa4c6be305b1ca666363c8ae73fdbe41cc0d81e11c3dc59308759befd0"
Jul 28 10:25:17.815932 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:25:17.815917    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2324aafa4c6be305b1ca666363c8ae73fdbe41cc0d81e11c3dc59308759befd0\": container with ID starting with 2324aafa4c6be305b1ca666363c8ae73fdbe41cc0d81e11c3dc59308759befd0 not found: ID does not exist" containerID="2324aafa4c6be305b1ca666363c8ae73fdbe41cc0d81e11c3dc59308759befd0"
Jul 28 10:25:17.815981 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:25:17.815940    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2324aafa4c6be305b1ca666363c8ae73fdbe41cc0d81e11c3dc59308759befd0"} err="failed to get container status \"2324aafa4c6be305b1ca666363c8ae73fdbe41cc0d81e11c3dc59308759befd0\": rpc error: code = NotFound desc = could not find container \"2324aafa4c6be305b1ca666363c8ae73fdbe41cc0d81e11c3dc59308759befd0\": container with ID starting with 2324aafa4c6be305b1ca666363c8ae73fdbe41cc0d81e11c3dc59308759befd0 not found: ID does not exist"
Jul 28 10:25:17.815981 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:25:17.815956    2579 scope.go:117] "RemoveContainer" containerID="ca449dba1e7688014aa15a6568cdb54b8a5b16f4f02f8f5d5dd95ec15b691328"
Jul 28 10:25:17.816164 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:25:17.816145    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca449dba1e7688014aa15a6568cdb54b8a5b16f4f02f8f5d5dd95ec15b691328\": container with ID starting with ca449dba1e7688014aa15a6568cdb54b8a5b16f4f02f8f5d5dd95ec15b691328 not found: ID does not exist" containerID="ca449dba1e7688014aa15a6568cdb54b8a5b16f4f02f8f5d5dd95ec15b691328"
Jul 28 10:25:17.816209 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:25:17.816171    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca449dba1e7688014aa15a6568cdb54b8a5b16f4f02f8f5d5dd95ec15b691328"} err="failed to get container status \"ca449dba1e7688014aa15a6568cdb54b8a5b16f4f02f8f5d5dd95ec15b691328\": rpc error: code = NotFound desc = could not find container \"ca449dba1e7688014aa15a6568cdb54b8a5b16f4f02f8f5d5dd95ec15b691328\": container with ID starting with ca449dba1e7688014aa15a6568cdb54b8a5b16f4f02f8f5d5dd95ec15b691328 not found: ID does not exist"
Jul 28 10:25:17.841434 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:25:17.841397    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-68fdf86f947d52k"]
Jul 28 10:25:17.855304 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:25:17.855274    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-68fdf86f947d52k"]
Jul 28 10:25:18.275483 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:25:18.275399    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b9557a90-dc91-4177-a83f-79a0b7526722" path="/var/lib/kubelet/pods/b9557a90-dc91-4177-a83f-79a0b7526722/volumes"
Jul 28 10:25:18.340977 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:25:18.340940    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:25:18.421779 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:25:18.421742    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:25:19.045966 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:25:19.045935    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:25:19.141556 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:25:19.141504    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:25:20.541354 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:25:20.541324    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:25:25.178731 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:25:25.178698    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:25:27.598946 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:25:27.598910    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:25:27.660317 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:25:27.660284    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:25:31.029364 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:25:31.029324    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:25:31.096036 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:25:31.095996    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:25:33.434852 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:25:33.434817    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:25:39.617324 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:25:39.617228    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:25:46.792357 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:25:46.792328    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-5nmb8"
Jul 28 10:25:56.400765 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:25:56.400730    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:26:00.523300 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:26:00.523256    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:26:00.758803 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:26:00.758761    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:26:00.873003 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:26:00.872961    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:26:00.900574 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:26:00.900523    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Jul 28 10:26:00.900938 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:26:00.900920    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9557a90-dc91-4177-a83f-79a0b7526722" containerName="storage-initializer"
Jul 28 10:26:00.901033 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:26:00.900940    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9557a90-dc91-4177-a83f-79a0b7526722" containerName="storage-initializer"
Jul 28 10:26:00.901033 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:26:00.900969    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9557a90-dc91-4177-a83f-79a0b7526722" containerName="main"
Jul 28 10:26:00.901033 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:26:00.900978    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9557a90-dc91-4177-a83f-79a0b7526722" containerName="main"
Jul 28 10:26:00.901201 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:26:00.901083    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9557a90-dc91-4177-a83f-79a0b7526722" containerName="main"
Jul 28 10:26:00.903981 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:26:00.903958    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 28 10:26:00.906318 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:26:00.906298    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"kube-root-ca.crt\""
Jul 28 10:26:00.906617 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:26:00.906597    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"seaweedfs-s3-creds\""
Jul 28 10:26:00.906998 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:26:00.906982    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"default-dockercfg-l5vsp\""
Jul 28 10:26:00.907081 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:26:00.907012    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"openshift-service-ca.crt\""
Jul 28 10:26:00.907081 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:26:00.907030    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisvc-model-fb-opt-125m-route-dc21cb-57cf7736-dockercfg-tb698\""
Jul 28 10:26:00.907183 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:26:00.907082    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs\""
Jul 28 10:26:00.916562 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:26:00.916527    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Jul 28 10:26:00.979139 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:26:00.979107    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche5mxcv"]
Jul 28 10:26:00.981526 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:26:00.981502    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche5mxcv"
Jul 28 10:26:00.984073 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:26:00.984039    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisvc-model-fb-opt-125m-route-dc21cb14-epp-sa-dockercfg-4pv5f\""
Jul 28 10:26:00.997061 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:26:00.997034    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche5mxcv"]
Jul 28 10:26:01.016171 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:26:01.016139    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:26:01.035309 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:26:01.035275    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8p7h\" (UniqueName: \"kubernetes.io/projected/509d0231-9d51-423e-b642-8fafc6390ab1-kube-api-access-h8p7h\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"509d0231-9d51-423e-b642-8fafc6390ab1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 28 10:26:01.035522 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:26:01.035338    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/509d0231-9d51-423e-b642-8fafc6390ab1-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"509d0231-9d51-423e-b642-8fafc6390ab1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 28 10:26:01.035522 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:26:01.035364    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/509d0231-9d51-423e-b642-8fafc6390ab1-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"509d0231-9d51-423e-b642-8fafc6390ab1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 28 10:26:01.035522 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:26:01.035406    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/509d0231-9d51-423e-b642-8fafc6390ab1-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"509d0231-9d51-423e-b642-8fafc6390ab1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 28 10:26:01.035522 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:26:01.035450    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/509d0231-9d51-423e-b642-8fafc6390ab1-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"509d0231-9d51-423e-b642-8fafc6390ab1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 28 10:26:01.035522 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:26:01.035482    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zz6kx\" (UniqueName: \"kubernetes.io/projected/9496df9e-69de-4e07-b399-917edf4ee0d1-kube-api-access-zz6kx\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche5mxcv\" (UID: \"9496df9e-69de-4e07-b399-917edf4ee0d1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche5mxcv"
Jul 28 10:26:01.035522 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:26:01.035517    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9496df9e-69de-4e07-b399-917edf4ee0d1-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche5mxcv\" (UID: \"9496df9e-69de-4e07-b399-917edf4ee0d1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche5mxcv"
Jul 28 10:26:01.035810 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:26:01.035628    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/509d0231-9d51-423e-b642-8fafc6390ab1-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"509d0231-9d51-423e-b642-8fafc6390ab1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 28 10:26:01.035810 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:26:01.035667    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/509d0231-9d51-423e-b642-8fafc6390ab1-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"509d0231-9d51-423e-b642-8fafc6390ab1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 28 10:26:01.136849 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:26:01.136744    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/509d0231-9d51-423e-b642-8fafc6390ab1-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"509d0231-9d51-423e-b642-8fafc6390ab1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 28 10:26:01.136849 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:26:01.136784    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/509d0231-9d51-423e-b642-8fafc6390ab1-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"509d0231-9d51-423e-b642-8fafc6390ab1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 28 10:26:01.136849 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:26:01.136804    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/509d0231-9d51-423e-b642-8fafc6390ab1-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"509d0231-9d51-423e-b642-8fafc6390ab1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 28 10:26:01.136849 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:26:01.136830    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/509d0231-9d51-423e-b642-8fafc6390ab1-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"509d0231-9d51-423e-b642-8fafc6390ab1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 28 10:26:01.136849 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:26:01.136853    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zz6kx\" (UniqueName: \"kubernetes.io/projected/9496df9e-69de-4e07-b399-917edf4ee0d1-kube-api-access-zz6kx\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche5mxcv\" (UID: \"9496df9e-69de-4e07-b399-917edf4ee0d1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche5mxcv"
Jul 28 10:26:01.137263 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:26:01.136886    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9496df9e-69de-4e07-b399-917edf4ee0d1-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche5mxcv\" (UID: \"9496df9e-69de-4e07-b399-917edf4ee0d1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche5mxcv"
Jul 28 10:26:01.137263 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:26:01.136931    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/509d0231-9d51-423e-b642-8fafc6390ab1-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"509d0231-9d51-423e-b642-8fafc6390ab1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 28 10:26:01.137263 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:26:01.136961    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/509d0231-9d51-423e-b642-8fafc6390ab1-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"509d0231-9d51-423e-b642-8fafc6390ab1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 28 10:26:01.137263 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:26:01.137001    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8p7h\" (UniqueName: \"kubernetes.io/projected/509d0231-9d51-423e-b642-8fafc6390ab1-kube-api-access-h8p7h\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"509d0231-9d51-423e-b642-8fafc6390ab1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 28 10:26:01.137263 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:26:01.137164    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/509d0231-9d51-423e-b642-8fafc6390ab1-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"509d0231-9d51-423e-b642-8fafc6390ab1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 28 10:26:01.137437 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:26:01.137285    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/509d0231-9d51-423e-b642-8fafc6390ab1-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"509d0231-9d51-423e-b642-8fafc6390ab1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 28 10:26:01.137437 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:26:01.137334    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/509d0231-9d51-423e-b642-8fafc6390ab1-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"509d0231-9d51-423e-b642-8fafc6390ab1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 28 10:26:01.137437 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:26:01.137356    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/509d0231-9d51-423e-b642-8fafc6390ab1-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"509d0231-9d51-423e-b642-8fafc6390ab1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 28 10:26:01.139402 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:26:01.139369    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/509d0231-9d51-423e-b642-8fafc6390ab1-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"509d0231-9d51-423e-b642-8fafc6390ab1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 28 10:26:01.139550 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:26:01.139434    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/509d0231-9d51-423e-b642-8fafc6390ab1-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"509d0231-9d51-423e-b642-8fafc6390ab1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 28 10:26:01.139864 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:26:01.139843    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9496df9e-69de-4e07-b399-917edf4ee0d1-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche5mxcv\" (UID: \"9496df9e-69de-4e07-b399-917edf4ee0d1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche5mxcv"
Jul 28 10:26:01.145755 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:26:01.145729    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8p7h\" (UniqueName: \"kubernetes.io/projected/509d0231-9d51-423e-b642-8fafc6390ab1-kube-api-access-h8p7h\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"509d0231-9d51-423e-b642-8fafc6390ab1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 28 10:26:01.146120 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:26:01.146089    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zz6kx\" (UniqueName: \"kubernetes.io/projected/9496df9e-69de-4e07-b399-917edf4ee0d1-kube-api-access-zz6kx\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche5mxcv\" (UID: \"9496df9e-69de-4e07-b399-917edf4ee0d1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche5mxcv"
Jul 28 10:26:01.213983 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:26:01.213941    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 28 10:26:01.293677 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:26:01.293611    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche5mxcv"
Jul 28 10:26:01.359987 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:26:01.359956    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Jul 28 10:26:01.362121 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:26:01.362069    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod509d0231_9d51_423e_b642_8fafc6390ab1.slice/crio-e1dfefa5943c48caeeef140d74005242c96403cb829ac4bb5976e79ce68218f0 WatchSource:0}: Error finding container e1dfefa5943c48caeeef140d74005242c96403cb829ac4bb5976e79ce68218f0: Status 404 returned error can't find the container with id e1dfefa5943c48caeeef140d74005242c96403cb829ac4bb5976e79ce68218f0
Jul 28 10:26:01.446048 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:26:01.446000    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche5mxcv"]
Jul 28 10:26:01.448513 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:26:01.448476    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9496df9e_69de_4e07_b399_917edf4ee0d1.slice/crio-2a21083e89e30ee84273a8f7b184b3b3a08096d9c0d033be7981dd34a8199178 WatchSource:0}: Error finding container 2a21083e89e30ee84273a8f7b184b3b3a08096d9c0d033be7981dd34a8199178: Status 404 returned error can't find the container with id 2a21083e89e30ee84273a8f7b184b3b3a08096d9c0d033be7981dd34a8199178
Jul 28 10:26:02.006661 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:26:02.006624    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche5mxcv" event={"ID":"9496df9e-69de-4e07-b399-917edf4ee0d1","Type":"ContainerStarted","Data":"7a99d9e9cb99c6992a67d9f2f3c3ae266fe79b05ab131eed3bc6a5bca8560a0e"}
Jul 28 10:26:02.007133 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:26:02.006684    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche5mxcv" event={"ID":"9496df9e-69de-4e07-b399-917edf4ee0d1","Type":"ContainerStarted","Data":"2a21083e89e30ee84273a8f7b184b3b3a08096d9c0d033be7981dd34a8199178"}
Jul 28 10:26:02.007133 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:26:02.006807    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche5mxcv"
Jul 28 10:26:02.008188 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:26:02.008165    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"509d0231-9d51-423e-b642-8fafc6390ab1","Type":"ContainerStarted","Data":"c651e0579b8e873fd2ed0b7b4fa459c1a188a4afbb95e784fc027bad41faca63"}
Jul 28 10:26:02.008291 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:26:02.008208    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"509d0231-9d51-423e-b642-8fafc6390ab1","Type":"ContainerStarted","Data":"e1dfefa5943c48caeeef140d74005242c96403cb829ac4bb5976e79ce68218f0"}
Jul 28 10:26:02.026220 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:26:02.026157    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche5mxcv" podStartSLOduration=2.026134495 podStartE2EDuration="2.026134495s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:26:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:26:02.024072049 +0000 UTC m=+1096.287468380" watchObservedRunningTime="2026-07-28 10:26:02.026134495 +0000 UTC m=+1096.289530825"
Jul 28 10:26:06.038031 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:26:06.037995    2579 generic.go:356] "Generic (PLEG): container finished" podID="509d0231-9d51-423e-b642-8fafc6390ab1" containerID="c651e0579b8e873fd2ed0b7b4fa459c1a188a4afbb95e784fc027bad41faca63" exitCode=0
Jul 28 10:26:06.038467 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:26:06.038061    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"509d0231-9d51-423e-b642-8fafc6390ab1","Type":"ContainerDied","Data":"c651e0579b8e873fd2ed0b7b4fa459c1a188a4afbb95e784fc027bad41faca63"}
Jul 28 10:26:07.045989 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:26:07.045947    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"509d0231-9d51-423e-b642-8fafc6390ab1","Type":"ContainerStarted","Data":"8f0165a95a78f39c394815645994d3f334809af17fb067f1af0ce5a523a93049"}
Jul 28 10:26:07.067910 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:26:07.067853    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podStartSLOduration=2.634100792 podStartE2EDuration="7.067835511s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.433734719s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:26:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:26:07.065871418 +0000 UTC m=+1101.329267748" watchObservedRunningTime="2026-07-28 10:26:07.067835511 +0000 UTC m=+1101.331231857"
Jul 28 10:26:07.214653 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:26:07.214605    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 28 10:26:11.214859 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:26:11.214817    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 28 10:26:11.216691 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:26:11.216653    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="509d0231-9d51-423e-b642-8fafc6390ab1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/health\": dial tcp 10.134.0.39:8000: connect: connection refused"
Jul 28 10:26:21.215077 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:26:21.215029    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="509d0231-9d51-423e-b642-8fafc6390ab1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/health\": dial tcp 10.134.0.39:8000: connect: connection refused"
Jul 28 10:26:21.849089 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:26:21.849053    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:26:31.215073 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:26:31.215022    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="509d0231-9d51-423e-b642-8fafc6390ab1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/health\": dial tcp 10.134.0.39:8000: connect: connection refused"
Jul 28 10:26:33.017205 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:26:33.017169    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche5mxcv"
Jul 28 10:26:41.214941 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:26:41.214896    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="509d0231-9d51-423e-b642-8fafc6390ab1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/health\": dial tcp 10.134.0.39:8000: connect: connection refused"
Jul 28 10:26:51.215180 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:26:51.215135    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="509d0231-9d51-423e-b642-8fafc6390ab1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/health\": dial tcp 10.134.0.39:8000: connect: connection refused"
Jul 28 10:27:01.214701 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:27:01.214658    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="509d0231-9d51-423e-b642-8fafc6390ab1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/health\": dial tcp 10.134.0.39:8000: connect: connection refused"
Jul 28 10:27:11.214580 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:27:11.214518    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="509d0231-9d51-423e-b642-8fafc6390ab1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/health\": dial tcp 10.134.0.39:8000: connect: connection refused"
Jul 28 10:27:21.215311 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:27:21.215268    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="509d0231-9d51-423e-b642-8fafc6390ab1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/health\": dial tcp 10.134.0.39:8000: connect: connection refused"
Jul 28 10:27:31.214675 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:27:31.214633    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="509d0231-9d51-423e-b642-8fafc6390ab1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/health\": dial tcp 10.134.0.39:8000: connect: connection refused"
Jul 28 10:27:41.224490 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:27:41.224460    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 28 10:27:41.232096 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:27:41.232069    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 28 10:28:40.483616 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:28:40.483442    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-5nmb8"]
Jul 28 10:28:40.524843 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:28:40.524805    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:28:40.584095 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:28:40.584028    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:28:46.112695 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:28:46.112588    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:28:46.136359 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:28:46.136324    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Jul 28 10:28:46.142452 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:28:46.142423    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche5mxcv"]
Jul 28 10:28:46.191191 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:28:46.191158    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:28:46.791982 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:28:46.791943    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-5nmb8" podUID="9bbe32a0-9b8e-4546-b3c1-d2e8ea4ec8f7" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 10:28:53.016502 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:28:53.016466    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche5mxcv" podUID="9496df9e-69de-4e07-b399-917edf4ee0d1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 10:28:55.510485 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:28:55.510442    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-5nmb8" podUID="9bbe32a0-9b8e-4546-b3c1-d2e8ea4ec8f7" containerName="main" containerID="cri-o://fdabfeb18e06aa0df1f750ab76c873e8704e4512b284cbb84b0443a06acb9b2e" gracePeriod=45
Jul 28 10:28:55.764493 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:28:55.764435    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-5nmb8"
Jul 28 10:28:55.823647 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:28:55.823616    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q9p7j\" (UniqueName: \"kubernetes.io/projected/9bbe32a0-9b8e-4546-b3c1-d2e8ea4ec8f7-kube-api-access-q9p7j\") pod \"9bbe32a0-9b8e-4546-b3c1-d2e8ea4ec8f7\" (UID: \"9bbe32a0-9b8e-4546-b3c1-d2e8ea4ec8f7\") "
Jul 28 10:28:55.823647 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:28:55.823663    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9bbe32a0-9b8e-4546-b3c1-d2e8ea4ec8f7-tls-certs\") pod \"9bbe32a0-9b8e-4546-b3c1-d2e8ea4ec8f7\" (UID: \"9bbe32a0-9b8e-4546-b3c1-d2e8ea4ec8f7\") "
Jul 28 10:28:55.825940 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:28:55.825911    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9bbe32a0-9b8e-4546-b3c1-d2e8ea4ec8f7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9bbe32a0-9b8e-4546-b3c1-d2e8ea4ec8f7" (UID: "9bbe32a0-9b8e-4546-b3c1-d2e8ea4ec8f7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:28:55.826049 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:28:55.825943    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bbe32a0-9b8e-4546-b3c1-d2e8ea4ec8f7-kube-api-access-q9p7j" (OuterVolumeSpecName: "kube-api-access-q9p7j") pod "9bbe32a0-9b8e-4546-b3c1-d2e8ea4ec8f7" (UID: "9bbe32a0-9b8e-4546-b3c1-d2e8ea4ec8f7"). InnerVolumeSpecName "kube-api-access-q9p7j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:28:55.910554 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:28:55.910508    2579 generic.go:356] "Generic (PLEG): container finished" podID="9bbe32a0-9b8e-4546-b3c1-d2e8ea4ec8f7" containerID="fdabfeb18e06aa0df1f750ab76c873e8704e4512b284cbb84b0443a06acb9b2e" exitCode=0
Jul 28 10:28:55.910718 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:28:55.910562    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-5nmb8" event={"ID":"9bbe32a0-9b8e-4546-b3c1-d2e8ea4ec8f7","Type":"ContainerDied","Data":"fdabfeb18e06aa0df1f750ab76c873e8704e4512b284cbb84b0443a06acb9b2e"}
Jul 28 10:28:55.910718 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:28:55.910568    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-5nmb8"
Jul 28 10:28:55.910718 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:28:55.910607    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-5nmb8" event={"ID":"9bbe32a0-9b8e-4546-b3c1-d2e8ea4ec8f7","Type":"ContainerDied","Data":"1a6446c79d78f0d372d5ff5aa7d07e2d4c18e1a0b9e24c4b1fbeb0d614761d29"}
Jul 28 10:28:55.910718 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:28:55.910630    2579 scope.go:117] "RemoveContainer" containerID="fdabfeb18e06aa0df1f750ab76c873e8704e4512b284cbb84b0443a06acb9b2e"
Jul 28 10:28:55.919631 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:28:55.919612    2579 scope.go:117] "RemoveContainer" containerID="fdabfeb18e06aa0df1f750ab76c873e8704e4512b284cbb84b0443a06acb9b2e"
Jul 28 10:28:55.919883 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:28:55.919864    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdabfeb18e06aa0df1f750ab76c873e8704e4512b284cbb84b0443a06acb9b2e\": container with ID starting with fdabfeb18e06aa0df1f750ab76c873e8704e4512b284cbb84b0443a06acb9b2e not found: ID does not exist" containerID="fdabfeb18e06aa0df1f750ab76c873e8704e4512b284cbb84b0443a06acb9b2e"
Jul 28 10:28:55.919929 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:28:55.919892    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdabfeb18e06aa0df1f750ab76c873e8704e4512b284cbb84b0443a06acb9b2e"} err="failed to get container status \"fdabfeb18e06aa0df1f750ab76c873e8704e4512b284cbb84b0443a06acb9b2e\": rpc error: code = NotFound desc = could not find container \"fdabfeb18e06aa0df1f750ab76c873e8704e4512b284cbb84b0443a06acb9b2e\": container with ID starting with fdabfeb18e06aa0df1f750ab76c873e8704e4512b284cbb84b0443a06acb9b2e not found: ID does not exist"
Jul 28 10:28:55.924829 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:28:55.924814    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9p7j\" (UniqueName: \"kubernetes.io/projected/9bbe32a0-9b8e-4546-b3c1-d2e8ea4ec8f7-kube-api-access-q9p7j\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 10:28:55.924886 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:28:55.924832    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9bbe32a0-9b8e-4546-b3c1-d2e8ea4ec8f7-tls-certs\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 10:28:55.930973 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:28:55.930955    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-5nmb8"]
Jul 28 10:28:55.934726 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:28:55.934707    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-5dbb5856d8-5nmb8"]
Jul 28 10:28:56.276256 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:28:56.276223    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9bbe32a0-9b8e-4546-b3c1-d2e8ea4ec8f7" path="/var/lib/kubelet/pods/9bbe32a0-9b8e-4546-b3c1-d2e8ea4ec8f7/volumes"
Jul 28 10:29:00.229841 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:29:00.229807    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:29:00.329682 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:29:00.329639    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:29:01.159470 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:29:01.159426    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="509d0231-9d51-423e-b642-8fafc6390ab1" containerName="main" containerID="cri-o://8f0165a95a78f39c394815645994d3f334809af17fb067f1af0ce5a523a93049" gracePeriod=45
Jul 28 10:29:01.167157 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:29:01.167122    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche5mxcv" podUID="9496df9e-69de-4e07-b399-917edf4ee0d1" containerName="main" containerID="cri-o://7a99d9e9cb99c6992a67d9f2f3c3ae266fe79b05ab131eed3bc6a5bca8560a0e" gracePeriod=45
Jul 28 10:29:01.416508 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:29:01.416429    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche5mxcv"
Jul 28 10:29:01.470900 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:29:01.470860    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zz6kx\" (UniqueName: \"kubernetes.io/projected/9496df9e-69de-4e07-b399-917edf4ee0d1-kube-api-access-zz6kx\") pod \"9496df9e-69de-4e07-b399-917edf4ee0d1\" (UID: \"9496df9e-69de-4e07-b399-917edf4ee0d1\") "
Jul 28 10:29:01.471090 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:29:01.470924    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9496df9e-69de-4e07-b399-917edf4ee0d1-tls-certs\") pod \"9496df9e-69de-4e07-b399-917edf4ee0d1\" (UID: \"9496df9e-69de-4e07-b399-917edf4ee0d1\") "
Jul 28 10:29:01.473105 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:29:01.473077    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9496df9e-69de-4e07-b399-917edf4ee0d1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9496df9e-69de-4e07-b399-917edf4ee0d1" (UID: "9496df9e-69de-4e07-b399-917edf4ee0d1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:29:01.473289 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:29:01.473148    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9496df9e-69de-4e07-b399-917edf4ee0d1-kube-api-access-zz6kx" (OuterVolumeSpecName: "kube-api-access-zz6kx") pod "9496df9e-69de-4e07-b399-917edf4ee0d1" (UID: "9496df9e-69de-4e07-b399-917edf4ee0d1"). InnerVolumeSpecName "kube-api-access-zz6kx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:29:01.572441 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:29:01.572403    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zz6kx\" (UniqueName: \"kubernetes.io/projected/9496df9e-69de-4e07-b399-917edf4ee0d1-kube-api-access-zz6kx\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 10:29:01.572441 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:29:01.572436    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9496df9e-69de-4e07-b399-917edf4ee0d1-tls-certs\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 10:29:01.916740 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:29:01.916713    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 28 10:29:01.952691 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:29:01.952663    2579 generic.go:356] "Generic (PLEG): container finished" podID="9496df9e-69de-4e07-b399-917edf4ee0d1" containerID="7a99d9e9cb99c6992a67d9f2f3c3ae266fe79b05ab131eed3bc6a5bca8560a0e" exitCode=0
Jul 28 10:29:01.952897 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:29:01.952710    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche5mxcv"
Jul 28 10:29:01.952897 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:29:01.952749    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche5mxcv" event={"ID":"9496df9e-69de-4e07-b399-917edf4ee0d1","Type":"ContainerDied","Data":"7a99d9e9cb99c6992a67d9f2f3c3ae266fe79b05ab131eed3bc6a5bca8560a0e"}
Jul 28 10:29:01.952897 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:29:01.952818    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche5mxcv" event={"ID":"9496df9e-69de-4e07-b399-917edf4ee0d1","Type":"ContainerDied","Data":"2a21083e89e30ee84273a8f7b184b3b3a08096d9c0d033be7981dd34a8199178"}
Jul 28 10:29:01.952897 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:29:01.952851    2579 scope.go:117] "RemoveContainer" containerID="7a99d9e9cb99c6992a67d9f2f3c3ae266fe79b05ab131eed3bc6a5bca8560a0e"
Jul 28 10:29:01.954576 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:29:01.954530    2579 generic.go:356] "Generic (PLEG): container finished" podID="509d0231-9d51-423e-b642-8fafc6390ab1" containerID="8f0165a95a78f39c394815645994d3f334809af17fb067f1af0ce5a523a93049" exitCode=0
Jul 28 10:29:01.954712 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:29:01.954598    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"509d0231-9d51-423e-b642-8fafc6390ab1","Type":"ContainerDied","Data":"8f0165a95a78f39c394815645994d3f334809af17fb067f1af0ce5a523a93049"}
Jul 28 10:29:01.954712 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:29:01.954609    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 28 10:29:01.954712 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:29:01.954642    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"509d0231-9d51-423e-b642-8fafc6390ab1","Type":"ContainerDied","Data":"e1dfefa5943c48caeeef140d74005242c96403cb829ac4bb5976e79ce68218f0"}
Jul 28 10:29:01.962799 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:29:01.962781    2579 scope.go:117] "RemoveContainer" containerID="7a99d9e9cb99c6992a67d9f2f3c3ae266fe79b05ab131eed3bc6a5bca8560a0e"
Jul 28 10:29:01.963081 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:29:01.963056    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a99d9e9cb99c6992a67d9f2f3c3ae266fe79b05ab131eed3bc6a5bca8560a0e\": container with ID starting with 7a99d9e9cb99c6992a67d9f2f3c3ae266fe79b05ab131eed3bc6a5bca8560a0e not found: ID does not exist" containerID="7a99d9e9cb99c6992a67d9f2f3c3ae266fe79b05ab131eed3bc6a5bca8560a0e"
Jul 28 10:29:01.963135 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:29:01.963097    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a99d9e9cb99c6992a67d9f2f3c3ae266fe79b05ab131eed3bc6a5bca8560a0e"} err="failed to get container status \"7a99d9e9cb99c6992a67d9f2f3c3ae266fe79b05ab131eed3bc6a5bca8560a0e\": rpc error: code = NotFound desc = could not find container \"7a99d9e9cb99c6992a67d9f2f3c3ae266fe79b05ab131eed3bc6a5bca8560a0e\": container with ID starting with 7a99d9e9cb99c6992a67d9f2f3c3ae266fe79b05ab131eed3bc6a5bca8560a0e not found: ID does not exist"
Jul 28 10:29:01.963135 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:29:01.963120    2579 scope.go:117] "RemoveContainer" containerID="8f0165a95a78f39c394815645994d3f334809af17fb067f1af0ce5a523a93049"
Jul 28 10:29:01.972974 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:29:01.972834    2579 scope.go:117] "RemoveContainer" containerID="c651e0579b8e873fd2ed0b7b4fa459c1a188a4afbb95e784fc027bad41faca63"
Jul 28 10:29:01.974971 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:29:01.974950    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche5mxcv"]
Jul 28 10:29:01.975083 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:29:01.975065    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/509d0231-9d51-423e-b642-8fafc6390ab1-kserve-provision-location\") pod \"509d0231-9d51-423e-b642-8fafc6390ab1\" (UID: \"509d0231-9d51-423e-b642-8fafc6390ab1\") "
Jul 28 10:29:01.975127 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:29:01.975114    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/509d0231-9d51-423e-b642-8fafc6390ab1-model-cache\") pod \"509d0231-9d51-423e-b642-8fafc6390ab1\" (UID: \"509d0231-9d51-423e-b642-8fafc6390ab1\") "
Jul 28 10:29:01.975163 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:29:01.975141    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/509d0231-9d51-423e-b642-8fafc6390ab1-tls-certs\") pod \"509d0231-9d51-423e-b642-8fafc6390ab1\" (UID: \"509d0231-9d51-423e-b642-8fafc6390ab1\") "
Jul 28 10:29:01.975197 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:29:01.975166    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/509d0231-9d51-423e-b642-8fafc6390ab1-tmp-dir\") pod \"509d0231-9d51-423e-b642-8fafc6390ab1\" (UID: \"509d0231-9d51-423e-b642-8fafc6390ab1\") "
Jul 28 10:29:01.975197 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:29:01.975191    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h8p7h\" (UniqueName: \"kubernetes.io/projected/509d0231-9d51-423e-b642-8fafc6390ab1-kube-api-access-h8p7h\") pod \"509d0231-9d51-423e-b642-8fafc6390ab1\" (UID: \"509d0231-9d51-423e-b642-8fafc6390ab1\") "
Jul 28 10:29:01.975281 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:29:01.975234    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/509d0231-9d51-423e-b642-8fafc6390ab1-home\") pod \"509d0231-9d51-423e-b642-8fafc6390ab1\" (UID: \"509d0231-9d51-423e-b642-8fafc6390ab1\") "
Jul 28 10:29:01.975281 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:29:01.975269    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/509d0231-9d51-423e-b642-8fafc6390ab1-dshm\") pod \"509d0231-9d51-423e-b642-8fafc6390ab1\" (UID: \"509d0231-9d51-423e-b642-8fafc6390ab1\") "
Jul 28 10:29:01.976961 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:29:01.975426    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/509d0231-9d51-423e-b642-8fafc6390ab1-model-cache" (OuterVolumeSpecName: "model-cache") pod "509d0231-9d51-423e-b642-8fafc6390ab1" (UID: "509d0231-9d51-423e-b642-8fafc6390ab1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:29:01.976961 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:29:01.976903    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/509d0231-9d51-423e-b642-8fafc6390ab1-home" (OuterVolumeSpecName: "home") pod "509d0231-9d51-423e-b642-8fafc6390ab1" (UID: "509d0231-9d51-423e-b642-8fafc6390ab1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:29:01.977781 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:29:01.977742    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/509d0231-9d51-423e-b642-8fafc6390ab1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "509d0231-9d51-423e-b642-8fafc6390ab1" (UID: "509d0231-9d51-423e-b642-8fafc6390ab1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:29:01.977918 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:29:01.977897    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/509d0231-9d51-423e-b642-8fafc6390ab1-kube-api-access-h8p7h" (OuterVolumeSpecName: "kube-api-access-h8p7h") pod "509d0231-9d51-423e-b642-8fafc6390ab1" (UID: "509d0231-9d51-423e-b642-8fafc6390ab1"). InnerVolumeSpecName "kube-api-access-h8p7h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:29:01.978303 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:29:01.978278    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/509d0231-9d51-423e-b642-8fafc6390ab1-dshm" (OuterVolumeSpecName: "dshm") pod "509d0231-9d51-423e-b642-8fafc6390ab1" (UID: "509d0231-9d51-423e-b642-8fafc6390ab1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:29:01.980101 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:29:01.980079    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche5mxcv"]
Jul 28 10:29:01.983260 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:29:01.983243    2579 scope.go:117] "RemoveContainer" containerID="8f0165a95a78f39c394815645994d3f334809af17fb067f1af0ce5a523a93049"
Jul 28 10:29:01.983623 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:29:01.983603    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f0165a95a78f39c394815645994d3f334809af17fb067f1af0ce5a523a93049\": container with ID starting with 8f0165a95a78f39c394815645994d3f334809af17fb067f1af0ce5a523a93049 not found: ID does not exist" containerID="8f0165a95a78f39c394815645994d3f334809af17fb067f1af0ce5a523a93049"
Jul 28 10:29:01.983685 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:29:01.983632    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f0165a95a78f39c394815645994d3f334809af17fb067f1af0ce5a523a93049"} err="failed to get container status \"8f0165a95a78f39c394815645994d3f334809af17fb067f1af0ce5a523a93049\": rpc error: code = NotFound desc = could not find container \"8f0165a95a78f39c394815645994d3f334809af17fb067f1af0ce5a523a93049\": container with ID starting with 8f0165a95a78f39c394815645994d3f334809af17fb067f1af0ce5a523a93049 not found: ID does not exist"
Jul 28 10:29:01.983685 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:29:01.983652    2579 scope.go:117] "RemoveContainer" containerID="c651e0579b8e873fd2ed0b7b4fa459c1a188a4afbb95e784fc027bad41faca63"
Jul 28 10:29:01.983925 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:29:01.983903    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c651e0579b8e873fd2ed0b7b4fa459c1a188a4afbb95e784fc027bad41faca63\": container with ID starting with c651e0579b8e873fd2ed0b7b4fa459c1a188a4afbb95e784fc027bad41faca63 not found: ID does not exist" containerID="c651e0579b8e873fd2ed0b7b4fa459c1a188a4afbb95e784fc027bad41faca63"
Jul 28 10:29:01.984014 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:29:01.983929    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c651e0579b8e873fd2ed0b7b4fa459c1a188a4afbb95e784fc027bad41faca63"} err="failed to get container status \"c651e0579b8e873fd2ed0b7b4fa459c1a188a4afbb95e784fc027bad41faca63\": rpc error: code = NotFound desc = could not find container \"c651e0579b8e873fd2ed0b7b4fa459c1a188a4afbb95e784fc027bad41faca63\": container with ID starting with c651e0579b8e873fd2ed0b7b4fa459c1a188a4afbb95e784fc027bad41faca63 not found: ID does not exist"
Jul 28 10:29:01.989413 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:29:01.989388    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/509d0231-9d51-423e-b642-8fafc6390ab1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "509d0231-9d51-423e-b642-8fafc6390ab1" (UID: "509d0231-9d51-423e-b642-8fafc6390ab1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:29:02.043407 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:29:02.043358    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/509d0231-9d51-423e-b642-8fafc6390ab1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "509d0231-9d51-423e-b642-8fafc6390ab1" (UID: "509d0231-9d51-423e-b642-8fafc6390ab1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:29:02.075860 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:29:02.075833    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/509d0231-9d51-423e-b642-8fafc6390ab1-model-cache\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 10:29:02.075860 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:29:02.075859    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/509d0231-9d51-423e-b642-8fafc6390ab1-tls-certs\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 10:29:02.076003 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:29:02.075889    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/509d0231-9d51-423e-b642-8fafc6390ab1-tmp-dir\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 10:29:02.076003 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:29:02.075899    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8p7h\" (UniqueName: \"kubernetes.io/projected/509d0231-9d51-423e-b642-8fafc6390ab1-kube-api-access-h8p7h\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 10:29:02.076003 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:29:02.075911    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/509d0231-9d51-423e-b642-8fafc6390ab1-home\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 10:29:02.076003 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:29:02.075921    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/509d0231-9d51-423e-b642-8fafc6390ab1-dshm\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 10:29:02.076003 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:29:02.075929    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/509d0231-9d51-423e-b642-8fafc6390ab1-kserve-provision-location\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 10:29:02.279488 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:29:02.279438    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9496df9e-69de-4e07-b399-917edf4ee0d1" path="/var/lib/kubelet/pods/9496df9e-69de-4e07-b399-917edf4ee0d1/volumes"
Jul 28 10:29:02.282168 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:29:02.282136    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Jul 28 10:29:02.283683 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:29:02.283659    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Jul 28 10:29:04.275876 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:29:04.275837    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="509d0231-9d51-423e-b642-8fafc6390ab1" path="/var/lib/kubelet/pods/509d0231-9d51-423e-b642-8fafc6390ab1/volumes"
Jul 28 10:29:06.432682 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:29:06.432653    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:29:10.079003 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:29:10.078972    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-59fc46f4dd-qfmvf"]
Jul 28 10:29:10.079364 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:29:10.079277    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9bbe32a0-9b8e-4546-b3c1-d2e8ea4ec8f7" containerName="main"
Jul 28 10:29:10.079364 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:29:10.079288    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="9bbe32a0-9b8e-4546-b3c1-d2e8ea4ec8f7" containerName="main"
Jul 28 10:29:10.079364 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:29:10.079299    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="509d0231-9d51-423e-b642-8fafc6390ab1" containerName="main"
Jul 28 10:29:10.079364 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:29:10.079304    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="509d0231-9d51-423e-b642-8fafc6390ab1" containerName="main"
Jul 28 10:29:10.079364 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:29:10.079320    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9496df9e-69de-4e07-b399-917edf4ee0d1" containerName="main"
Jul 28 10:29:10.079364 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:29:10.079325    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="9496df9e-69de-4e07-b399-917edf4ee0d1" containerName="main"
Jul 28 10:29:10.079364 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:29:10.079334    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="509d0231-9d51-423e-b642-8fafc6390ab1" containerName="storage-initializer"
Jul 28 10:29:10.079364 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:29:10.079339    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="509d0231-9d51-423e-b642-8fafc6390ab1" containerName="storage-initializer"
Jul 28 10:29:10.079644 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:29:10.079385    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="9bbe32a0-9b8e-4546-b3c1-d2e8ea4ec8f7" containerName="main"
Jul 28 10:29:10.079644 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:29:10.079395    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="9496df9e-69de-4e07-b399-917edf4ee0d1" containerName="main"
Jul 28 10:29:10.079644 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:29:10.079400    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="509d0231-9d51-423e-b642-8fafc6390ab1" containerName="main"
Jul 28 10:29:10.084005 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:29:10.083986    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-59fc46f4dd-qfmvf"
Jul 28 10:29:10.086046 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:29:10.086024    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"openshift-service-ca.crt\""
Jul 28 10:29:10.086158 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:29:10.086054    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"kube-root-ca.crt\""
Jul 28 10:29:10.086351 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:29:10.086332    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"auth-invalid-token-test-kserve-self-signed-certs\""
Jul 28 10:29:10.086412 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:29:10.086396    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"default-dockercfg-rsdnv\""
Jul 28 10:29:10.086455 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:29:10.086429    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"seaweedfs-s3-creds\""
Jul 28 10:29:10.094308 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:29:10.094287    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-59fc46f4dd-qfmvf"]
Jul 28 10:29:10.127417 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:29:10.127384    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:29:10.244502 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:29:10.244464    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eb61c5ec-0ef4-4d6f-9fc5-2c0351ddf4fd-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-59fc46f4dd-qfmvf\" (UID: \"eb61c5ec-0ef4-4d6f-9fc5-2c0351ddf4fd\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-59fc46f4dd-qfmvf"
Jul 28 10:29:10.244674 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:29:10.244516    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb61c5ec-0ef4-4d6f-9fc5-2c0351ddf4fd-home\") pod \"auth-invalid-token-test-kserve-59fc46f4dd-qfmvf\" (UID: \"eb61c5ec-0ef4-4d6f-9fc5-2c0351ddf4fd\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-59fc46f4dd-qfmvf"
Jul 28 10:29:10.244674 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:29:10.244586    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb61c5ec-0ef4-4d6f-9fc5-2c0351ddf4fd-dshm\") pod \"auth-invalid-token-test-kserve-59fc46f4dd-qfmvf\" (UID: \"eb61c5ec-0ef4-4d6f-9fc5-2c0351ddf4fd\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-59fc46f4dd-qfmvf"
Jul 28 10:29:10.244674 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:29:10.244649    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb61c5ec-0ef4-4d6f-9fc5-2c0351ddf4fd-tls-certs\") pod \"auth-invalid-token-test-kserve-59fc46f4dd-qfmvf\" (UID: \"eb61c5ec-0ef4-4d6f-9fc5-2c0351ddf4fd\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-59fc46f4dd-qfmvf"
Jul 28 10:29:10.244808 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:29:10.244702    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb61c5ec-0ef4-4d6f-9fc5-2c0351ddf4fd-tmp-dir\") pod \"auth-invalid-token-test-kserve-59fc46f4dd-qfmvf\" (UID: \"eb61c5ec-0ef4-4d6f-9fc5-2c0351ddf4fd\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-59fc46f4dd-qfmvf"
Jul 28 10:29:10.244808 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:29:10.244740    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdrdh\" (UniqueName: \"kubernetes.io/projected/eb61c5ec-0ef4-4d6f-9fc5-2c0351ddf4fd-kube-api-access-tdrdh\") pod \"auth-invalid-token-test-kserve-59fc46f4dd-qfmvf\" (UID: \"eb61c5ec-0ef4-4d6f-9fc5-2c0351ddf4fd\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-59fc46f4dd-qfmvf"
Jul 28 10:29:10.244808 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:29:10.244761    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb61c5ec-0ef4-4d6f-9fc5-2c0351ddf4fd-model-cache\") pod \"auth-invalid-token-test-kserve-59fc46f4dd-qfmvf\" (UID: \"eb61c5ec-0ef4-4d6f-9fc5-2c0351ddf4fd\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-59fc46f4dd-qfmvf"
Jul 28 10:29:10.295652 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:29:10.295617    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dcp8rdn"]
Jul 28 10:29:10.299305 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:29:10.299283    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dcp8rdn"
Jul 28 10:29:10.301362 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:29:10.301340    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"auth-invalid-token-test-epp-sa-dockercfg-vqmc5\""
Jul 28 10:29:10.308422 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:29:10.308394    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dcp8rdn"]
Jul 28 10:29:10.330037 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:29:10.329948    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:29:10.346053 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:29:10.346021    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdrdh\" (UniqueName: \"kubernetes.io/projected/eb61c5ec-0ef4-4d6f-9fc5-2c0351ddf4fd-kube-api-access-tdrdh\") pod \"auth-invalid-token-test-kserve-59fc46f4dd-qfmvf\" (UID: \"eb61c5ec-0ef4-4d6f-9fc5-2c0351ddf4fd\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-59fc46f4dd-qfmvf"
Jul 28 10:29:10.346198 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:29:10.346061    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb61c5ec-0ef4-4d6f-9fc5-2c0351ddf4fd-model-cache\") pod \"auth-invalid-token-test-kserve-59fc46f4dd-qfmvf\" (UID: \"eb61c5ec-0ef4-4d6f-9fc5-2c0351ddf4fd\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-59fc46f4dd-qfmvf"
Jul 28 10:29:10.346198 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:29:10.346091    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eb61c5ec-0ef4-4d6f-9fc5-2c0351ddf4fd-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-59fc46f4dd-qfmvf\" (UID: \"eb61c5ec-0ef4-4d6f-9fc5-2c0351ddf4fd\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-59fc46f4dd-qfmvf"
Jul 28 10:29:10.346198 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:29:10.346119    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb61c5ec-0ef4-4d6f-9fc5-2c0351ddf4fd-home\") pod \"auth-invalid-token-test-kserve-59fc46f4dd-qfmvf\" (UID: \"eb61c5ec-0ef4-4d6f-9fc5-2c0351ddf4fd\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-59fc46f4dd-qfmvf"
Jul 28 10:29:10.346198 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:29:10.346140    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb61c5ec-0ef4-4d6f-9fc5-2c0351ddf4fd-dshm\") pod \"auth-invalid-token-test-kserve-59fc46f4dd-qfmvf\" (UID: \"eb61c5ec-0ef4-4d6f-9fc5-2c0351ddf4fd\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-59fc46f4dd-qfmvf"
Jul 28 10:29:10.346198 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:29:10.346186    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb61c5ec-0ef4-4d6f-9fc5-2c0351ddf4fd-tls-certs\") pod \"auth-invalid-token-test-kserve-59fc46f4dd-qfmvf\" (UID: \"eb61c5ec-0ef4-4d6f-9fc5-2c0351ddf4fd\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-59fc46f4dd-qfmvf"
Jul 28 10:29:10.346482 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:29:10.346212    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb61c5ec-0ef4-4d6f-9fc5-2c0351ddf4fd-tmp-dir\") pod \"auth-invalid-token-test-kserve-59fc46f4dd-qfmvf\" (UID: \"eb61c5ec-0ef4-4d6f-9fc5-2c0351ddf4fd\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-59fc46f4dd-qfmvf"
Jul 28 10:29:10.346527 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:29:10.346487    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eb61c5ec-0ef4-4d6f-9fc5-2c0351ddf4fd-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-59fc46f4dd-qfmvf\" (UID: \"eb61c5ec-0ef4-4d6f-9fc5-2c0351ddf4fd\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-59fc46f4dd-qfmvf"
Jul 28 10:29:10.346667 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:29:10.346641    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb61c5ec-0ef4-4d6f-9fc5-2c0351ddf4fd-model-cache\") pod \"auth-invalid-token-test-kserve-59fc46f4dd-qfmvf\" (UID: \"eb61c5ec-0ef4-4d6f-9fc5-2c0351ddf4fd\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-59fc46f4dd-qfmvf"
Jul 28 10:29:10.346743 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:29:10.346669    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb61c5ec-0ef4-4d6f-9fc5-2c0351ddf4fd-tmp-dir\") pod \"auth-invalid-token-test-kserve-59fc46f4dd-qfmvf\" (UID: \"eb61c5ec-0ef4-4d6f-9fc5-2c0351ddf4fd\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-59fc46f4dd-qfmvf"
Jul 28 10:29:10.346743 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:29:10.346685    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb61c5ec-0ef4-4d6f-9fc5-2c0351ddf4fd-home\") pod \"auth-invalid-token-test-kserve-59fc46f4dd-qfmvf\" (UID: \"eb61c5ec-0ef4-4d6f-9fc5-2c0351ddf4fd\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-59fc46f4dd-qfmvf"
Jul 28 10:29:10.348528 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:29:10.348503    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb61c5ec-0ef4-4d6f-9fc5-2c0351ddf4fd-dshm\") pod \"auth-invalid-token-test-kserve-59fc46f4dd-qfmvf\" (UID: \"eb61c5ec-0ef4-4d6f-9fc5-2c0351ddf4fd\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-59fc46f4dd-qfmvf"
Jul 28 10:29:10.348805 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:29:10.348786    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb61c5ec-0ef4-4d6f-9fc5-2c0351ddf4fd-tls-certs\") pod \"auth-invalid-token-test-kserve-59fc46f4dd-qfmvf\" (UID: \"eb61c5ec-0ef4-4d6f-9fc5-2c0351ddf4fd\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-59fc46f4dd-qfmvf"
Jul 28 10:29:10.354583 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:29:10.354558    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdrdh\" (UniqueName: \"kubernetes.io/projected/eb61c5ec-0ef4-4d6f-9fc5-2c0351ddf4fd-kube-api-access-tdrdh\") pod \"auth-invalid-token-test-kserve-59fc46f4dd-qfmvf\" (UID: \"eb61c5ec-0ef4-4d6f-9fc5-2c0351ddf4fd\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-59fc46f4dd-qfmvf"
Jul 28 10:29:10.394649 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:29:10.394610    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-59fc46f4dd-qfmvf"
Jul 28 10:29:10.447273 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:29:10.447240    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgqdt\" (UniqueName: \"kubernetes.io/projected/ec727af7-04e6-4cef-9c20-644fc83cbb03-kube-api-access-bgqdt\") pod \"auth-invalid-token-test-kserve-router-scheduler-6d845bb7dcp8rdn\" (UID: \"ec727af7-04e6-4cef-9c20-644fc83cbb03\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dcp8rdn"
Jul 28 10:29:10.447441 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:29:10.447359    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec727af7-04e6-4cef-9c20-644fc83cbb03-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-6d845bb7dcp8rdn\" (UID: \"ec727af7-04e6-4cef-9c20-644fc83cbb03\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dcp8rdn"
Jul 28 10:29:10.542628 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:29:10.542601    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-59fc46f4dd-qfmvf"]
Jul 28 10:29:10.544652 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:29:10.544620    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb61c5ec_0ef4_4d6f_9fc5_2c0351ddf4fd.slice/crio-9ca65b503e603336271d82048400dea0d7d88825a633e9309930bdc90c57d473 WatchSource:0}: Error finding container 9ca65b503e603336271d82048400dea0d7d88825a633e9309930bdc90c57d473: Status 404 returned error can't find the container with id 9ca65b503e603336271d82048400dea0d7d88825a633e9309930bdc90c57d473
Jul 28 10:29:10.548742 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:29:10.548718    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec727af7-04e6-4cef-9c20-644fc83cbb03-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-6d845bb7dcp8rdn\" (UID: \"ec727af7-04e6-4cef-9c20-644fc83cbb03\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dcp8rdn"
Jul 28 10:29:10.548826 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:29:10.548781    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgqdt\" (UniqueName: \"kubernetes.io/projected/ec727af7-04e6-4cef-9c20-644fc83cbb03-kube-api-access-bgqdt\") pod \"auth-invalid-token-test-kserve-router-scheduler-6d845bb7dcp8rdn\" (UID: \"ec727af7-04e6-4cef-9c20-644fc83cbb03\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dcp8rdn"
Jul 28 10:29:10.551311 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:29:10.551288    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec727af7-04e6-4cef-9c20-644fc83cbb03-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-6d845bb7dcp8rdn\" (UID: \"ec727af7-04e6-4cef-9c20-644fc83cbb03\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dcp8rdn"
Jul 28 10:29:10.556806 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:29:10.556775    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgqdt\" (UniqueName: \"kubernetes.io/projected/ec727af7-04e6-4cef-9c20-644fc83cbb03-kube-api-access-bgqdt\") pod \"auth-invalid-token-test-kserve-router-scheduler-6d845bb7dcp8rdn\" (UID: \"ec727af7-04e6-4cef-9c20-644fc83cbb03\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dcp8rdn"
Jul 28 10:29:10.612132 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:29:10.612106    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dcp8rdn"
Jul 28 10:29:10.763566 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:29:10.763500    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dcp8rdn"]
Jul 28 10:29:10.765711 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:29:10.765684    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec727af7_04e6_4cef_9c20_644fc83cbb03.slice/crio-25084ffc38f8ce75add1ff16c6df8c79f50535bcc0e0f242aa22d82cb7993f49 WatchSource:0}: Error finding container 25084ffc38f8ce75add1ff16c6df8c79f50535bcc0e0f242aa22d82cb7993f49: Status 404 returned error can't find the container with id 25084ffc38f8ce75add1ff16c6df8c79f50535bcc0e0f242aa22d82cb7993f49
Jul 28 10:29:11.000031 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:29:10.999906    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dcp8rdn" event={"ID":"ec727af7-04e6-4cef-9c20-644fc83cbb03","Type":"ContainerStarted","Data":"9deebabc788d1f4700179710ec9fe42f46e712ed5bff72f025aaac0a1280ac90"}
Jul 28 10:29:11.000031 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:29:10.999978    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dcp8rdn"
Jul 28 10:29:11.000031 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:29:11.000002    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dcp8rdn" event={"ID":"ec727af7-04e6-4cef-9c20-644fc83cbb03","Type":"ContainerStarted","Data":"25084ffc38f8ce75add1ff16c6df8c79f50535bcc0e0f242aa22d82cb7993f49"}
Jul 28 10:29:11.001856 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:29:11.001824    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-59fc46f4dd-qfmvf" event={"ID":"eb61c5ec-0ef4-4d6f-9fc5-2c0351ddf4fd","Type":"ContainerStarted","Data":"2bafdde7c71a479f8457080d72f7738a887c0fd4307e1dfabefb60caeb64fa4d"}
Jul 28 10:29:11.002016 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:29:11.001878    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-59fc46f4dd-qfmvf" event={"ID":"eb61c5ec-0ef4-4d6f-9fc5-2c0351ddf4fd","Type":"ContainerStarted","Data":"9ca65b503e603336271d82048400dea0d7d88825a633e9309930bdc90c57d473"}
Jul 28 10:29:11.018830 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:29:11.018766    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dcp8rdn" podStartSLOduration=1.01874597 podStartE2EDuration="1.01874597s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:29:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:29:11.018133496 +0000 UTC m=+1285.281529824" watchObservedRunningTime="2026-07-28 10:29:11.01874597 +0000 UTC m=+1285.282142299"
Jul 28 10:29:12.342209 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:29:12.342167    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:29:12.443116 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:29:12.443077    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:29:15.026685 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:29:15.026591    2579 generic.go:356] "Generic (PLEG): container finished" podID="eb61c5ec-0ef4-4d6f-9fc5-2c0351ddf4fd" containerID="2bafdde7c71a479f8457080d72f7738a887c0fd4307e1dfabefb60caeb64fa4d" exitCode=0
Jul 28 10:29:15.026685 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:29:15.026654    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-59fc46f4dd-qfmvf" event={"ID":"eb61c5ec-0ef4-4d6f-9fc5-2c0351ddf4fd","Type":"ContainerDied","Data":"2bafdde7c71a479f8457080d72f7738a887c0fd4307e1dfabefb60caeb64fa4d"}
Jul 28 10:29:16.034299 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:29:16.034257    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-59fc46f4dd-qfmvf" event={"ID":"eb61c5ec-0ef4-4d6f-9fc5-2c0351ddf4fd","Type":"ContainerStarted","Data":"b3b1cbfa91188892f4be62643cceddd6b7b7367a82622bb6c55b2b1b2f8db83a"}
Jul 28 10:29:16.055621 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:29:16.055565    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-59fc46f4dd-qfmvf" podStartSLOduration=2.068697476 podStartE2EDuration="6.05554835s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.986850874s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:29:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:29:16.054585355 +0000 UTC m=+1290.317981681" watchObservedRunningTime="2026-07-28 10:29:16.05554835 +0000 UTC m=+1290.318944673"
Jul 28 10:29:16.395366 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:29:16.395287    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-59fc46f4dd-qfmvf"
Jul 28 10:29:20.395289 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:29:20.395250    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-59fc46f4dd-qfmvf"
Jul 28 10:29:20.396849 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:29:20.396824    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-59fc46f4dd-qfmvf" podUID="eb61c5ec-0ef4-4d6f-9fc5-2c0351ddf4fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused"
Jul 28 10:29:21.221849 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:29:21.221789    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:29:21.317516 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:29:21.317479    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:29:27.694943 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:29:27.694910    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:29:30.024340 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:29:30.024308    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:29:30.395077 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:29:30.395036    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-59fc46f4dd-qfmvf" podUID="eb61c5ec-0ef4-4d6f-9fc5-2c0351ddf4fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused"
Jul 28 10:29:33.777511 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:29:33.777474    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:29:33.842721 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:29:33.842684    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:29:38.543382 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:29:38.543344    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:29:40.395465 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:29:40.395421    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-59fc46f4dd-qfmvf" podUID="eb61c5ec-0ef4-4d6f-9fc5-2c0351ddf4fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused"
Jul 28 10:29:42.009354 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:29:42.009324    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dcp8rdn"
Jul 28 10:29:47.129643 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:29:47.129601    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:29:47.331079 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:29:47.331045    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5jjgnb"]
Jul 28 10:29:47.334037 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:29:47.334013    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5jjgnb"
Jul 28 10:29:47.337057 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:29:47.337033    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"kube-root-ca.crt\""
Jul 28 10:29:47.337193 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:29:47.337035    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"scheduler-inline-config-test-epp-sa-dockercfg-s5xjb\""
Jul 28 10:29:47.337193 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:29:47.337035    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"openshift-service-ca.crt\""
Jul 28 10:29:47.337193 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:29:47.337043    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"default-dockercfg-52jn5\""
Jul 28 10:29:47.337193 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:29:47.337156    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"scheduler-inline-config-test-kserve-self-signed-certs\""
Jul 28 10:29:47.349753 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:29:47.349722    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5jjgnb"]
Jul 28 10:29:47.395489 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:29:47.395394    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:29:47.457239 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:29:47.457191    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsn8w\" (UniqueName: \"kubernetes.io/projected/7e3f0a3e-42fa-470e-a281-77ef77b6a0ac-kube-api-access-hsn8w\") pod \"scheduler-inline-config-test-kserve-router-scheduler-6bbd5jjgnb\" (UID: \"7e3f0a3e-42fa-470e-a281-77ef77b6a0ac\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5jjgnb"
Jul 28 10:29:47.457416 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:29:47.457386    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e3f0a3e-42fa-470e-a281-77ef77b6a0ac-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-6bbd5jjgnb\" (UID: \"7e3f0a3e-42fa-470e-a281-77ef77b6a0ac\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5jjgnb"
Jul 28 10:29:47.558799 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:29:47.558761    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e3f0a3e-42fa-470e-a281-77ef77b6a0ac-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-6bbd5jjgnb\" (UID: \"7e3f0a3e-42fa-470e-a281-77ef77b6a0ac\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5jjgnb"
Jul 28 10:29:47.558998 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:29:47.558817    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsn8w\" (UniqueName: \"kubernetes.io/projected/7e3f0a3e-42fa-470e-a281-77ef77b6a0ac-kube-api-access-hsn8w\") pod \"scheduler-inline-config-test-kserve-router-scheduler-6bbd5jjgnb\" (UID: \"7e3f0a3e-42fa-470e-a281-77ef77b6a0ac\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5jjgnb"
Jul 28 10:29:47.561285 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:29:47.561258    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e3f0a3e-42fa-470e-a281-77ef77b6a0ac-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-6bbd5jjgnb\" (UID: \"7e3f0a3e-42fa-470e-a281-77ef77b6a0ac\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5jjgnb"
Jul 28 10:29:47.566514 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:29:47.566492    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsn8w\" (UniqueName: \"kubernetes.io/projected/7e3f0a3e-42fa-470e-a281-77ef77b6a0ac-kube-api-access-hsn8w\") pod \"scheduler-inline-config-test-kserve-router-scheduler-6bbd5jjgnb\" (UID: \"7e3f0a3e-42fa-470e-a281-77ef77b6a0ac\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5jjgnb"
Jul 28 10:29:47.648272 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:29:47.648182    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5jjgnb"
Jul 28 10:29:47.786447 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:29:47.786422    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5jjgnb"]
Jul 28 10:29:47.788225 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:29:47.788185    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e3f0a3e_42fa_470e_a281_77ef77b6a0ac.slice/crio-7d5970da7960eaa05202e6b7103a2a8695166857b3ae0c1a1381b71c3f07cd67 WatchSource:0}: Error finding container 7d5970da7960eaa05202e6b7103a2a8695166857b3ae0c1a1381b71c3f07cd67: Status 404 returned error can't find the container with id 7d5970da7960eaa05202e6b7103a2a8695166857b3ae0c1a1381b71c3f07cd67
Jul 28 10:29:48.209175 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:29:48.209138    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5jjgnb" event={"ID":"7e3f0a3e-42fa-470e-a281-77ef77b6a0ac","Type":"ContainerStarted","Data":"e9bf90a57c21f5afcb6e0da7894e31e263bde346e862e8cf8287a9a1842edeba"}
Jul 28 10:29:48.209582 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:29:48.209196    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5jjgnb" event={"ID":"7e3f0a3e-42fa-470e-a281-77ef77b6a0ac","Type":"ContainerStarted","Data":"7d5970da7960eaa05202e6b7103a2a8695166857b3ae0c1a1381b71c3f07cd67"}
Jul 28 10:29:48.209582 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:29:48.209240    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5jjgnb"
Jul 28 10:29:48.226123 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:29:48.226070    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5jjgnb" podStartSLOduration=1.226055657 podStartE2EDuration="1.226055657s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:29:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:29:48.224902668 +0000 UTC m=+1322.488298999" watchObservedRunningTime="2026-07-28 10:29:48.226055657 +0000 UTC m=+1322.489451985"
Jul 28 10:29:50.395806 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:29:50.395758    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-59fc46f4dd-qfmvf" podUID="eb61c5ec-0ef4-4d6f-9fc5-2c0351ddf4fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused"
Jul 28 10:30:00.395024 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:30:00.394980    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-59fc46f4dd-qfmvf" podUID="eb61c5ec-0ef4-4d6f-9fc5-2c0351ddf4fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused"
Jul 28 10:30:10.395403 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:30:10.395358    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-59fc46f4dd-qfmvf" podUID="eb61c5ec-0ef4-4d6f-9fc5-2c0351ddf4fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused"
Jul 28 10:30:19.218706 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:30:19.218676    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5jjgnb"
Jul 28 10:30:20.395277 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:30:20.395235    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-59fc46f4dd-qfmvf" podUID="eb61c5ec-0ef4-4d6f-9fc5-2c0351ddf4fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused"
Jul 28 10:30:22.061513 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:30:22.061470    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:30:25.314202 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:30:25.314167    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5jjgnb"]
Jul 28 10:30:25.324010 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:30:25.323964    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:30:25.416864 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:30:25.416797    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:30:29.218614 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:30:29.218515    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5jjgnb" podUID="7e3f0a3e-42fa-470e-a281-77ef77b6a0ac" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 10:30:30.395527 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:30:30.395480    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-59fc46f4dd-qfmvf" podUID="eb61c5ec-0ef4-4d6f-9fc5-2c0351ddf4fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused"
Jul 28 10:30:39.218708 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:30:39.218666    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5jjgnb" podUID="7e3f0a3e-42fa-470e-a281-77ef77b6a0ac" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 10:30:40.342423 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:30:40.342348    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5jjgnb" podUID="7e3f0a3e-42fa-470e-a281-77ef77b6a0ac" containerName="main" containerID="cri-o://e9bf90a57c21f5afcb6e0da7894e31e263bde346e862e8cf8287a9a1842edeba" gracePeriod=45
Jul 28 10:30:40.395424 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:30:40.395368    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-59fc46f4dd-qfmvf" podUID="eb61c5ec-0ef4-4d6f-9fc5-2c0351ddf4fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused"
Jul 28 10:30:40.495430 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:30:40.495389    2579 generic.go:356] "Generic (PLEG): container finished" podID="7e3f0a3e-42fa-470e-a281-77ef77b6a0ac" containerID="e9bf90a57c21f5afcb6e0da7894e31e263bde346e862e8cf8287a9a1842edeba" exitCode=0
Jul 28 10:30:40.495687 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:30:40.495456    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5jjgnb" event={"ID":"7e3f0a3e-42fa-470e-a281-77ef77b6a0ac","Type":"ContainerDied","Data":"e9bf90a57c21f5afcb6e0da7894e31e263bde346e862e8cf8287a9a1842edeba"}
Jul 28 10:30:40.602680 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:30:40.602611    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5jjgnb"
Jul 28 10:30:40.739002 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:30:40.738966    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hsn8w\" (UniqueName: \"kubernetes.io/projected/7e3f0a3e-42fa-470e-a281-77ef77b6a0ac-kube-api-access-hsn8w\") pod \"7e3f0a3e-42fa-470e-a281-77ef77b6a0ac\" (UID: \"7e3f0a3e-42fa-470e-a281-77ef77b6a0ac\") "
Jul 28 10:30:40.739002 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:30:40.739001    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e3f0a3e-42fa-470e-a281-77ef77b6a0ac-tls-certs\") pod \"7e3f0a3e-42fa-470e-a281-77ef77b6a0ac\" (UID: \"7e3f0a3e-42fa-470e-a281-77ef77b6a0ac\") "
Jul 28 10:30:40.741320 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:30:40.741292    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e3f0a3e-42fa-470e-a281-77ef77b6a0ac-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7e3f0a3e-42fa-470e-a281-77ef77b6a0ac" (UID: "7e3f0a3e-42fa-470e-a281-77ef77b6a0ac"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:30:40.741434 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:30:40.741298    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e3f0a3e-42fa-470e-a281-77ef77b6a0ac-kube-api-access-hsn8w" (OuterVolumeSpecName: "kube-api-access-hsn8w") pod "7e3f0a3e-42fa-470e-a281-77ef77b6a0ac" (UID: "7e3f0a3e-42fa-470e-a281-77ef77b6a0ac"). InnerVolumeSpecName "kube-api-access-hsn8w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:30:40.840156 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:30:40.840115    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hsn8w\" (UniqueName: \"kubernetes.io/projected/7e3f0a3e-42fa-470e-a281-77ef77b6a0ac-kube-api-access-hsn8w\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 10:30:40.840156 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:30:40.840149    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e3f0a3e-42fa-470e-a281-77ef77b6a0ac-tls-certs\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 10:30:41.501913 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:30:41.501877    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5jjgnb" event={"ID":"7e3f0a3e-42fa-470e-a281-77ef77b6a0ac","Type":"ContainerDied","Data":"7d5970da7960eaa05202e6b7103a2a8695166857b3ae0c1a1381b71c3f07cd67"}
Jul 28 10:30:41.502367 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:30:41.501948    2579 scope.go:117] "RemoveContainer" containerID="e9bf90a57c21f5afcb6e0da7894e31e263bde346e862e8cf8287a9a1842edeba"
Jul 28 10:30:41.502367 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:30:41.501882    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5jjgnb"
Jul 28 10:30:41.525987 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:30:41.525945    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5jjgnb"]
Jul 28 10:30:41.531652 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:30:41.531618    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5jjgnb"]
Jul 28 10:30:42.276643 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:30:42.276610    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e3f0a3e-42fa-470e-a281-77ef77b6a0ac" path="/var/lib/kubelet/pods/7e3f0a3e-42fa-470e-a281-77ef77b6a0ac/volumes"
Jul 28 10:30:45.325412 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:30:45.325375    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:30:45.423289 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:30:45.423245    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:30:50.395564 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:30:50.395485    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-59fc46f4dd-qfmvf" podUID="eb61c5ec-0ef4-4d6f-9fc5-2c0351ddf4fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused"
Jul 28 10:30:51.388404 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:30:51.388364    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:30:51.614945 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:30:51.614903    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schevj7hn"]
Jul 28 10:30:51.615303 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:30:51.615241    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e3f0a3e-42fa-470e-a281-77ef77b6a0ac" containerName="main"
Jul 28 10:30:51.615303 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:30:51.615252    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e3f0a3e-42fa-470e-a281-77ef77b6a0ac" containerName="main"
Jul 28 10:30:51.615376 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:30:51.615304    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e3f0a3e-42fa-470e-a281-77ef77b6a0ac" containerName="main"
Jul 28 10:30:51.619576 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:30:51.619553    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schevj7hn"
Jul 28 10:30:51.622574 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:30:51.622530    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"openshift-service-ca.crt\""
Jul 28 10:30:51.622732 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:30:51.622591    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"kube-root-ca.crt\""
Jul 28 10:30:51.622732 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:30:51.622530    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"llmisv431d7e30f13554296364403a5b6365f5-kserve-self-signed-certs\""
Jul 28 10:30:51.622732 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:30:51.622677    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"default-dockercfg-klrxv\""
Jul 28 10:30:51.622931 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:30:51.622770    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-epp-sa-dockercfg-949l6\""
Jul 28 10:30:51.628360 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:30:51.628336    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schevj7hn"]
Jul 28 10:30:51.640477 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:30:51.640410    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:30:51.740392 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:30:51.740359    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a24a487-b865-44b9-8479-81aaf4a9eacc-tls-certs\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schevj7hn\" (UID: \"6a24a487-b865-44b9-8479-81aaf4a9eacc\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schevj7hn"
Jul 28 10:30:51.740583 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:30:51.740413    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpcq2\" (UniqueName: \"kubernetes.io/projected/6a24a487-b865-44b9-8479-81aaf4a9eacc-kube-api-access-kpcq2\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schevj7hn\" (UID: \"6a24a487-b865-44b9-8479-81aaf4a9eacc\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schevj7hn"
Jul 28 10:30:51.841894 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:30:51.841851    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpcq2\" (UniqueName: \"kubernetes.io/projected/6a24a487-b865-44b9-8479-81aaf4a9eacc-kube-api-access-kpcq2\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schevj7hn\" (UID: \"6a24a487-b865-44b9-8479-81aaf4a9eacc\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schevj7hn"
Jul 28 10:30:51.842200 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:30:51.841956    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a24a487-b865-44b9-8479-81aaf4a9eacc-tls-certs\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schevj7hn\" (UID: \"6a24a487-b865-44b9-8479-81aaf4a9eacc\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schevj7hn"
Jul 28 10:30:51.844608 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:30:51.844582    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a24a487-b865-44b9-8479-81aaf4a9eacc-tls-certs\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schevj7hn\" (UID: \"6a24a487-b865-44b9-8479-81aaf4a9eacc\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schevj7hn"
Jul 28 10:30:51.850396 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:30:51.850363    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpcq2\" (UniqueName: \"kubernetes.io/projected/6a24a487-b865-44b9-8479-81aaf4a9eacc-kube-api-access-kpcq2\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schevj7hn\" (UID: \"6a24a487-b865-44b9-8479-81aaf4a9eacc\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schevj7hn"
Jul 28 10:30:51.931190 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:30:51.931089    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schevj7hn"
Jul 28 10:30:52.039159 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:30:52.039129    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:30:52.076511 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:30:52.076460    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schevj7hn"]
Jul 28 10:30:52.078850 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:30:52.078818    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a24a487_b865_44b9_8479_81aaf4a9eacc.slice/crio-e203fa7b0419ed482ac01c1aae333864cd516d4c6b687107967370173908affd WatchSource:0}: Error finding container e203fa7b0419ed482ac01c1aae333864cd516d4c6b687107967370173908affd: Status 404 returned error can't find the container with id e203fa7b0419ed482ac01c1aae333864cd516d4c6b687107967370173908affd
Jul 28 10:30:52.561948 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:30:52.561909    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schevj7hn" event={"ID":"6a24a487-b865-44b9-8479-81aaf4a9eacc","Type":"ContainerStarted","Data":"2db2922455aa610302f8fa1cb5a43f6ea5d2579079f3b1e4c810fe7c29f98baa"}
Jul 28 10:30:52.562162 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:30:52.561981    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schevj7hn"
Jul 28 10:30:52.562162 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:30:52.562004    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schevj7hn" event={"ID":"6a24a487-b865-44b9-8479-81aaf4a9eacc","Type":"ContainerStarted","Data":"e203fa7b0419ed482ac01c1aae333864cd516d4c6b687107967370173908affd"}
Jul 28 10:30:52.592392 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:30:52.592336    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schevj7hn" podStartSLOduration=1.592318074 podStartE2EDuration="1.592318074s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:30:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:30:52.589453251 +0000 UTC m=+1386.852849580" watchObservedRunningTime="2026-07-28 10:30:52.592318074 +0000 UTC m=+1386.855714403"
Jul 28 10:30:56.317269 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:30:56.317202    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:30:57.890652 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:30:57.890590    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:30:57.954311 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:30:57.954276    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:31:00.395284 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:31:00.395231    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-59fc46f4dd-qfmvf" podUID="eb61c5ec-0ef4-4d6f-9fc5-2c0351ddf4fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused"
Jul 28 10:31:10.405794 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:31:10.405761    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-59fc46f4dd-qfmvf"
Jul 28 10:31:10.413775 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:31:10.413750    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-59fc46f4dd-qfmvf"
Jul 28 10:31:11.710249 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:31:11.710217    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dcp8rdn"]
Jul 28 10:31:11.716874 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:31:11.716797    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:31:11.717665 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:31:11.717642    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-59fc46f4dd-qfmvf"]
Jul 28 10:31:11.816528 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:31:11.816477    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:31:12.009246 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:31:12.009153    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dcp8rdn" podUID="ec727af7-04e6-4cef-9c20-644fc83cbb03" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 10:31:22.008957 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:31:22.008915    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dcp8rdn" podUID="ec727af7-04e6-4cef-9c20-644fc83cbb03" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 10:31:23.568297 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:31:23.568267    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schevj7hn"
Jul 28 10:31:24.957527 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:31:24.957482    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:31:24.959255 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:31:24.959227    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schevj7hn"]
Jul 28 10:31:25.015645 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:31:25.015606    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:31:26.729947 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:31:26.729906    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dcp8rdn" podUID="ec727af7-04e6-4cef-9c20-644fc83cbb03" containerName="main" containerID="cri-o://9deebabc788d1f4700179710ec9fe42f46e712ed5bff72f025aaac0a1280ac90" gracePeriod=45
Jul 28 10:31:26.743934 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:31:26.743893    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-59fc46f4dd-qfmvf" podUID="eb61c5ec-0ef4-4d6f-9fc5-2c0351ddf4fd" containerName="main" containerID="cri-o://b3b1cbfa91188892f4be62643cceddd6b7b7367a82622bb6c55b2b1b2f8db83a" gracePeriod=45
Jul 28 10:31:26.984422 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:31:26.984353    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dcp8rdn"
Jul 28 10:31:27.147444 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:31:27.147406    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec727af7-04e6-4cef-9c20-644fc83cbb03-tls-certs\") pod \"ec727af7-04e6-4cef-9c20-644fc83cbb03\" (UID: \"ec727af7-04e6-4cef-9c20-644fc83cbb03\") "
Jul 28 10:31:27.147657 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:31:27.147492    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bgqdt\" (UniqueName: \"kubernetes.io/projected/ec727af7-04e6-4cef-9c20-644fc83cbb03-kube-api-access-bgqdt\") pod \"ec727af7-04e6-4cef-9c20-644fc83cbb03\" (UID: \"ec727af7-04e6-4cef-9c20-644fc83cbb03\") "
Jul 28 10:31:27.149692 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:31:27.149660    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec727af7-04e6-4cef-9c20-644fc83cbb03-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ec727af7-04e6-4cef-9c20-644fc83cbb03" (UID: "ec727af7-04e6-4cef-9c20-644fc83cbb03"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:31:27.149692 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:31:27.149680    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec727af7-04e6-4cef-9c20-644fc83cbb03-kube-api-access-bgqdt" (OuterVolumeSpecName: "kube-api-access-bgqdt") pod "ec727af7-04e6-4cef-9c20-644fc83cbb03" (UID: "ec727af7-04e6-4cef-9c20-644fc83cbb03"). InnerVolumeSpecName "kube-api-access-bgqdt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:31:27.248668 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:31:27.248579    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec727af7-04e6-4cef-9c20-644fc83cbb03-tls-certs\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 10:31:27.248668 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:31:27.248613    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bgqdt\" (UniqueName: \"kubernetes.io/projected/ec727af7-04e6-4cef-9c20-644fc83cbb03-kube-api-access-bgqdt\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 10:31:27.407232 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:31:27.407187    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-59fc46f4dd-qfmvf" podUID="eb61c5ec-0ef4-4d6f-9fc5-2c0351ddf4fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused"
Jul 28 10:31:27.475092 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:31:27.475066    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-59fc46f4dd-qfmvf"
Jul 28 10:31:27.652214 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:31:27.652184    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb61c5ec-0ef4-4d6f-9fc5-2c0351ddf4fd-dshm\") pod \"eb61c5ec-0ef4-4d6f-9fc5-2c0351ddf4fd\" (UID: \"eb61c5ec-0ef4-4d6f-9fc5-2c0351ddf4fd\") "
Jul 28 10:31:27.652392 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:31:27.652229    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb61c5ec-0ef4-4d6f-9fc5-2c0351ddf4fd-tmp-dir\") pod \"eb61c5ec-0ef4-4d6f-9fc5-2c0351ddf4fd\" (UID: \"eb61c5ec-0ef4-4d6f-9fc5-2c0351ddf4fd\") "
Jul 28 10:31:27.652392 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:31:27.652273    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eb61c5ec-0ef4-4d6f-9fc5-2c0351ddf4fd-kserve-provision-location\") pod \"eb61c5ec-0ef4-4d6f-9fc5-2c0351ddf4fd\" (UID: \"eb61c5ec-0ef4-4d6f-9fc5-2c0351ddf4fd\") "
Jul 28 10:31:27.652392 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:31:27.652304    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tdrdh\" (UniqueName: \"kubernetes.io/projected/eb61c5ec-0ef4-4d6f-9fc5-2c0351ddf4fd-kube-api-access-tdrdh\") pod \"eb61c5ec-0ef4-4d6f-9fc5-2c0351ddf4fd\" (UID: \"eb61c5ec-0ef4-4d6f-9fc5-2c0351ddf4fd\") "
Jul 28 10:31:27.652392 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:31:27.652327    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb61c5ec-0ef4-4d6f-9fc5-2c0351ddf4fd-model-cache\") pod \"eb61c5ec-0ef4-4d6f-9fc5-2c0351ddf4fd\" (UID: \"eb61c5ec-0ef4-4d6f-9fc5-2c0351ddf4fd\") "
Jul 28 10:31:27.652392 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:31:27.652376    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb61c5ec-0ef4-4d6f-9fc5-2c0351ddf4fd-tls-certs\") pod \"eb61c5ec-0ef4-4d6f-9fc5-2c0351ddf4fd\" (UID: \"eb61c5ec-0ef4-4d6f-9fc5-2c0351ddf4fd\") "
Jul 28 10:31:27.652675 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:31:27.652409    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb61c5ec-0ef4-4d6f-9fc5-2c0351ddf4fd-home\") pod \"eb61c5ec-0ef4-4d6f-9fc5-2c0351ddf4fd\" (UID: \"eb61c5ec-0ef4-4d6f-9fc5-2c0351ddf4fd\") "
Jul 28 10:31:27.652675 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:31:27.652605    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb61c5ec-0ef4-4d6f-9fc5-2c0351ddf4fd-model-cache" (OuterVolumeSpecName: "model-cache") pod "eb61c5ec-0ef4-4d6f-9fc5-2c0351ddf4fd" (UID: "eb61c5ec-0ef4-4d6f-9fc5-2c0351ddf4fd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:31:27.653241 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:31:27.653212    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb61c5ec-0ef4-4d6f-9fc5-2c0351ddf4fd-home" (OuterVolumeSpecName: "home") pod "eb61c5ec-0ef4-4d6f-9fc5-2c0351ddf4fd" (UID: "eb61c5ec-0ef4-4d6f-9fc5-2c0351ddf4fd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:31:27.654592 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:31:27.654562    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb61c5ec-0ef4-4d6f-9fc5-2c0351ddf4fd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "eb61c5ec-0ef4-4d6f-9fc5-2c0351ddf4fd" (UID: "eb61c5ec-0ef4-4d6f-9fc5-2c0351ddf4fd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:31:27.654887 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:31:27.654860    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb61c5ec-0ef4-4d6f-9fc5-2c0351ddf4fd-dshm" (OuterVolumeSpecName: "dshm") pod "eb61c5ec-0ef4-4d6f-9fc5-2c0351ddf4fd" (UID: "eb61c5ec-0ef4-4d6f-9fc5-2c0351ddf4fd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:31:27.654995 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:31:27.654913    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb61c5ec-0ef4-4d6f-9fc5-2c0351ddf4fd-kube-api-access-tdrdh" (OuterVolumeSpecName: "kube-api-access-tdrdh") pod "eb61c5ec-0ef4-4d6f-9fc5-2c0351ddf4fd" (UID: "eb61c5ec-0ef4-4d6f-9fc5-2c0351ddf4fd"). InnerVolumeSpecName "kube-api-access-tdrdh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:31:27.664214 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:31:27.664189    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb61c5ec-0ef4-4d6f-9fc5-2c0351ddf4fd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "eb61c5ec-0ef4-4d6f-9fc5-2c0351ddf4fd" (UID: "eb61c5ec-0ef4-4d6f-9fc5-2c0351ddf4fd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:31:27.716699 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:31:27.716522    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb61c5ec-0ef4-4d6f-9fc5-2c0351ddf4fd-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "eb61c5ec-0ef4-4d6f-9fc5-2c0351ddf4fd" (UID: "eb61c5ec-0ef4-4d6f-9fc5-2c0351ddf4fd"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:31:27.748040 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:31:27.748012    2579 generic.go:356] "Generic (PLEG): container finished" podID="ec727af7-04e6-4cef-9c20-644fc83cbb03" containerID="9deebabc788d1f4700179710ec9fe42f46e712ed5bff72f025aaac0a1280ac90" exitCode=0
Jul 28 10:31:27.748423 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:31:27.748068    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dcp8rdn"
Jul 28 10:31:27.748423 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:31:27.748112    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dcp8rdn" event={"ID":"ec727af7-04e6-4cef-9c20-644fc83cbb03","Type":"ContainerDied","Data":"9deebabc788d1f4700179710ec9fe42f46e712ed5bff72f025aaac0a1280ac90"}
Jul 28 10:31:27.748423 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:31:27.748193    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dcp8rdn" event={"ID":"ec727af7-04e6-4cef-9c20-644fc83cbb03","Type":"ContainerDied","Data":"25084ffc38f8ce75add1ff16c6df8c79f50535bcc0e0f242aa22d82cb7993f49"}
Jul 28 10:31:27.748423 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:31:27.748289    2579 scope.go:117] "RemoveContainer" containerID="9deebabc788d1f4700179710ec9fe42f46e712ed5bff72f025aaac0a1280ac90"
Jul 28 10:31:27.750289 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:31:27.750197    2579 generic.go:356] "Generic (PLEG): container finished" podID="eb61c5ec-0ef4-4d6f-9fc5-2c0351ddf4fd" containerID="b3b1cbfa91188892f4be62643cceddd6b7b7367a82622bb6c55b2b1b2f8db83a" exitCode=0
Jul 28 10:31:27.750289 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:31:27.750276    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-59fc46f4dd-qfmvf"
Jul 28 10:31:27.750405 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:31:27.750278    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-59fc46f4dd-qfmvf" event={"ID":"eb61c5ec-0ef4-4d6f-9fc5-2c0351ddf4fd","Type":"ContainerDied","Data":"b3b1cbfa91188892f4be62643cceddd6b7b7367a82622bb6c55b2b1b2f8db83a"}
Jul 28 10:31:27.750405 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:31:27.750350    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-59fc46f4dd-qfmvf" event={"ID":"eb61c5ec-0ef4-4d6f-9fc5-2c0351ddf4fd","Type":"ContainerDied","Data":"9ca65b503e603336271d82048400dea0d7d88825a633e9309930bdc90c57d473"}
Jul 28 10:31:27.753060 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:31:27.753029    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb61c5ec-0ef4-4d6f-9fc5-2c0351ddf4fd-tls-certs\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 10:31:27.753158 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:31:27.753062    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb61c5ec-0ef4-4d6f-9fc5-2c0351ddf4fd-home\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 10:31:27.753158 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:31:27.753074    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb61c5ec-0ef4-4d6f-9fc5-2c0351ddf4fd-dshm\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 10:31:27.753158 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:31:27.753084    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb61c5ec-0ef4-4d6f-9fc5-2c0351ddf4fd-tmp-dir\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 10:31:27.753158 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:31:27.753097    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eb61c5ec-0ef4-4d6f-9fc5-2c0351ddf4fd-kserve-provision-location\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 10:31:27.753158 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:31:27.753109    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdrdh\" (UniqueName: \"kubernetes.io/projected/eb61c5ec-0ef4-4d6f-9fc5-2c0351ddf4fd-kube-api-access-tdrdh\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 10:31:27.753158 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:31:27.753121    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb61c5ec-0ef4-4d6f-9fc5-2c0351ddf4fd-model-cache\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 10:31:27.757680 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:31:27.757663    2579 scope.go:117] "RemoveContainer" containerID="9deebabc788d1f4700179710ec9fe42f46e712ed5bff72f025aaac0a1280ac90"
Jul 28 10:31:27.757999 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:31:27.757973    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9deebabc788d1f4700179710ec9fe42f46e712ed5bff72f025aaac0a1280ac90\": container with ID starting with 9deebabc788d1f4700179710ec9fe42f46e712ed5bff72f025aaac0a1280ac90 not found: ID does not exist" containerID="9deebabc788d1f4700179710ec9fe42f46e712ed5bff72f025aaac0a1280ac90"
Jul 28 10:31:27.758115 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:31:27.758008    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9deebabc788d1f4700179710ec9fe42f46e712ed5bff72f025aaac0a1280ac90"} err="failed to get container status \"9deebabc788d1f4700179710ec9fe42f46e712ed5bff72f025aaac0a1280ac90\": rpc error: code = NotFound desc = could not find container \"9deebabc788d1f4700179710ec9fe42f46e712ed5bff72f025aaac0a1280ac90\": container with ID starting with 9deebabc788d1f4700179710ec9fe42f46e712ed5bff72f025aaac0a1280ac90 not found: ID does not exist"
Jul 28 10:31:27.758115 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:31:27.758032    2579 scope.go:117] "RemoveContainer" containerID="b3b1cbfa91188892f4be62643cceddd6b7b7367a82622bb6c55b2b1b2f8db83a"
Jul 28 10:31:27.766466 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:31:27.766448    2579 scope.go:117] "RemoveContainer" containerID="2bafdde7c71a479f8457080d72f7738a887c0fd4307e1dfabefb60caeb64fa4d"
Jul 28 10:31:27.774278 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:31:27.774258    2579 scope.go:117] "RemoveContainer" containerID="b3b1cbfa91188892f4be62643cceddd6b7b7367a82622bb6c55b2b1b2f8db83a"
Jul 28 10:31:27.774567 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:31:27.774525    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3b1cbfa91188892f4be62643cceddd6b7b7367a82622bb6c55b2b1b2f8db83a\": container with ID starting with b3b1cbfa91188892f4be62643cceddd6b7b7367a82622bb6c55b2b1b2f8db83a not found: ID does not exist" containerID="b3b1cbfa91188892f4be62643cceddd6b7b7367a82622bb6c55b2b1b2f8db83a"
Jul 28 10:31:27.774649 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:31:27.774573    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3b1cbfa91188892f4be62643cceddd6b7b7367a82622bb6c55b2b1b2f8db83a"} err="failed to get container status \"b3b1cbfa91188892f4be62643cceddd6b7b7367a82622bb6c55b2b1b2f8db83a\": rpc error: code = NotFound desc = could not find container \"b3b1cbfa91188892f4be62643cceddd6b7b7367a82622bb6c55b2b1b2f8db83a\": container with ID starting with b3b1cbfa91188892f4be62643cceddd6b7b7367a82622bb6c55b2b1b2f8db83a not found: ID does not exist"
Jul 28 10:31:27.774649 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:31:27.774591    2579 scope.go:117] "RemoveContainer" containerID="2bafdde7c71a479f8457080d72f7738a887c0fd4307e1dfabefb60caeb64fa4d"
Jul 28 10:31:27.774833 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:31:27.774817    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bafdde7c71a479f8457080d72f7738a887c0fd4307e1dfabefb60caeb64fa4d\": container with ID starting with 2bafdde7c71a479f8457080d72f7738a887c0fd4307e1dfabefb60caeb64fa4d not found: ID does not exist" containerID="2bafdde7c71a479f8457080d72f7738a887c0fd4307e1dfabefb60caeb64fa4d"
Jul 28 10:31:27.774883 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:31:27.774836    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bafdde7c71a479f8457080d72f7738a887c0fd4307e1dfabefb60caeb64fa4d"} err="failed to get container status \"2bafdde7c71a479f8457080d72f7738a887c0fd4307e1dfabefb60caeb64fa4d\": rpc error: code = NotFound desc = could not find container \"2bafdde7c71a479f8457080d72f7738a887c0fd4307e1dfabefb60caeb64fa4d\": container with ID starting with 2bafdde7c71a479f8457080d72f7738a887c0fd4307e1dfabefb60caeb64fa4d not found: ID does not exist"
Jul 28 10:31:27.775172 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:31:27.775154    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-59fc46f4dd-qfmvf"]
Jul 28 10:31:27.778937 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:31:27.778914    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-59fc46f4dd-qfmvf"]
Jul 28 10:31:27.789992 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:31:27.789971    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dcp8rdn"]
Jul 28 10:31:27.791996 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:31:27.791974    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dcp8rdn"]
Jul 28 10:31:28.276264 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:31:28.276230    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eb61c5ec-0ef4-4d6f-9fc5-2c0351ddf4fd" path="/var/lib/kubelet/pods/eb61c5ec-0ef4-4d6f-9fc5-2c0351ddf4fd/volumes"
Jul 28 10:31:28.276657 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:31:28.276643    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec727af7-04e6-4cef-9c20-644fc83cbb03" path="/var/lib/kubelet/pods/ec727af7-04e6-4cef-9c20-644fc83cbb03/volumes"
Jul 28 10:31:31.711216 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:31:31.711185    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:31:31.812153 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:31:31.812123    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:31:33.568743 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:31:33.568708    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schevj7hn" podUID="6a24a487-b865-44b9-8479-81aaf4a9eacc" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 10:31:37.699409 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:31:37.699375    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:31:39.984283 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:31:39.984238    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schevj7hn" podUID="6a24a487-b865-44b9-8479-81aaf4a9eacc" containerName="main" containerID="cri-o://2db2922455aa610302f8fa1cb5a43f6ea5d2579079f3b1e4c810fe7c29f98baa" gracePeriod=45
Jul 28 10:31:40.236586 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:31:40.236495    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schevj7hn"
Jul 28 10:31:40.350685 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:31:40.350644    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a24a487-b865-44b9-8479-81aaf4a9eacc-tls-certs\") pod \"6a24a487-b865-44b9-8479-81aaf4a9eacc\" (UID: \"6a24a487-b865-44b9-8479-81aaf4a9eacc\") "
Jul 28 10:31:40.350886 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:31:40.350731    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kpcq2\" (UniqueName: \"kubernetes.io/projected/6a24a487-b865-44b9-8479-81aaf4a9eacc-kube-api-access-kpcq2\") pod \"6a24a487-b865-44b9-8479-81aaf4a9eacc\" (UID: \"6a24a487-b865-44b9-8479-81aaf4a9eacc\") "
Jul 28 10:31:40.352887 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:31:40.352853    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a24a487-b865-44b9-8479-81aaf4a9eacc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6a24a487-b865-44b9-8479-81aaf4a9eacc" (UID: "6a24a487-b865-44b9-8479-81aaf4a9eacc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:31:40.353004 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:31:40.352888    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a24a487-b865-44b9-8479-81aaf4a9eacc-kube-api-access-kpcq2" (OuterVolumeSpecName: "kube-api-access-kpcq2") pod "6a24a487-b865-44b9-8479-81aaf4a9eacc" (UID: "6a24a487-b865-44b9-8479-81aaf4a9eacc"). InnerVolumeSpecName "kube-api-access-kpcq2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:31:40.452124 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:31:40.452079    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a24a487-b865-44b9-8479-81aaf4a9eacc-tls-certs\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 10:31:40.452124 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:31:40.452115    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kpcq2\" (UniqueName: \"kubernetes.io/projected/6a24a487-b865-44b9-8479-81aaf4a9eacc-kube-api-access-kpcq2\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 10:31:40.821067 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:31:40.821021    2579 generic.go:356] "Generic (PLEG): container finished" podID="6a24a487-b865-44b9-8479-81aaf4a9eacc" containerID="2db2922455aa610302f8fa1cb5a43f6ea5d2579079f3b1e4c810fe7c29f98baa" exitCode=0
Jul 28 10:31:40.821248 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:31:40.821078    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schevj7hn"
Jul 28 10:31:40.821248 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:31:40.821091    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schevj7hn" event={"ID":"6a24a487-b865-44b9-8479-81aaf4a9eacc","Type":"ContainerDied","Data":"2db2922455aa610302f8fa1cb5a43f6ea5d2579079f3b1e4c810fe7c29f98baa"}
Jul 28 10:31:40.821248 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:31:40.821148    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schevj7hn" event={"ID":"6a24a487-b865-44b9-8479-81aaf4a9eacc","Type":"ContainerDied","Data":"e203fa7b0419ed482ac01c1aae333864cd516d4c6b687107967370173908affd"}
Jul 28 10:31:40.821248 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:31:40.821177    2579 scope.go:117] "RemoveContainer" containerID="2db2922455aa610302f8fa1cb5a43f6ea5d2579079f3b1e4c810fe7c29f98baa"
Jul 28 10:31:40.829994 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:31:40.829975    2579 scope.go:117] "RemoveContainer" containerID="2db2922455aa610302f8fa1cb5a43f6ea5d2579079f3b1e4c810fe7c29f98baa"
Jul 28 10:31:40.830235 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:31:40.830220    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2db2922455aa610302f8fa1cb5a43f6ea5d2579079f3b1e4c810fe7c29f98baa\": container with ID starting with 2db2922455aa610302f8fa1cb5a43f6ea5d2579079f3b1e4c810fe7c29f98baa not found: ID does not exist" containerID="2db2922455aa610302f8fa1cb5a43f6ea5d2579079f3b1e4c810fe7c29f98baa"
Jul 28 10:31:40.830277 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:31:40.830243    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2db2922455aa610302f8fa1cb5a43f6ea5d2579079f3b1e4c810fe7c29f98baa"} err="failed to get container status \"2db2922455aa610302f8fa1cb5a43f6ea5d2579079f3b1e4c810fe7c29f98baa\": rpc error: code = NotFound desc = could not find container \"2db2922455aa610302f8fa1cb5a43f6ea5d2579079f3b1e4c810fe7c29f98baa\": container with ID starting with 2db2922455aa610302f8fa1cb5a43f6ea5d2579079f3b1e4c810fe7c29f98baa not found: ID does not exist"
Jul 28 10:31:40.842814 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:31:40.842791    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schevj7hn"]
Jul 28 10:31:40.848650 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:31:40.848626    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schevj7hn"]
Jul 28 10:31:42.275610 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:31:42.275577    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a24a487-b865-44b9-8479-81aaf4a9eacc" path="/var/lib/kubelet/pods/6a24a487-b865-44b9-8479-81aaf4a9eacc/volumes"
Jul 28 10:31:43.592366 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:31:43.592325    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:31:44.710445 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:31:44.710401    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:31:44.811191 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:31:44.811154    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:31:51.704502 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:31:51.704460    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:31:56.298233 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:31:56.298193    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:31:56.369383 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:31:56.369341    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:31:58.426598 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:31:58.426531    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:31:58.683348 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:31:58.683271    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-mxjkf"]
Jul 28 10:31:58.683614 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:31:58.683600    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb61c5ec-0ef4-4d6f-9fc5-2c0351ddf4fd" containerName="storage-initializer"
Jul 28 10:31:58.683663 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:31:58.683616    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb61c5ec-0ef4-4d6f-9fc5-2c0351ddf4fd" containerName="storage-initializer"
Jul 28 10:31:58.683663 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:31:58.683629    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a24a487-b865-44b9-8479-81aaf4a9eacc" containerName="main"
Jul 28 10:31:58.683663 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:31:58.683635    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a24a487-b865-44b9-8479-81aaf4a9eacc" containerName="main"
Jul 28 10:31:58.683663 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:31:58.683647    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb61c5ec-0ef4-4d6f-9fc5-2c0351ddf4fd" containerName="main"
Jul 28 10:31:58.683663 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:31:58.683652    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb61c5ec-0ef4-4d6f-9fc5-2c0351ddf4fd" containerName="main"
Jul 28 10:31:58.683663 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:31:58.683661    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec727af7-04e6-4cef-9c20-644fc83cbb03" containerName="main"
Jul 28 10:31:58.683663 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:31:58.683666    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec727af7-04e6-4cef-9c20-644fc83cbb03" containerName="main"
Jul 28 10:31:58.683875 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:31:58.683714    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a24a487-b865-44b9-8479-81aaf4a9eacc" containerName="main"
Jul 28 10:31:58.683875 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:31:58.683723    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb61c5ec-0ef4-4d6f-9fc5-2c0351ddf4fd" containerName="main"
Jul 28 10:31:58.683875 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:31:58.683730    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec727af7-04e6-4cef-9c20-644fc83cbb03" containerName="main"
Jul 28 10:31:58.688087 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:31:58.688071    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-mxjkf"
Jul 28 10:31:58.690339 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:31:58.690319    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"kube-root-ca.crt\""
Jul 28 10:31:58.690462 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:31:58.690363    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"auth-disabled-test-kserve-self-signed-certs\""
Jul 28 10:31:58.690528 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:31:58.690492    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"default-dockercfg-rmxfc\""
Jul 28 10:31:58.690617 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:31:58.690532    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"auth-disabled-test-epp-sa-dockercfg-cwcg4\""
Jul 28 10:31:58.690617 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:31:58.690575    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"openshift-service-ca.crt\""
Jul 28 10:31:58.697415 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:31:58.697392    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-mxjkf"]
Jul 28 10:31:58.720521 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:31:58.720487    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:31:58.788796 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:31:58.788759    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbdqc\" (UniqueName: \"kubernetes.io/projected/a4cc23d3-d5a0-49b5-a88a-0b50bf6e2e6e-kube-api-access-nbdqc\") pod \"auth-disabled-test-kserve-router-scheduler-7ccdc4c597-mxjkf\" (UID: \"a4cc23d3-d5a0-49b5-a88a-0b50bf6e2e6e\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-mxjkf"
Jul 28 10:31:58.788957 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:31:58.788806    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4cc23d3-d5a0-49b5-a88a-0b50bf6e2e6e-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-7ccdc4c597-mxjkf\" (UID: \"a4cc23d3-d5a0-49b5-a88a-0b50bf6e2e6e\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-mxjkf"
Jul 28 10:31:58.889512 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:31:58.889478    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbdqc\" (UniqueName: \"kubernetes.io/projected/a4cc23d3-d5a0-49b5-a88a-0b50bf6e2e6e-kube-api-access-nbdqc\") pod \"auth-disabled-test-kserve-router-scheduler-7ccdc4c597-mxjkf\" (UID: \"a4cc23d3-d5a0-49b5-a88a-0b50bf6e2e6e\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-mxjkf"
Jul 28 10:31:58.889729 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:31:58.889523    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4cc23d3-d5a0-49b5-a88a-0b50bf6e2e6e-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-7ccdc4c597-mxjkf\" (UID: \"a4cc23d3-d5a0-49b5-a88a-0b50bf6e2e6e\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-mxjkf"
Jul 28 10:31:58.892044 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:31:58.892018    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4cc23d3-d5a0-49b5-a88a-0b50bf6e2e6e-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-7ccdc4c597-mxjkf\" (UID: \"a4cc23d3-d5a0-49b5-a88a-0b50bf6e2e6e\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-mxjkf"
Jul 28 10:31:58.898171 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:31:58.898147    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbdqc\" (UniqueName: \"kubernetes.io/projected/a4cc23d3-d5a0-49b5-a88a-0b50bf6e2e6e-kube-api-access-nbdqc\") pod \"auth-disabled-test-kserve-router-scheduler-7ccdc4c597-mxjkf\" (UID: \"a4cc23d3-d5a0-49b5-a88a-0b50bf6e2e6e\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-mxjkf"
Jul 28 10:31:58.999391 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:31:58.999301    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-mxjkf"
Jul 28 10:31:59.129962 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:31:59.129939    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-mxjkf"]
Jul 28 10:31:59.132719 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:31:59.132684    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4cc23d3_d5a0_49b5_a88a_0b50bf6e2e6e.slice/crio-c2c8e038f9de93c5ecefbe7ae82c88d52bd65e4c2d1198dda0bf11086081bbdc WatchSource:0}: Error finding container c2c8e038f9de93c5ecefbe7ae82c88d52bd65e4c2d1198dda0bf11086081bbdc: Status 404 returned error can't find the container with id c2c8e038f9de93c5ecefbe7ae82c88d52bd65e4c2d1198dda0bf11086081bbdc
Jul 28 10:31:59.912757 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:31:59.912721    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-mxjkf" event={"ID":"a4cc23d3-d5a0-49b5-a88a-0b50bf6e2e6e","Type":"ContainerStarted","Data":"8aa8026894e671c88251948f09245609c1a8b3c70906b0c83c14639d7245be1a"}
Jul 28 10:31:59.913314 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:31:59.912780    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-mxjkf" event={"ID":"a4cc23d3-d5a0-49b5-a88a-0b50bf6e2e6e","Type":"ContainerStarted","Data":"c2c8e038f9de93c5ecefbe7ae82c88d52bd65e4c2d1198dda0bf11086081bbdc"}
Jul 28 10:31:59.913314 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:31:59.912808    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-mxjkf"
Jul 28 10:31:59.934242 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:31:59.934193    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-mxjkf" podStartSLOduration=1.934175712 podStartE2EDuration="1.934175712s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:31:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:31:59.931674107 +0000 UTC m=+1454.195070455" watchObservedRunningTime="2026-07-28 10:31:59.934175712 +0000 UTC m=+1454.197572040"
Jul 28 10:32:20.286478 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:32:20.286437    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:32:20.385843 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:32:20.385799    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:32:22.041022 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:32:22.040981    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:32:30.918935 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:32:30.918903    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-mxjkf"
Jul 28 10:32:30.977135 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:32:30.977102    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:32:39.616917 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:32:39.616874    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:32:41.047330 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:32:41.047272    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:32:56.691502 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:32:56.691468    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:32:56.789739 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:32:56.789685    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:33:16.741096 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:33:16.741003    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:33:16.839058 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:33:16.839020    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:33:22.824752 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:33:22.824714    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:33:28.568390 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:33:28.568350    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:33:28.670500 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:33:28.670457    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:33:28.989522 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:33:28.989484    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:33:29.181139 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:33:29.181106    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfchd9b"]
Jul 28 10:33:29.184191 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:33:29.184170    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfchd9b"
Jul 28 10:33:29.187039 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:33:29.187016    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"kube-root-ca.crt\""
Jul 28 10:33:29.187039 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:33:29.187036    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-epp-sa-dockercfg-k84ps\""
Jul 28 10:33:29.187202 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:33:29.187021    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"default-dockercfg-4sjg9\""
Jul 28 10:33:29.187202 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:33:29.187070    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-kserve-self-signed-certs\""
Jul 28 10:33:29.187202 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:33:29.187016    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"openshift-service-ca.crt\""
Jul 28 10:33:29.196047 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:33:29.195996    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfchd9b"]
Jul 28 10:33:29.209435 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:33:29.209403    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:33:29.274687 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:33:29.274650    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfxl9\" (UniqueName: \"kubernetes.io/projected/83a26e40-4f54-432e-be39-8102a186c5b3-kube-api-access-rfxl9\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfchd9b\" (UID: \"83a26e40-4f54-432e-be39-8102a186c5b3\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfchd9b"
Jul 28 10:33:29.274932 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:33:29.274707    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83a26e40-4f54-432e-be39-8102a186c5b3-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfchd9b\" (UID: \"83a26e40-4f54-432e-be39-8102a186c5b3\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfchd9b"
Jul 28 10:33:29.375655 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:33:29.375620    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfxl9\" (UniqueName: \"kubernetes.io/projected/83a26e40-4f54-432e-be39-8102a186c5b3-kube-api-access-rfxl9\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfchd9b\" (UID: \"83a26e40-4f54-432e-be39-8102a186c5b3\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfchd9b"
Jul 28 10:33:29.375819 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:33:29.375683    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83a26e40-4f54-432e-be39-8102a186c5b3-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfchd9b\" (UID: \"83a26e40-4f54-432e-be39-8102a186c5b3\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfchd9b"
Jul 28 10:33:29.378062 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:33:29.378038    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83a26e40-4f54-432e-be39-8102a186c5b3-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfchd9b\" (UID: \"83a26e40-4f54-432e-be39-8102a186c5b3\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfchd9b"
Jul 28 10:33:29.383029 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:33:29.383001    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfxl9\" (UniqueName: \"kubernetes.io/projected/83a26e40-4f54-432e-be39-8102a186c5b3-kube-api-access-rfxl9\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfchd9b\" (UID: \"83a26e40-4f54-432e-be39-8102a186c5b3\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfchd9b"
Jul 28 10:33:29.495183 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:33:29.495090    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfchd9b"
Jul 28 10:33:29.647825 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:33:29.647795    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfchd9b"]
Jul 28 10:33:29.650127 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:33:29.650102    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83a26e40_4f54_432e_be39_8102a186c5b3.slice/crio-a1ab5d45acc311c11fbf330f898be481603d033892e1759f1956caa1909353a6 WatchSource:0}: Error finding container a1ab5d45acc311c11fbf330f898be481603d033892e1759f1956caa1909353a6: Status 404 returned error can't find the container with id a1ab5d45acc311c11fbf330f898be481603d033892e1759f1956caa1909353a6
Jul 28 10:33:30.341203 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:33:30.341165    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfchd9b" event={"ID":"83a26e40-4f54-432e-be39-8102a186c5b3","Type":"ContainerStarted","Data":"b33fb51b77d44c5e44fe0757b48b11e4ad8c0d2864963b9b417123260f262fb7"}
Jul 28 10:33:30.341428 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:33:30.341247    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfchd9b" event={"ID":"83a26e40-4f54-432e-be39-8102a186c5b3","Type":"ContainerStarted","Data":"a1ab5d45acc311c11fbf330f898be481603d033892e1759f1956caa1909353a6"}
Jul 28 10:33:30.341428 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:33:30.341293    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfchd9b"
Jul 28 10:33:30.362479 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:33:30.362410    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfchd9b" podStartSLOduration=1.362387861 podStartE2EDuration="1.362387861s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:33:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:33:30.359631074 +0000 UTC m=+1544.623027405" watchObservedRunningTime="2026-07-28 10:33:30.362387861 +0000 UTC m=+1544.625784191"
Jul 28 10:33:36.405373 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:33:36.405323    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:34:01.347974 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:34:01.347944    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfchd9b"
Jul 28 10:34:02.732305 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:34:02.732267    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfchd9b"]
Jul 28 10:34:02.741844 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:34:02.741808    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:34:02.840475 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:34:02.840440    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:34:04.252962 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:34:04.252925    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:34:04.253523 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:34:04.253501    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-mxjkf"]
Jul 28 10:34:04.349465 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:34:04.349428    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:34:10.919565 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:34:10.919502    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-mxjkf" podUID="a4cc23d3-d5a0-49b5-a88a-0b50bf6e2e6e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 10:34:11.348950 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:34:11.348910    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfchd9b" podUID="83a26e40-4f54-432e-be39-8102a186c5b3" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 10:34:13.943104 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:34:13.942856    2579 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfchd9b" message="" err="command '/bin/sleep 15' exited with 137: "
Jul 28 10:34:13.943104 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:34:13.942887    2579 kuberuntime_container.go:755] "PreStop hook failed" err="command '/bin/sleep 15' exited with 137: " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfchd9b" podUID="83a26e40-4f54-432e-be39-8102a186c5b3" containerName="main" containerID="cri-o://b33fb51b77d44c5e44fe0757b48b11e4ad8c0d2864963b9b417123260f262fb7"
Jul 28 10:34:13.943104 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:34:13.942944    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfchd9b" podUID="83a26e40-4f54-432e-be39-8102a186c5b3" containerName="main" containerID="cri-o://b33fb51b77d44c5e44fe0757b48b11e4ad8c0d2864963b9b417123260f262fb7" gracePeriod=49
Jul 28 10:34:14.548503 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:34:14.548477    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-2541cd65_scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfchd9b_83a26e40-4f54-432e-be39-8102a186c5b3/main/0.log"
Jul 28 10:34:14.548676 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:34:14.548515    2579 generic.go:356] "Generic (PLEG): container finished" podID="83a26e40-4f54-432e-be39-8102a186c5b3" containerID="b33fb51b77d44c5e44fe0757b48b11e4ad8c0d2864963b9b417123260f262fb7" exitCode=1
Jul 28 10:34:14.548676 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:34:14.548594    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfchd9b" event={"ID":"83a26e40-4f54-432e-be39-8102a186c5b3","Type":"ContainerDied","Data":"b33fb51b77d44c5e44fe0757b48b11e4ad8c0d2864963b9b417123260f262fb7"}
Jul 28 10:34:14.587101 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:34:14.587075    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-2541cd65_scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfchd9b_83a26e40-4f54-432e-be39-8102a186c5b3/main/0.log"
Jul 28 10:34:14.589507 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:34:14.589491    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfchd9b"
Jul 28 10:34:14.751942 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:34:14.751844    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rfxl9\" (UniqueName: \"kubernetes.io/projected/83a26e40-4f54-432e-be39-8102a186c5b3-kube-api-access-rfxl9\") pod \"83a26e40-4f54-432e-be39-8102a186c5b3\" (UID: \"83a26e40-4f54-432e-be39-8102a186c5b3\") "
Jul 28 10:34:14.752125 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:34:14.751958    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83a26e40-4f54-432e-be39-8102a186c5b3-tls-certs\") pod \"83a26e40-4f54-432e-be39-8102a186c5b3\" (UID: \"83a26e40-4f54-432e-be39-8102a186c5b3\") "
Jul 28 10:34:14.754098 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:34:14.754061    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83a26e40-4f54-432e-be39-8102a186c5b3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "83a26e40-4f54-432e-be39-8102a186c5b3" (UID: "83a26e40-4f54-432e-be39-8102a186c5b3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:34:14.754098 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:34:14.754095    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83a26e40-4f54-432e-be39-8102a186c5b3-kube-api-access-rfxl9" (OuterVolumeSpecName: "kube-api-access-rfxl9") pod "83a26e40-4f54-432e-be39-8102a186c5b3" (UID: "83a26e40-4f54-432e-be39-8102a186c5b3"). InnerVolumeSpecName "kube-api-access-rfxl9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:34:14.853319 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:34:14.853282    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83a26e40-4f54-432e-be39-8102a186c5b3-tls-certs\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 10:34:14.853319 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:34:14.853313    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rfxl9\" (UniqueName: \"kubernetes.io/projected/83a26e40-4f54-432e-be39-8102a186c5b3-kube-api-access-rfxl9\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 10:34:15.555007 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:34:15.554979    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-2541cd65_scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfchd9b_83a26e40-4f54-432e-be39-8102a186c5b3/main/0.log"
Jul 28 10:34:15.555409 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:34:15.555068    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfchd9b"
Jul 28 10:34:15.555409 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:34:15.555078    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfchd9b" event={"ID":"83a26e40-4f54-432e-be39-8102a186c5b3","Type":"ContainerDied","Data":"a1ab5d45acc311c11fbf330f898be481603d033892e1759f1956caa1909353a6"}
Jul 28 10:34:15.555409 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:34:15.555135    2579 scope.go:117] "RemoveContainer" containerID="b33fb51b77d44c5e44fe0757b48b11e4ad8c0d2864963b9b417123260f262fb7"
Jul 28 10:34:15.592223 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:34:15.592198    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfchd9b"]
Jul 28 10:34:15.608929 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:34:15.608902    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfchd9b"]
Jul 28 10:34:16.276170 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:34:16.276128    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="83a26e40-4f54-432e-be39-8102a186c5b3" path="/var/lib/kubelet/pods/83a26e40-4f54-432e-be39-8102a186c5b3/volumes"
Jul 28 10:34:19.274195 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:34:19.274130    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-mxjkf" podUID="a4cc23d3-d5a0-49b5-a88a-0b50bf6e2e6e" containerName="main" containerID="cri-o://8aa8026894e671c88251948f09245609c1a8b3c70906b0c83c14639d7245be1a" gracePeriod=45
Jul 28 10:34:19.525322 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:34:19.525251    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-mxjkf"
Jul 28 10:34:19.574558 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:34:19.574502    2579 generic.go:356] "Generic (PLEG): container finished" podID="a4cc23d3-d5a0-49b5-a88a-0b50bf6e2e6e" containerID="8aa8026894e671c88251948f09245609c1a8b3c70906b0c83c14639d7245be1a" exitCode=0
Jul 28 10:34:19.574764 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:34:19.574573    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-mxjkf"
Jul 28 10:34:19.574764 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:34:19.574574    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-mxjkf" event={"ID":"a4cc23d3-d5a0-49b5-a88a-0b50bf6e2e6e","Type":"ContainerDied","Data":"8aa8026894e671c88251948f09245609c1a8b3c70906b0c83c14639d7245be1a"}
Jul 28 10:34:19.574764 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:34:19.574643    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-mxjkf" event={"ID":"a4cc23d3-d5a0-49b5-a88a-0b50bf6e2e6e","Type":"ContainerDied","Data":"c2c8e038f9de93c5ecefbe7ae82c88d52bd65e4c2d1198dda0bf11086081bbdc"}
Jul 28 10:34:19.574764 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:34:19.574674    2579 scope.go:117] "RemoveContainer" containerID="8aa8026894e671c88251948f09245609c1a8b3c70906b0c83c14639d7245be1a"
Jul 28 10:34:19.585608 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:34:19.585584    2579 scope.go:117] "RemoveContainer" containerID="8aa8026894e671c88251948f09245609c1a8b3c70906b0c83c14639d7245be1a"
Jul 28 10:34:19.585925 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:34:19.585903    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8aa8026894e671c88251948f09245609c1a8b3c70906b0c83c14639d7245be1a\": container with ID starting with 8aa8026894e671c88251948f09245609c1a8b3c70906b0c83c14639d7245be1a not found: ID does not exist" containerID="8aa8026894e671c88251948f09245609c1a8b3c70906b0c83c14639d7245be1a"
Jul 28 10:34:19.585992 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:34:19.585940    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8aa8026894e671c88251948f09245609c1a8b3c70906b0c83c14639d7245be1a"} err="failed to get container status \"8aa8026894e671c88251948f09245609c1a8b3c70906b0c83c14639d7245be1a\": rpc error: code = NotFound desc = could not find container \"8aa8026894e671c88251948f09245609c1a8b3c70906b0c83c14639d7245be1a\": container with ID starting with 8aa8026894e671c88251948f09245609c1a8b3c70906b0c83c14639d7245be1a not found: ID does not exist"
Jul 28 10:34:19.693560 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:34:19.693508    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nbdqc\" (UniqueName: \"kubernetes.io/projected/a4cc23d3-d5a0-49b5-a88a-0b50bf6e2e6e-kube-api-access-nbdqc\") pod \"a4cc23d3-d5a0-49b5-a88a-0b50bf6e2e6e\" (UID: \"a4cc23d3-d5a0-49b5-a88a-0b50bf6e2e6e\") "
Jul 28 10:34:19.693756 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:34:19.693586    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4cc23d3-d5a0-49b5-a88a-0b50bf6e2e6e-tls-certs\") pod \"a4cc23d3-d5a0-49b5-a88a-0b50bf6e2e6e\" (UID: \"a4cc23d3-d5a0-49b5-a88a-0b50bf6e2e6e\") "
Jul 28 10:34:19.695796 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:34:19.695766    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4cc23d3-d5a0-49b5-a88a-0b50bf6e2e6e-kube-api-access-nbdqc" (OuterVolumeSpecName: "kube-api-access-nbdqc") pod "a4cc23d3-d5a0-49b5-a88a-0b50bf6e2e6e" (UID: "a4cc23d3-d5a0-49b5-a88a-0b50bf6e2e6e"). InnerVolumeSpecName "kube-api-access-nbdqc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:34:19.695796 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:34:19.695783    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4cc23d3-d5a0-49b5-a88a-0b50bf6e2e6e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a4cc23d3-d5a0-49b5-a88a-0b50bf6e2e6e" (UID: "a4cc23d3-d5a0-49b5-a88a-0b50bf6e2e6e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:34:19.794895 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:34:19.794858    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nbdqc\" (UniqueName: \"kubernetes.io/projected/a4cc23d3-d5a0-49b5-a88a-0b50bf6e2e6e-kube-api-access-nbdqc\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 10:34:19.794895 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:34:19.794889    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4cc23d3-d5a0-49b5-a88a-0b50bf6e2e6e-tls-certs\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 10:34:19.897561 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:34:19.897512    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-mxjkf"]
Jul 28 10:34:19.899321 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:34:19.899300    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-7ccdc4c597-mxjkf"]
Jul 28 10:34:20.275965 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:34:20.275884    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a4cc23d3-d5a0-49b5-a88a-0b50bf6e2e6e" path="/var/lib/kubelet/pods/a4cc23d3-d5a0-49b5-a88a-0b50bf6e2e6e/volumes"
Jul 28 10:34:22.761838 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:34:22.761799    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:34:22.859316 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:34:22.859283    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:34:23.872099 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:34:23.872051    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:34:23.937598 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:34:23.937558    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:34:24.025191 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:34:24.025152    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:34:24.118243 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:34:24.118206    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:34:29.892245 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:34:29.892207    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:34:30.212592 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:34:30.212494    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:34:31.722381 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:34:31.722341    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:34:31.915446 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:34:31.915412    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795j5kxj"]
Jul 28 10:34:31.915762 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:34:31.915750    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83a26e40-4f54-432e-be39-8102a186c5b3" containerName="main"
Jul 28 10:34:31.915809 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:34:31.915764    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="83a26e40-4f54-432e-be39-8102a186c5b3" containerName="main"
Jul 28 10:34:31.915809 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:34:31.915774    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4cc23d3-d5a0-49b5-a88a-0b50bf6e2e6e" containerName="main"
Jul 28 10:34:31.915809 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:34:31.915780    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4cc23d3-d5a0-49b5-a88a-0b50bf6e2e6e" containerName="main"
Jul 28 10:34:31.915897 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:34:31.915827    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4cc23d3-d5a0-49b5-a88a-0b50bf6e2e6e" containerName="main"
Jul 28 10:34:31.915897 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:34:31.915837    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="83a26e40-4f54-432e-be39-8102a186c5b3" containerName="main"
Jul 28 10:34:31.920061 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:34:31.920038    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795j5kxj"
Jul 28 10:34:31.922556 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:34:31.922521    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"kube-root-ca.crt\""
Jul 28 10:34:31.923038 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:34:31.923017    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"scheduler-custom-template-test-epp-sa-dockercfg-bmcht\""
Jul 28 10:34:31.923116 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:34:31.923101    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"default-dockercfg-c6wgl\""
Jul 28 10:34:31.923171 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:34:31.923021    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"scheduler-custom-template-test-kserve-self-signed-certs\""
Jul 28 10:34:31.923228 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:34:31.923184    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"openshift-service-ca.crt\""
Jul 28 10:34:31.928011 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:34:31.927992    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795j5kxj"]
Jul 28 10:34:32.082533 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:34:32.082500    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:34:32.095899 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:34:32.095874    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2811f8ff-bf83-482f-8800-a97f86a1fd05-tls-certs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-795j5kxj\" (UID: \"2811f8ff-bf83-482f-8800-a97f86a1fd05\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795j5kxj"
Jul 28 10:34:32.096024 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:34:32.095906    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htlg9\" (UniqueName: \"kubernetes.io/projected/2811f8ff-bf83-482f-8800-a97f86a1fd05-kube-api-access-htlg9\") pod \"scheduler-custom-template-test-kserve-router-scheduler-795j5kxj\" (UID: \"2811f8ff-bf83-482f-8800-a97f86a1fd05\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795j5kxj"
Jul 28 10:34:32.096024 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:34:32.095944    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/2811f8ff-bf83-482f-8800-a97f86a1fd05-tokenizer-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-795j5kxj\" (UID: \"2811f8ff-bf83-482f-8800-a97f86a1fd05\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795j5kxj"
Jul 28 10:34:32.096024 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:34:32.095999    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/2811f8ff-bf83-482f-8800-a97f86a1fd05-cachi2-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-795j5kxj\" (UID: \"2811f8ff-bf83-482f-8800-a97f86a1fd05\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795j5kxj"
Jul 28 10:34:32.197488 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:34:32.197450    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2811f8ff-bf83-482f-8800-a97f86a1fd05-tls-certs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-795j5kxj\" (UID: \"2811f8ff-bf83-482f-8800-a97f86a1fd05\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795j5kxj"
Jul 28 10:34:32.197714 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:34:32.197503    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htlg9\" (UniqueName: \"kubernetes.io/projected/2811f8ff-bf83-482f-8800-a97f86a1fd05-kube-api-access-htlg9\") pod \"scheduler-custom-template-test-kserve-router-scheduler-795j5kxj\" (UID: \"2811f8ff-bf83-482f-8800-a97f86a1fd05\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795j5kxj"
Jul 28 10:34:32.197714 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:34:32.197589    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/2811f8ff-bf83-482f-8800-a97f86a1fd05-tokenizer-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-795j5kxj\" (UID: \"2811f8ff-bf83-482f-8800-a97f86a1fd05\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795j5kxj"
Jul 28 10:34:32.197714 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:34:32.197620    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/2811f8ff-bf83-482f-8800-a97f86a1fd05-cachi2-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-795j5kxj\" (UID: \"2811f8ff-bf83-482f-8800-a97f86a1fd05\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795j5kxj"
Jul 28 10:34:32.198075 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:34:32.198049    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/2811f8ff-bf83-482f-8800-a97f86a1fd05-cachi2-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-795j5kxj\" (UID: \"2811f8ff-bf83-482f-8800-a97f86a1fd05\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795j5kxj"
Jul 28 10:34:32.198152 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:34:32.198059    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/2811f8ff-bf83-482f-8800-a97f86a1fd05-tokenizer-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-795j5kxj\" (UID: \"2811f8ff-bf83-482f-8800-a97f86a1fd05\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795j5kxj"
Jul 28 10:34:32.200099 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:34:32.200077    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2811f8ff-bf83-482f-8800-a97f86a1fd05-tls-certs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-795j5kxj\" (UID: \"2811f8ff-bf83-482f-8800-a97f86a1fd05\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795j5kxj"
Jul 28 10:34:32.205805 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:34:32.205780    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-htlg9\" (UniqueName: \"kubernetes.io/projected/2811f8ff-bf83-482f-8800-a97f86a1fd05-kube-api-access-htlg9\") pod \"scheduler-custom-template-test-kserve-router-scheduler-795j5kxj\" (UID: \"2811f8ff-bf83-482f-8800-a97f86a1fd05\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795j5kxj"
Jul 28 10:34:32.231207 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:34:32.231179    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795j5kxj"
Jul 28 10:34:32.366258 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:34:32.366216    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795j5kxj"]
Jul 28 10:34:32.368443 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:34:32.368415    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2811f8ff_bf83_482f_8800_a97f86a1fd05.slice/crio-3da6ab4c91f2a51361bdfc523ee2643482f0588bb9a26a0a6c56b1e1beae3c8f WatchSource:0}: Error finding container 3da6ab4c91f2a51361bdfc523ee2643482f0588bb9a26a0a6c56b1e1beae3c8f: Status 404 returned error can't find the container with id 3da6ab4c91f2a51361bdfc523ee2643482f0588bb9a26a0a6c56b1e1beae3c8f
Jul 28 10:34:32.648995 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:34:32.648892    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795j5kxj" event={"ID":"2811f8ff-bf83-482f-8800-a97f86a1fd05","Type":"ContainerStarted","Data":"7dc3e01b097a7c0b5b8a2eb5b58756c6819e317d200aa11edd1ac2c063055ab4"}
Jul 28 10:34:32.648995 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:34:32.648966    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795j5kxj" event={"ID":"2811f8ff-bf83-482f-8800-a97f86a1fd05","Type":"ContainerStarted","Data":"3da6ab4c91f2a51361bdfc523ee2643482f0588bb9a26a0a6c56b1e1beae3c8f"}
Jul 28 10:34:32.648995 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:34:32.648988    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795j5kxj"
Jul 28 10:34:32.669786 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:34:32.669730    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795j5kxj" podStartSLOduration=1.669714519 podStartE2EDuration="1.669714519s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:34:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:34:32.667814424 +0000 UTC m=+1606.931210756" watchObservedRunningTime="2026-07-28 10:34:32.669714519 +0000 UTC m=+1606.933110848"
Jul 28 10:34:35.779580 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:34:35.779518    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:34:35.878766 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:34:35.878726    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:34:36.361044 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:34:36.360998    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:34:36.464233 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:34:36.464194    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:34:38.279455 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:34:38.279369    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:34:38.442290 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:34:38.442257    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-986gr"]
Jul 28 10:34:38.445567 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:34:38.445525    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-986gr"
Jul 28 10:34:38.448344 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:34:38.448310    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4\"/\"kube-root-ca.crt\""
Jul 28 10:34:38.448487 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:34:38.448358    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4\"/\"autoscale-hpa-deploy-kserve-self-signed-certs\""
Jul 28 10:34:38.448487 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:34:38.448315    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4\"/\"openshift-service-ca.crt\""
Jul 28 10:34:38.448487 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:34:38.448315    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4\"/\"default-dockercfg-9wxqq\""
Jul 28 10:34:38.448487 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:34:38.448395    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4\"/\"autoscale-hpa-deploy-epp-sa-dockercfg-tj654\""
Jul 28 10:34:38.454838 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:34:38.454817    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-986gr"]
Jul 28 10:34:38.473703 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:34:38.473670    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:34:38.542126 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:34:38.542088    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dee8a861-2e0c-482b-822c-fbbea81567c0-tls-certs\") pod \"autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-986gr\" (UID: \"dee8a861-2e0c-482b-822c-fbbea81567c0\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-986gr"
Jul 28 10:34:38.542354 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:34:38.542143    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scf2w\" (UniqueName: \"kubernetes.io/projected/dee8a861-2e0c-482b-822c-fbbea81567c0-kube-api-access-scf2w\") pod \"autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-986gr\" (UID: \"dee8a861-2e0c-482b-822c-fbbea81567c0\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-986gr"
Jul 28 10:34:38.643694 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:34:38.643655    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dee8a861-2e0c-482b-822c-fbbea81567c0-tls-certs\") pod \"autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-986gr\" (UID: \"dee8a861-2e0c-482b-822c-fbbea81567c0\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-986gr"
Jul 28 10:34:38.643893 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:34:38.643700    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scf2w\" (UniqueName: \"kubernetes.io/projected/dee8a861-2e0c-482b-822c-fbbea81567c0-kube-api-access-scf2w\") pod \"autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-986gr\" (UID: \"dee8a861-2e0c-482b-822c-fbbea81567c0\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-986gr"
Jul 28 10:34:38.646231 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:34:38.646209    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dee8a861-2e0c-482b-822c-fbbea81567c0-tls-certs\") pod \"autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-986gr\" (UID: \"dee8a861-2e0c-482b-822c-fbbea81567c0\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-986gr"
Jul 28 10:34:38.651216 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:34:38.651191    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-scf2w\" (UniqueName: \"kubernetes.io/projected/dee8a861-2e0c-482b-822c-fbbea81567c0-kube-api-access-scf2w\") pod \"autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-986gr\" (UID: \"dee8a861-2e0c-482b-822c-fbbea81567c0\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-986gr"
Jul 28 10:34:38.756788 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:34:38.756742    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-986gr"
Jul 28 10:34:38.895302 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:34:38.895101    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-986gr"]
Jul 28 10:34:38.898036 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:34:38.898008    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddee8a861_2e0c_482b_822c_fbbea81567c0.slice/crio-bfb7d59f12370f428667595270df60098246aeaa64d96c488d9baea094b7b6d4 WatchSource:0}: Error finding container bfb7d59f12370f428667595270df60098246aeaa64d96c488d9baea094b7b6d4: Status 404 returned error can't find the container with id bfb7d59f12370f428667595270df60098246aeaa64d96c488d9baea094b7b6d4
Jul 28 10:34:39.691163 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:34:39.691122    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-986gr" event={"ID":"dee8a861-2e0c-482b-822c-fbbea81567c0","Type":"ContainerStarted","Data":"b084f0289ceef1c761395f7e18b1e76ef72b7c400c6f51f83a51c2bb5f224741"}
Jul 28 10:34:39.691632 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:34:39.691183    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-986gr" event={"ID":"dee8a861-2e0c-482b-822c-fbbea81567c0","Type":"ContainerStarted","Data":"bfb7d59f12370f428667595270df60098246aeaa64d96c488d9baea094b7b6d4"}
Jul 28 10:34:39.691632 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:34:39.691302    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-986gr"
Jul 28 10:34:39.710060 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:34:39.709989    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-986gr" podStartSLOduration=1.709976301 podStartE2EDuration="1.709976301s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:34:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:34:39.707500751 +0000 UTC m=+1613.970897080" watchObservedRunningTime="2026-07-28 10:34:39.709976301 +0000 UTC m=+1613.973372630"
Jul 28 10:34:43.532101 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:34:43.532050    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:34:54.791432 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:34:54.791387    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:35:03.656527 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:35:03.656495    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795j5kxj"
Jul 28 10:35:04.799431 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:35:04.799387    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795j5kxj"]
Jul 28 10:35:04.806783 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:35:04.806752    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:35:04.907262 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:35:04.907227    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:35:10.698395 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:35:10.698362    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-986gr"
Jul 28 10:35:13.656973 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:35:13.656930    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795j5kxj" podUID="2811f8ff-bf83-482f-8800-a97f86a1fd05" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 10:35:19.820901 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:35:19.820839    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795j5kxj" podUID="2811f8ff-bf83-482f-8800-a97f86a1fd05" containerName="main" containerID="cri-o://7dc3e01b097a7c0b5b8a2eb5b58756c6819e317d200aa11edd1ac2c063055ab4" gracePeriod=45
Jul 28 10:35:20.063756 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:35:20.063727    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795j5kxj"
Jul 28 10:35:20.199472 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:35:20.199377    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2811f8ff-bf83-482f-8800-a97f86a1fd05-tls-certs\") pod \"2811f8ff-bf83-482f-8800-a97f86a1fd05\" (UID: \"2811f8ff-bf83-482f-8800-a97f86a1fd05\") "
Jul 28 10:35:20.199472 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:35:20.199435    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-htlg9\" (UniqueName: \"kubernetes.io/projected/2811f8ff-bf83-482f-8800-a97f86a1fd05-kube-api-access-htlg9\") pod \"2811f8ff-bf83-482f-8800-a97f86a1fd05\" (UID: \"2811f8ff-bf83-482f-8800-a97f86a1fd05\") "
Jul 28 10:35:20.199472 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:35:20.199456    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/2811f8ff-bf83-482f-8800-a97f86a1fd05-cachi2-cache\") pod \"2811f8ff-bf83-482f-8800-a97f86a1fd05\" (UID: \"2811f8ff-bf83-482f-8800-a97f86a1fd05\") "
Jul 28 10:35:20.199472 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:35:20.199474    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/2811f8ff-bf83-482f-8800-a97f86a1fd05-tokenizer-cache\") pod \"2811f8ff-bf83-482f-8800-a97f86a1fd05\" (UID: \"2811f8ff-bf83-482f-8800-a97f86a1fd05\") "
Jul 28 10:35:20.199849 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:35:20.199729    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2811f8ff-bf83-482f-8800-a97f86a1fd05-cachi2-cache" (OuterVolumeSpecName: "cachi2-cache") pod "2811f8ff-bf83-482f-8800-a97f86a1fd05" (UID: "2811f8ff-bf83-482f-8800-a97f86a1fd05"). InnerVolumeSpecName "cachi2-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:35:20.199849 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:35:20.199753    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2811f8ff-bf83-482f-8800-a97f86a1fd05-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "2811f8ff-bf83-482f-8800-a97f86a1fd05" (UID: "2811f8ff-bf83-482f-8800-a97f86a1fd05"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:35:20.201722 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:35:20.201698    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2811f8ff-bf83-482f-8800-a97f86a1fd05-kube-api-access-htlg9" (OuterVolumeSpecName: "kube-api-access-htlg9") pod "2811f8ff-bf83-482f-8800-a97f86a1fd05" (UID: "2811f8ff-bf83-482f-8800-a97f86a1fd05"). InnerVolumeSpecName "kube-api-access-htlg9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:35:20.201824 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:35:20.201726    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2811f8ff-bf83-482f-8800-a97f86a1fd05-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2811f8ff-bf83-482f-8800-a97f86a1fd05" (UID: "2811f8ff-bf83-482f-8800-a97f86a1fd05"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:35:20.300302 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:35:20.300272    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2811f8ff-bf83-482f-8800-a97f86a1fd05-tls-certs\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 10:35:20.300302 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:35:20.300297    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-htlg9\" (UniqueName: \"kubernetes.io/projected/2811f8ff-bf83-482f-8800-a97f86a1fd05-kube-api-access-htlg9\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 10:35:20.300302 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:35:20.300307    2579 reconciler_common.go:299] "Volume detached for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/2811f8ff-bf83-482f-8800-a97f86a1fd05-cachi2-cache\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 10:35:20.300515 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:35:20.300316    2579 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/2811f8ff-bf83-482f-8800-a97f86a1fd05-tokenizer-cache\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 10:35:20.899266 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:35:20.899232    2579 generic.go:356] "Generic (PLEG): container finished" podID="2811f8ff-bf83-482f-8800-a97f86a1fd05" containerID="7dc3e01b097a7c0b5b8a2eb5b58756c6819e317d200aa11edd1ac2c063055ab4" exitCode=0
Jul 28 10:35:20.899785 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:35:20.899280    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795j5kxj"
Jul 28 10:35:20.899785 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:35:20.899306    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795j5kxj" event={"ID":"2811f8ff-bf83-482f-8800-a97f86a1fd05","Type":"ContainerDied","Data":"7dc3e01b097a7c0b5b8a2eb5b58756c6819e317d200aa11edd1ac2c063055ab4"}
Jul 28 10:35:20.899785 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:35:20.899349    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795j5kxj" event={"ID":"2811f8ff-bf83-482f-8800-a97f86a1fd05","Type":"ContainerDied","Data":"3da6ab4c91f2a51361bdfc523ee2643482f0588bb9a26a0a6c56b1e1beae3c8f"}
Jul 28 10:35:20.899785 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:35:20.899370    2579 scope.go:117] "RemoveContainer" containerID="7dc3e01b097a7c0b5b8a2eb5b58756c6819e317d200aa11edd1ac2c063055ab4"
Jul 28 10:35:20.908164 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:35:20.908145    2579 scope.go:117] "RemoveContainer" containerID="7dc3e01b097a7c0b5b8a2eb5b58756c6819e317d200aa11edd1ac2c063055ab4"
Jul 28 10:35:20.908458 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:35:20.908436    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7dc3e01b097a7c0b5b8a2eb5b58756c6819e317d200aa11edd1ac2c063055ab4\": container with ID starting with 7dc3e01b097a7c0b5b8a2eb5b58756c6819e317d200aa11edd1ac2c063055ab4 not found: ID does not exist" containerID="7dc3e01b097a7c0b5b8a2eb5b58756c6819e317d200aa11edd1ac2c063055ab4"
Jul 28 10:35:20.908504 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:35:20.908467    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7dc3e01b097a7c0b5b8a2eb5b58756c6819e317d200aa11edd1ac2c063055ab4"} err="failed to get container status \"7dc3e01b097a7c0b5b8a2eb5b58756c6819e317d200aa11edd1ac2c063055ab4\": rpc error: code = NotFound desc = could not find container \"7dc3e01b097a7c0b5b8a2eb5b58756c6819e317d200aa11edd1ac2c063055ab4\": container with ID starting with 7dc3e01b097a7c0b5b8a2eb5b58756c6819e317d200aa11edd1ac2c063055ab4 not found: ID does not exist"
Jul 28 10:35:20.916328 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:35:20.916304    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795j5kxj"]
Jul 28 10:35:20.926140 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:35:20.926116    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795j5kxj"]
Jul 28 10:35:22.275885 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:35:22.275851    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2811f8ff-bf83-482f-8800-a97f86a1fd05" path="/var/lib/kubelet/pods/2811f8ff-bf83-482f-8800-a97f86a1fd05/volumes"
Jul 28 10:35:24.748673 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:35:24.748632    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:35:24.841288 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:35:24.841252    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:35:30.961802 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:35:30.961758    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:35:36.529748 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:35:36.529707    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:35:36.632235 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:35:36.632191    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:35:38.028341 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:35:38.028304    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-rk62r"]
Jul 28 10:35:38.028841 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:35:38.028691    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2811f8ff-bf83-482f-8800-a97f86a1fd05" containerName="main"
Jul 28 10:35:38.028841 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:35:38.028706    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="2811f8ff-bf83-482f-8800-a97f86a1fd05" containerName="main"
Jul 28 10:35:38.028841 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:35:38.028762    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="2811f8ff-bf83-482f-8800-a97f86a1fd05" containerName="main"
Jul 28 10:35:38.033320 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:35:38.033291    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-rk62r"
Jul 28 10:35:38.035473 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:35:38.035451    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"openshift-service-ca.crt\""
Jul 28 10:35:38.035641 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:35:38.035453    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"kube-root-ca.crt\""
Jul 28 10:35:38.036078 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:35:38.036058    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"default-dockercfg-pgj7g\""
Jul 28 10:35:38.036165 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:35:38.036101    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-kserve-self-signed-certs\""
Jul 28 10:35:38.040967 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:35:38.040413    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-rk62r"]
Jul 28 10:35:38.064867 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:35:38.064821    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:35:38.129308 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:35:38.129266    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a46c85da-b2e9-4bd9-8d5a-c35fd62755ef-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-rk62r\" (UID: \"a46c85da-b2e9-4bd9-8d5a-c35fd62755ef\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-rk62r"
Jul 28 10:35:38.129473 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:35:38.129364    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgpcj\" (UniqueName: \"kubernetes.io/projected/a46c85da-b2e9-4bd9-8d5a-c35fd62755ef-kube-api-access-pgpcj\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-rk62r\" (UID: \"a46c85da-b2e9-4bd9-8d5a-c35fd62755ef\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-rk62r"
Jul 28 10:35:38.129473 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:35:38.129396    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a46c85da-b2e9-4bd9-8d5a-c35fd62755ef-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-rk62r\" (UID: \"a46c85da-b2e9-4bd9-8d5a-c35fd62755ef\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-rk62r"
Jul 28 10:35:38.129473 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:35:38.129418    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a46c85da-b2e9-4bd9-8d5a-c35fd62755ef-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-rk62r\" (UID: \"a46c85da-b2e9-4bd9-8d5a-c35fd62755ef\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-rk62r"
Jul 28 10:35:38.129473 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:35:38.129439    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a46c85da-b2e9-4bd9-8d5a-c35fd62755ef-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-rk62r\" (UID: \"a46c85da-b2e9-4bd9-8d5a-c35fd62755ef\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-rk62r"
Jul 28 10:35:38.129473 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:35:38.129470    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a46c85da-b2e9-4bd9-8d5a-c35fd62755ef-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-rk62r\" (UID: \"a46c85da-b2e9-4bd9-8d5a-c35fd62755ef\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-rk62r"
Jul 28 10:35:38.230336 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:35:38.230302    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a46c85da-b2e9-4bd9-8d5a-c35fd62755ef-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-rk62r\" (UID: \"a46c85da-b2e9-4bd9-8d5a-c35fd62755ef\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-rk62r"
Jul 28 10:35:38.230523 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:35:38.230361    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a46c85da-b2e9-4bd9-8d5a-c35fd62755ef-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-rk62r\" (UID: \"a46c85da-b2e9-4bd9-8d5a-c35fd62755ef\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-rk62r"
Jul 28 10:35:38.230523 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:35:38.230397    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a46c85da-b2e9-4bd9-8d5a-c35fd62755ef-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-rk62r\" (UID: \"a46c85da-b2e9-4bd9-8d5a-c35fd62755ef\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-rk62r"
Jul 28 10:35:38.230523 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:35:38.230482    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgpcj\" (UniqueName: \"kubernetes.io/projected/a46c85da-b2e9-4bd9-8d5a-c35fd62755ef-kube-api-access-pgpcj\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-rk62r\" (UID: \"a46c85da-b2e9-4bd9-8d5a-c35fd62755ef\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-rk62r"
Jul 28 10:35:38.230523 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:35:38.230515    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a46c85da-b2e9-4bd9-8d5a-c35fd62755ef-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-rk62r\" (UID: \"a46c85da-b2e9-4bd9-8d5a-c35fd62755ef\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-rk62r"
Jul 28 10:35:38.230793 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:35:38.230565    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a46c85da-b2e9-4bd9-8d5a-c35fd62755ef-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-rk62r\" (UID: \"a46c85da-b2e9-4bd9-8d5a-c35fd62755ef\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-rk62r"
Jul 28 10:35:38.230847 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:35:38.230785    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a46c85da-b2e9-4bd9-8d5a-c35fd62755ef-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-rk62r\" (UID: \"a46c85da-b2e9-4bd9-8d5a-c35fd62755ef\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-rk62r"
Jul 28 10:35:38.230847 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:35:38.230802    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a46c85da-b2e9-4bd9-8d5a-c35fd62755ef-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-rk62r\" (UID: \"a46c85da-b2e9-4bd9-8d5a-c35fd62755ef\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-rk62r"
Jul 28 10:35:38.231059 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:35:38.231035    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a46c85da-b2e9-4bd9-8d5a-c35fd62755ef-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-rk62r\" (UID: \"a46c85da-b2e9-4bd9-8d5a-c35fd62755ef\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-rk62r"
Jul 28 10:35:38.232833 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:35:38.232802    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a46c85da-b2e9-4bd9-8d5a-c35fd62755ef-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-rk62r\" (UID: \"a46c85da-b2e9-4bd9-8d5a-c35fd62755ef\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-rk62r"
Jul 28 10:35:38.233121 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:35:38.233100    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a46c85da-b2e9-4bd9-8d5a-c35fd62755ef-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-rk62r\" (UID: \"a46c85da-b2e9-4bd9-8d5a-c35fd62755ef\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-rk62r"
Jul 28 10:35:38.242270 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:35:38.242245    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgpcj\" (UniqueName: \"kubernetes.io/projected/a46c85da-b2e9-4bd9-8d5a-c35fd62755ef-kube-api-access-pgpcj\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-rk62r\" (UID: \"a46c85da-b2e9-4bd9-8d5a-c35fd62755ef\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-rk62r"
Jul 28 10:35:38.278576 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:35:38.278459    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:35:38.345454 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:35:38.345423    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-rk62r"
Jul 28 10:35:38.483443 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:35:38.483418    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-rk62r"]
Jul 28 10:35:38.485392 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:35:38.485363    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda46c85da_b2e9_4bd9_8d5a_c35fd62755ef.slice/crio-46063c8f8fa55aa131ce168b3a347170b96950638bb191ed48ec913f342d51f4 WatchSource:0}: Error finding container 46063c8f8fa55aa131ce168b3a347170b96950638bb191ed48ec913f342d51f4: Status 404 returned error can't find the container with id 46063c8f8fa55aa131ce168b3a347170b96950638bb191ed48ec913f342d51f4
Jul 28 10:35:38.487194 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:35:38.487178    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 10:35:38.988058 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:35:38.988019    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-rk62r" event={"ID":"a46c85da-b2e9-4bd9-8d5a-c35fd62755ef","Type":"ContainerStarted","Data":"46063c8f8fa55aa131ce168b3a347170b96950638bb191ed48ec913f342d51f4"}
Jul 28 10:35:41.000598 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:35:41.000557    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-rk62r" event={"ID":"a46c85da-b2e9-4bd9-8d5a-c35fd62755ef","Type":"ContainerStarted","Data":"66e4527ef7a710ca0f831f6aeb8f3a3f02d9ad78f4d0079f5292dc78f0e6bae3"}
Jul 28 10:35:41.026028 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:35:41.025966    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-rk62r" podStartSLOduration=1.376768692 podStartE2EDuration="3.025950468s" totalImagesPullingTime="1.649181776s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:35:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:35:41.023256355 +0000 UTC m=+1675.286652685" watchObservedRunningTime="2026-07-28 10:35:41.025950468 +0000 UTC m=+1675.289346828"
Jul 28 10:35:41.345736 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:35:41.345685    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-rk62r"
Jul 28 10:35:48.346433 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:35:48.346396    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-rk62r"
Jul 28 10:35:48.359094 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:35:48.359068    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-rk62r"
Jul 28 10:35:49.049183 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:35:49.049151    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-rk62r"
Jul 28 10:35:53.373734 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:35:53.373692    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:36:04.546416 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:36:04.546375    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:36:39.378203 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:36:39.378167    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:37:12.336157 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:37:12.336117    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:37:12.359833 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:37:12.359799    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-rk62r"]
Jul 28 10:37:12.369100 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:37:12.369058    2579 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 28 10:37:12.369100 ip-10-0-141-50 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 10:37:12.369100 ip-10-0-141-50 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Jul 28 10:37:12.369100 ip-10-0-141-50 kubenswrapper[2579]:  > containerID="66e4527ef7a710ca0f831f6aeb8f3a3f02d9ad78f4d0079f5292dc78f0e6bae3" cmd=["/bin/sleep","15"]
Jul 28 10:37:12.369452 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:37:12.369117    2579 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-rk62r" message="" err=<
Jul 28 10:37:12.369452 ip-10-0-141-50 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 10:37:12.369452 ip-10-0-141-50 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Jul 28 10:37:12.369452 ip-10-0-141-50 kubenswrapper[2579]:  >
Jul 28 10:37:12.369452 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:37:12.369133    2579 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 28 10:37:12.369452 ip-10-0-141-50 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 10:37:12.369452 ip-10-0-141-50 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Jul 28 10:37:12.369452 ip-10-0-141-50 kubenswrapper[2579]:  > pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-rk62r" podUID="a46c85da-b2e9-4bd9-8d5a-c35fd62755ef" containerName="main" containerID="cri-o://66e4527ef7a710ca0f831f6aeb8f3a3f02d9ad78f4d0079f5292dc78f0e6bae3"
Jul 28 10:37:12.369452 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:37:12.369175    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-rk62r" podUID="a46c85da-b2e9-4bd9-8d5a-c35fd62755ef" containerName="main" containerID="cri-o://66e4527ef7a710ca0f831f6aeb8f3a3f02d9ad78f4d0079f5292dc78f0e6bae3" gracePeriod=60
Jul 28 10:37:12.480068 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:37:12.480030    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:37:12.722902 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:37:12.722873    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-rk62r"
Jul 28 10:37:12.816087 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:37:12.816058    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a46c85da-b2e9-4bd9-8d5a-c35fd62755ef-dshm\") pod \"a46c85da-b2e9-4bd9-8d5a-c35fd62755ef\" (UID: \"a46c85da-b2e9-4bd9-8d5a-c35fd62755ef\") "
Jul 28 10:37:12.816260 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:37:12.816125    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a46c85da-b2e9-4bd9-8d5a-c35fd62755ef-tmp-dir\") pod \"a46c85da-b2e9-4bd9-8d5a-c35fd62755ef\" (UID: \"a46c85da-b2e9-4bd9-8d5a-c35fd62755ef\") "
Jul 28 10:37:12.816260 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:37:12.816150    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a46c85da-b2e9-4bd9-8d5a-c35fd62755ef-tls-certs\") pod \"a46c85da-b2e9-4bd9-8d5a-c35fd62755ef\" (UID: \"a46c85da-b2e9-4bd9-8d5a-c35fd62755ef\") "
Jul 28 10:37:12.816260 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:37:12.816176    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pgpcj\" (UniqueName: \"kubernetes.io/projected/a46c85da-b2e9-4bd9-8d5a-c35fd62755ef-kube-api-access-pgpcj\") pod \"a46c85da-b2e9-4bd9-8d5a-c35fd62755ef\" (UID: \"a46c85da-b2e9-4bd9-8d5a-c35fd62755ef\") "
Jul 28 10:37:12.816260 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:37:12.816208    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a46c85da-b2e9-4bd9-8d5a-c35fd62755ef-home\") pod \"a46c85da-b2e9-4bd9-8d5a-c35fd62755ef\" (UID: \"a46c85da-b2e9-4bd9-8d5a-c35fd62755ef\") "
Jul 28 10:37:12.816260 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:37:12.816236    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a46c85da-b2e9-4bd9-8d5a-c35fd62755ef-model-cache\") pod \"a46c85da-b2e9-4bd9-8d5a-c35fd62755ef\" (UID: \"a46c85da-b2e9-4bd9-8d5a-c35fd62755ef\") "
Jul 28 10:37:12.816529 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:37:12.816425    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a46c85da-b2e9-4bd9-8d5a-c35fd62755ef-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a46c85da-b2e9-4bd9-8d5a-c35fd62755ef" (UID: "a46c85da-b2e9-4bd9-8d5a-c35fd62755ef"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:37:12.816529 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:37:12.816492    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a46c85da-b2e9-4bd9-8d5a-c35fd62755ef-home" (OuterVolumeSpecName: "home") pod "a46c85da-b2e9-4bd9-8d5a-c35fd62755ef" (UID: "a46c85da-b2e9-4bd9-8d5a-c35fd62755ef"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:37:12.816683 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:37:12.816560    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a46c85da-b2e9-4bd9-8d5a-c35fd62755ef-model-cache" (OuterVolumeSpecName: "model-cache") pod "a46c85da-b2e9-4bd9-8d5a-c35fd62755ef" (UID: "a46c85da-b2e9-4bd9-8d5a-c35fd62755ef"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:37:12.818367 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:37:12.818343    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a46c85da-b2e9-4bd9-8d5a-c35fd62755ef-dshm" (OuterVolumeSpecName: "dshm") pod "a46c85da-b2e9-4bd9-8d5a-c35fd62755ef" (UID: "a46c85da-b2e9-4bd9-8d5a-c35fd62755ef"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:37:12.818725 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:37:12.818702    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a46c85da-b2e9-4bd9-8d5a-c35fd62755ef-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a46c85da-b2e9-4bd9-8d5a-c35fd62755ef" (UID: "a46c85da-b2e9-4bd9-8d5a-c35fd62755ef"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:37:12.818830 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:37:12.818727    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a46c85da-b2e9-4bd9-8d5a-c35fd62755ef-kube-api-access-pgpcj" (OuterVolumeSpecName: "kube-api-access-pgpcj") pod "a46c85da-b2e9-4bd9-8d5a-c35fd62755ef" (UID: "a46c85da-b2e9-4bd9-8d5a-c35fd62755ef"). InnerVolumeSpecName "kube-api-access-pgpcj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:37:12.917106 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:37:12.917010    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a46c85da-b2e9-4bd9-8d5a-c35fd62755ef-model-cache\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 10:37:12.917106 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:37:12.917042    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a46c85da-b2e9-4bd9-8d5a-c35fd62755ef-dshm\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 10:37:12.917106 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:37:12.917054    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a46c85da-b2e9-4bd9-8d5a-c35fd62755ef-tmp-dir\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 10:37:12.917106 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:37:12.917061    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a46c85da-b2e9-4bd9-8d5a-c35fd62755ef-tls-certs\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 10:37:12.917106 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:37:12.917070    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pgpcj\" (UniqueName: \"kubernetes.io/projected/a46c85da-b2e9-4bd9-8d5a-c35fd62755ef-kube-api-access-pgpcj\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 10:37:12.917106 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:37:12.917078    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a46c85da-b2e9-4bd9-8d5a-c35fd62755ef-home\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 10:37:13.458868 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:37:13.458826    2579 generic.go:356] "Generic (PLEG): container finished" podID="a46c85da-b2e9-4bd9-8d5a-c35fd62755ef" containerID="66e4527ef7a710ca0f831f6aeb8f3a3f02d9ad78f4d0079f5292dc78f0e6bae3" exitCode=0
Jul 28 10:37:13.459360 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:37:13.458887    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-rk62r"
Jul 28 10:37:13.459360 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:37:13.458879    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-rk62r" event={"ID":"a46c85da-b2e9-4bd9-8d5a-c35fd62755ef","Type":"ContainerDied","Data":"66e4527ef7a710ca0f831f6aeb8f3a3f02d9ad78f4d0079f5292dc78f0e6bae3"}
Jul 28 10:37:13.459360 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:37:13.459028    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-rk62r" event={"ID":"a46c85da-b2e9-4bd9-8d5a-c35fd62755ef","Type":"ContainerDied","Data":"46063c8f8fa55aa131ce168b3a347170b96950638bb191ed48ec913f342d51f4"}
Jul 28 10:37:13.459360 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:37:13.459065    2579 scope.go:117] "RemoveContainer" containerID="66e4527ef7a710ca0f831f6aeb8f3a3f02d9ad78f4d0079f5292dc78f0e6bae3"
Jul 28 10:37:13.468487 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:37:13.468469    2579 scope.go:117] "RemoveContainer" containerID="66e4527ef7a710ca0f831f6aeb8f3a3f02d9ad78f4d0079f5292dc78f0e6bae3"
Jul 28 10:37:13.468805 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:37:13.468787    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66e4527ef7a710ca0f831f6aeb8f3a3f02d9ad78f4d0079f5292dc78f0e6bae3\": container with ID starting with 66e4527ef7a710ca0f831f6aeb8f3a3f02d9ad78f4d0079f5292dc78f0e6bae3 not found: ID does not exist" containerID="66e4527ef7a710ca0f831f6aeb8f3a3f02d9ad78f4d0079f5292dc78f0e6bae3"
Jul 28 10:37:13.468870 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:37:13.468815    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66e4527ef7a710ca0f831f6aeb8f3a3f02d9ad78f4d0079f5292dc78f0e6bae3"} err="failed to get container status \"66e4527ef7a710ca0f831f6aeb8f3a3f02d9ad78f4d0079f5292dc78f0e6bae3\": rpc error: code = NotFound desc = could not find container \"66e4527ef7a710ca0f831f6aeb8f3a3f02d9ad78f4d0079f5292dc78f0e6bae3\": container with ID starting with 66e4527ef7a710ca0f831f6aeb8f3a3f02d9ad78f4d0079f5292dc78f0e6bae3 not found: ID does not exist"
Jul 28 10:37:13.482522 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:37:13.482484    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-rk62r"]
Jul 28 10:37:13.484926 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:37:13.484900    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-rk62r"]
Jul 28 10:37:14.276829 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:37:14.276718    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a46c85da-b2e9-4bd9-8d5a-c35fd62755ef" path="/var/lib/kubelet/pods/a46c85da-b2e9-4bd9-8d5a-c35fd62755ef/volumes"
Jul 28 10:37:32.137240 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:37:32.137202    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:37:32.236509 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:37:32.236475    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:37:38.401128 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:37:38.401032    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:37:44.160996 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:37:44.160909    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:37:44.258314 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:37:44.258279    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:38:14.125175 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:38:14.125143    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fjzjqr"]
Jul 28 10:38:14.125656 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:38:14.125455    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a46c85da-b2e9-4bd9-8d5a-c35fd62755ef" containerName="main"
Jul 28 10:38:14.125656 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:38:14.125466    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a46c85da-b2e9-4bd9-8d5a-c35fd62755ef" containerName="main"
Jul 28 10:38:14.125656 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:38:14.125532    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a46c85da-b2e9-4bd9-8d5a-c35fd62755ef" containerName="main"
Jul 28 10:38:14.128486 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:38:14.128466    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fjzjqr"
Jul 28 10:38:14.131431 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:38:14.131408    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-kserve-self-signed-certs\""
Jul 28 10:38:14.131570 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:38:14.131461    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"openshift-service-ca.crt\""
Jul 28 10:38:14.131570 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:38:14.131474    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"default-dockercfg-6xmnv\""
Jul 28 10:38:14.131570 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:38:14.131463    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"kube-root-ca.crt\""
Jul 28 10:38:14.137212 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:38:14.137191    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fjzjqr"]
Jul 28 10:38:14.161842 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:38:14.161784    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:38:14.202050 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:38:14.202010    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crzvr\" (UniqueName: \"kubernetes.io/projected/c6bb1860-ddb4-473e-9c00-69934a26e168-kube-api-access-crzvr\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58fjzjqr\" (UID: \"c6bb1860-ddb4-473e-9c00-69934a26e168\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fjzjqr"
Jul 28 10:38:14.202236 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:38:14.202060    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6bb1860-ddb4-473e-9c00-69934a26e168-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58fjzjqr\" (UID: \"c6bb1860-ddb4-473e-9c00-69934a26e168\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fjzjqr"
Jul 28 10:38:14.202236 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:38:14.202165    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6bb1860-ddb4-473e-9c00-69934a26e168-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58fjzjqr\" (UID: \"c6bb1860-ddb4-473e-9c00-69934a26e168\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fjzjqr"
Jul 28 10:38:14.202236 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:38:14.202201    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6bb1860-ddb4-473e-9c00-69934a26e168-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58fjzjqr\" (UID: \"c6bb1860-ddb4-473e-9c00-69934a26e168\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fjzjqr"
Jul 28 10:38:14.202236 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:38:14.202219    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6bb1860-ddb4-473e-9c00-69934a26e168-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58fjzjqr\" (UID: \"c6bb1860-ddb4-473e-9c00-69934a26e168\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fjzjqr"
Jul 28 10:38:14.202400 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:38:14.202252    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6bb1860-ddb4-473e-9c00-69934a26e168-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58fjzjqr\" (UID: \"c6bb1860-ddb4-473e-9c00-69934a26e168\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fjzjqr"
Jul 28 10:38:14.303604 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:38:14.303565    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crzvr\" (UniqueName: \"kubernetes.io/projected/c6bb1860-ddb4-473e-9c00-69934a26e168-kube-api-access-crzvr\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58fjzjqr\" (UID: \"c6bb1860-ddb4-473e-9c00-69934a26e168\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fjzjqr"
Jul 28 10:38:14.303604 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:38:14.303609    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6bb1860-ddb4-473e-9c00-69934a26e168-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58fjzjqr\" (UID: \"c6bb1860-ddb4-473e-9c00-69934a26e168\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fjzjqr"
Jul 28 10:38:14.303880 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:38:14.303736    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6bb1860-ddb4-473e-9c00-69934a26e168-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58fjzjqr\" (UID: \"c6bb1860-ddb4-473e-9c00-69934a26e168\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fjzjqr"
Jul 28 10:38:14.303880 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:38:14.303774    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6bb1860-ddb4-473e-9c00-69934a26e168-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58fjzjqr\" (UID: \"c6bb1860-ddb4-473e-9c00-69934a26e168\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fjzjqr"
Jul 28 10:38:14.303880 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:38:14.303793    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6bb1860-ddb4-473e-9c00-69934a26e168-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58fjzjqr\" (UID: \"c6bb1860-ddb4-473e-9c00-69934a26e168\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fjzjqr"
Jul 28 10:38:14.303880 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:38:14.303813    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6bb1860-ddb4-473e-9c00-69934a26e168-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58fjzjqr\" (UID: \"c6bb1860-ddb4-473e-9c00-69934a26e168\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fjzjqr"
Jul 28 10:38:14.304140 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:38:14.304120    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6bb1860-ddb4-473e-9c00-69934a26e168-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58fjzjqr\" (UID: \"c6bb1860-ddb4-473e-9c00-69934a26e168\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fjzjqr"
Jul 28 10:38:14.304207 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:38:14.304162    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6bb1860-ddb4-473e-9c00-69934a26e168-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58fjzjqr\" (UID: \"c6bb1860-ddb4-473e-9c00-69934a26e168\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fjzjqr"
Jul 28 10:38:14.304207 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:38:14.304183    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6bb1860-ddb4-473e-9c00-69934a26e168-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58fjzjqr\" (UID: \"c6bb1860-ddb4-473e-9c00-69934a26e168\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fjzjqr"
Jul 28 10:38:14.306005 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:38:14.305980    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6bb1860-ddb4-473e-9c00-69934a26e168-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58fjzjqr\" (UID: \"c6bb1860-ddb4-473e-9c00-69934a26e168\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fjzjqr"
Jul 28 10:38:14.306179 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:38:14.306162    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6bb1860-ddb4-473e-9c00-69934a26e168-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58fjzjqr\" (UID: \"c6bb1860-ddb4-473e-9c00-69934a26e168\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fjzjqr"
Jul 28 10:38:14.312165 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:38:14.312141    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-crzvr\" (UniqueName: \"kubernetes.io/projected/c6bb1860-ddb4-473e-9c00-69934a26e168-kube-api-access-crzvr\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58fjzjqr\" (UID: \"c6bb1860-ddb4-473e-9c00-69934a26e168\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fjzjqr"
Jul 28 10:38:14.374282 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:38:14.374215    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:38:14.440902 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:38:14.440818    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fjzjqr"
Jul 28 10:38:14.578276 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:38:14.578251    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fjzjqr"]
Jul 28 10:38:14.580370 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:38:14.580338    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6bb1860_ddb4_473e_9c00_69934a26e168.slice/crio-8cc36b3c57cc073501b9d5c3948a99c729251d5c634d40b64f7e208611e4c90d WatchSource:0}: Error finding container 8cc36b3c57cc073501b9d5c3948a99c729251d5c634d40b64f7e208611e4c90d: Status 404 returned error can't find the container with id 8cc36b3c57cc073501b9d5c3948a99c729251d5c634d40b64f7e208611e4c90d
Jul 28 10:38:14.756766 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:38:14.756663    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fjzjqr" event={"ID":"c6bb1860-ddb4-473e-9c00-69934a26e168","Type":"ContainerStarted","Data":"85f889e8925da2d2e421040e5d83de8c06b30660c2aa9ce3478f1849ec9fed76"}
Jul 28 10:38:14.756766 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:38:14.756759    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fjzjqr" event={"ID":"c6bb1860-ddb4-473e-9c00-69934a26e168","Type":"ContainerStarted","Data":"8cc36b3c57cc073501b9d5c3948a99c729251d5c634d40b64f7e208611e4c90d"}
Jul 28 10:38:14.774483 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:38:14.774429    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fjzjqr" podStartSLOduration=0.774410148 podStartE2EDuration="774.410148ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:38:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:38:14.773680055 +0000 UTC m=+1829.037076385" watchObservedRunningTime="2026-07-28 10:38:14.774410148 +0000 UTC m=+1829.037806476"
Jul 28 10:38:15.441099 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:38:15.441063    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fjzjqr"
Jul 28 10:38:24.441133 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:38:24.441093    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fjzjqr"
Jul 28 10:38:24.454032 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:38:24.454002    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fjzjqr"
Jul 28 10:38:24.817368 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:38:24.817329    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fjzjqr"
Jul 28 10:38:42.866901 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:38:42.866861    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:38:48.636980 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:38:48.636943    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:38:48.662383 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:38:48.662347    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fjzjqr"]
Jul 28 10:38:48.671365 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:38:48.671327    2579 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 28 10:38:48.671365 ip-10-0-141-50 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 10:38:48.671365 ip-10-0-141-50 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Jul 28 10:38:48.671365 ip-10-0-141-50 kubenswrapper[2579]:  > containerID="85f889e8925da2d2e421040e5d83de8c06b30660c2aa9ce3478f1849ec9fed76" cmd=["/bin/sleep","15"]
Jul 28 10:38:48.671659 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:38:48.671379    2579 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fjzjqr" message="" err=<
Jul 28 10:38:48.671659 ip-10-0-141-50 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 10:38:48.671659 ip-10-0-141-50 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Jul 28 10:38:48.671659 ip-10-0-141-50 kubenswrapper[2579]:  >
Jul 28 10:38:48.671659 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:38:48.671392    2579 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 28 10:38:48.671659 ip-10-0-141-50 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 10:38:48.671659 ip-10-0-141-50 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Jul 28 10:38:48.671659 ip-10-0-141-50 kubenswrapper[2579]:  > pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fjzjqr" podUID="c6bb1860-ddb4-473e-9c00-69934a26e168" containerName="main" containerID="cri-o://85f889e8925da2d2e421040e5d83de8c06b30660c2aa9ce3478f1849ec9fed76"
Jul 28 10:38:48.671659 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:38:48.671438    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fjzjqr" podUID="c6bb1860-ddb4-473e-9c00-69934a26e168" containerName="main" containerID="cri-o://85f889e8925da2d2e421040e5d83de8c06b30660c2aa9ce3478f1849ec9fed76" gracePeriod=60
Jul 28 10:38:48.730223 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:38:48.730183    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:38:48.806917 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:38:48.806859    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fjzjqr" podUID="c6bb1860-ddb4-473e-9c00-69934a26e168" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": dial tcp 10.134.0.50:8000: connect: connection refused"
Jul 28 10:38:48.829826 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:38:48.829763    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:38:48.930427 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:38:48.930356    2579 generic.go:356] "Generic (PLEG): container finished" podID="c6bb1860-ddb4-473e-9c00-69934a26e168" containerID="85f889e8925da2d2e421040e5d83de8c06b30660c2aa9ce3478f1849ec9fed76" exitCode=0
Jul 28 10:38:48.930802 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:38:48.930434    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fjzjqr" event={"ID":"c6bb1860-ddb4-473e-9c00-69934a26e168","Type":"ContainerDied","Data":"85f889e8925da2d2e421040e5d83de8c06b30660c2aa9ce3478f1849ec9fed76"}
Jul 28 10:38:48.930802 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:38:48.930478    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fjzjqr" event={"ID":"c6bb1860-ddb4-473e-9c00-69934a26e168","Type":"ContainerDied","Data":"8cc36b3c57cc073501b9d5c3948a99c729251d5c634d40b64f7e208611e4c90d"}
Jul 28 10:38:48.930802 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:38:48.930498    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8cc36b3c57cc073501b9d5c3948a99c729251d5c634d40b64f7e208611e4c90d"
Jul 28 10:38:48.950294 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:38:48.950183    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fjzjqr"
Jul 28 10:38:48.993627 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:38:48.993594    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6bb1860-ddb4-473e-9c00-69934a26e168-model-cache\") pod \"c6bb1860-ddb4-473e-9c00-69934a26e168\" (UID: \"c6bb1860-ddb4-473e-9c00-69934a26e168\") "
Jul 28 10:38:48.993627 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:38:48.993627    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6bb1860-ddb4-473e-9c00-69934a26e168-tmp-dir\") pod \"c6bb1860-ddb4-473e-9c00-69934a26e168\" (UID: \"c6bb1860-ddb4-473e-9c00-69934a26e168\") "
Jul 28 10:38:48.993882 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:38:48.993643    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6bb1860-ddb4-473e-9c00-69934a26e168-dshm\") pod \"c6bb1860-ddb4-473e-9c00-69934a26e168\" (UID: \"c6bb1860-ddb4-473e-9c00-69934a26e168\") "
Jul 28 10:38:48.993882 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:38:48.993674    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-crzvr\" (UniqueName: \"kubernetes.io/projected/c6bb1860-ddb4-473e-9c00-69934a26e168-kube-api-access-crzvr\") pod \"c6bb1860-ddb4-473e-9c00-69934a26e168\" (UID: \"c6bb1860-ddb4-473e-9c00-69934a26e168\") "
Jul 28 10:38:48.993882 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:38:48.993706    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6bb1860-ddb4-473e-9c00-69934a26e168-tls-certs\") pod \"c6bb1860-ddb4-473e-9c00-69934a26e168\" (UID: \"c6bb1860-ddb4-473e-9c00-69934a26e168\") "
Jul 28 10:38:48.993882 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:38:48.993776    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6bb1860-ddb4-473e-9c00-69934a26e168-home\") pod \"c6bb1860-ddb4-473e-9c00-69934a26e168\" (UID: \"c6bb1860-ddb4-473e-9c00-69934a26e168\") "
Jul 28 10:38:48.994071 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:38:48.993886    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6bb1860-ddb4-473e-9c00-69934a26e168-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c6bb1860-ddb4-473e-9c00-69934a26e168" (UID: "c6bb1860-ddb4-473e-9c00-69934a26e168"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:38:48.994071 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:38:48.993918    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6bb1860-ddb4-473e-9c00-69934a26e168-model-cache" (OuterVolumeSpecName: "model-cache") pod "c6bb1860-ddb4-473e-9c00-69934a26e168" (UID: "c6bb1860-ddb4-473e-9c00-69934a26e168"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:38:48.994166 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:38:48.994073    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6bb1860-ddb4-473e-9c00-69934a26e168-home" (OuterVolumeSpecName: "home") pod "c6bb1860-ddb4-473e-9c00-69934a26e168" (UID: "c6bb1860-ddb4-473e-9c00-69934a26e168"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:38:48.994312 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:38:48.994287    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6bb1860-ddb4-473e-9c00-69934a26e168-home\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 10:38:48.994387 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:38:48.994319    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6bb1860-ddb4-473e-9c00-69934a26e168-model-cache\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 10:38:48.994387 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:38:48.994335    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6bb1860-ddb4-473e-9c00-69934a26e168-tmp-dir\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 10:38:48.995957 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:38:48.995928    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6bb1860-ddb4-473e-9c00-69934a26e168-kube-api-access-crzvr" (OuterVolumeSpecName: "kube-api-access-crzvr") pod "c6bb1860-ddb4-473e-9c00-69934a26e168" (UID: "c6bb1860-ddb4-473e-9c00-69934a26e168"). InnerVolumeSpecName "kube-api-access-crzvr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:38:48.996058 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:38:48.996045    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6bb1860-ddb4-473e-9c00-69934a26e168-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c6bb1860-ddb4-473e-9c00-69934a26e168" (UID: "c6bb1860-ddb4-473e-9c00-69934a26e168"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:38:48.996141 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:38:48.996120    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6bb1860-ddb4-473e-9c00-69934a26e168-dshm" (OuterVolumeSpecName: "dshm") pod "c6bb1860-ddb4-473e-9c00-69934a26e168" (UID: "c6bb1860-ddb4-473e-9c00-69934a26e168"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:38:49.095118 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:38:49.095088    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6bb1860-ddb4-473e-9c00-69934a26e168-dshm\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 10:38:49.095118 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:38:49.095115    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crzvr\" (UniqueName: \"kubernetes.io/projected/c6bb1860-ddb4-473e-9c00-69934a26e168-kube-api-access-crzvr\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 10:38:49.095118 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:38:49.095126    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6bb1860-ddb4-473e-9c00-69934a26e168-tls-certs\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 10:38:49.934903 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:38:49.934871    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fjzjqr"
Jul 28 10:38:49.961004 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:38:49.960964    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fjzjqr"]
Jul 28 10:38:49.965940 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:38:49.965903    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fjzjqr"]
Jul 28 10:38:50.275725 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:38:50.275648    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c6bb1860-ddb4-473e-9c00-69934a26e168" path="/var/lib/kubelet/pods/c6bb1860-ddb4-473e-9c00-69934a26e168/volumes"
Jul 28 10:39:08.124175 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:39:08.124084    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:39:08.219904 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:39:08.219861    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:39:14.448012 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:39:14.447977    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:39:14.609765 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:39:14.609730    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7b5c48c787-pkzjr"]
Jul 28 10:39:14.610055 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:39:14.610044    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6bb1860-ddb4-473e-9c00-69934a26e168" containerName="main"
Jul 28 10:39:14.610100 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:39:14.610057    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6bb1860-ddb4-473e-9c00-69934a26e168" containerName="main"
Jul 28 10:39:14.610135 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:39:14.610109    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6bb1860-ddb4-473e-9c00-69934a26e168" containerName="main"
Jul 28 10:39:14.613399 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:39:14.613376    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7b5c48c787-pkzjr"
Jul 28 10:39:14.615658 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:39:14.615634    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"default-dockercfg-hvlmd\""
Jul 28 10:39:14.615768 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:39:14.615632    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"openshift-service-ca.crt\""
Jul 28 10:39:14.615768 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:39:14.615620    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"kube-root-ca.crt\""
Jul 28 10:39:14.615768 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:39:14.615670    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"seaweedfs-s3-creds\""
Jul 28 10:39:14.616249 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:39:14.616230    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"tokenizer-clean-path-test-kserve-self-signed-certs\""
Jul 28 10:39:14.623727 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:39:14.623699    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:39:14.624511 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:39:14.624488    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7b5c48c787-pkzjr"]
Jul 28 10:39:14.708214 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:39:14.708120    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1adf61d-d995-4434-8a0a-34e9b312f1f4-dshm\") pod \"tokenizer-clean-path-test-kserve-7b5c48c787-pkzjr\" (UID: \"e1adf61d-d995-4434-8a0a-34e9b312f1f4\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7b5c48c787-pkzjr"
Jul 28 10:39:14.708214 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:39:14.708156    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1adf61d-d995-4434-8a0a-34e9b312f1f4-home\") pod \"tokenizer-clean-path-test-kserve-7b5c48c787-pkzjr\" (UID: \"e1adf61d-d995-4434-8a0a-34e9b312f1f4\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7b5c48c787-pkzjr"
Jul 28 10:39:14.708214 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:39:14.708176    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1adf61d-d995-4434-8a0a-34e9b312f1f4-tls-certs\") pod \"tokenizer-clean-path-test-kserve-7b5c48c787-pkzjr\" (UID: \"e1adf61d-d995-4434-8a0a-34e9b312f1f4\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7b5c48c787-pkzjr"
Jul 28 10:39:14.708439 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:39:14.708292    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1adf61d-d995-4434-8a0a-34e9b312f1f4-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-7b5c48c787-pkzjr\" (UID: \"e1adf61d-d995-4434-8a0a-34e9b312f1f4\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7b5c48c787-pkzjr"
Jul 28 10:39:14.708439 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:39:14.708323    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e1adf61d-d995-4434-8a0a-34e9b312f1f4-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-7b5c48c787-pkzjr\" (UID: \"e1adf61d-d995-4434-8a0a-34e9b312f1f4\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7b5c48c787-pkzjr"
Jul 28 10:39:14.708439 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:39:14.708346    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdgsc\" (UniqueName: \"kubernetes.io/projected/e1adf61d-d995-4434-8a0a-34e9b312f1f4-kube-api-access-hdgsc\") pod \"tokenizer-clean-path-test-kserve-7b5c48c787-pkzjr\" (UID: \"e1adf61d-d995-4434-8a0a-34e9b312f1f4\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7b5c48c787-pkzjr"
Jul 28 10:39:14.708439 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:39:14.708401    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1adf61d-d995-4434-8a0a-34e9b312f1f4-model-cache\") pod \"tokenizer-clean-path-test-kserve-7b5c48c787-pkzjr\" (UID: \"e1adf61d-d995-4434-8a0a-34e9b312f1f4\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7b5c48c787-pkzjr"
Jul 28 10:39:14.809310 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:39:14.809269    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1adf61d-d995-4434-8a0a-34e9b312f1f4-dshm\") pod \"tokenizer-clean-path-test-kserve-7b5c48c787-pkzjr\" (UID: \"e1adf61d-d995-4434-8a0a-34e9b312f1f4\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7b5c48c787-pkzjr"
Jul 28 10:39:14.809310 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:39:14.809319    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1adf61d-d995-4434-8a0a-34e9b312f1f4-home\") pod \"tokenizer-clean-path-test-kserve-7b5c48c787-pkzjr\" (UID: \"e1adf61d-d995-4434-8a0a-34e9b312f1f4\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7b5c48c787-pkzjr"
Jul 28 10:39:14.809558 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:39:14.809346    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1adf61d-d995-4434-8a0a-34e9b312f1f4-tls-certs\") pod \"tokenizer-clean-path-test-kserve-7b5c48c787-pkzjr\" (UID: \"e1adf61d-d995-4434-8a0a-34e9b312f1f4\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7b5c48c787-pkzjr"
Jul 28 10:39:14.809558 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:39:14.809422    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1adf61d-d995-4434-8a0a-34e9b312f1f4-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-7b5c48c787-pkzjr\" (UID: \"e1adf61d-d995-4434-8a0a-34e9b312f1f4\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7b5c48c787-pkzjr"
Jul 28 10:39:14.809558 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:39:14.809447    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e1adf61d-d995-4434-8a0a-34e9b312f1f4-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-7b5c48c787-pkzjr\" (UID: \"e1adf61d-d995-4434-8a0a-34e9b312f1f4\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7b5c48c787-pkzjr"
Jul 28 10:39:14.809558 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:39:14.809474    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdgsc\" (UniqueName: \"kubernetes.io/projected/e1adf61d-d995-4434-8a0a-34e9b312f1f4-kube-api-access-hdgsc\") pod \"tokenizer-clean-path-test-kserve-7b5c48c787-pkzjr\" (UID: \"e1adf61d-d995-4434-8a0a-34e9b312f1f4\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7b5c48c787-pkzjr"
Jul 28 10:39:14.809558 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:39:14.809510    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1adf61d-d995-4434-8a0a-34e9b312f1f4-model-cache\") pod \"tokenizer-clean-path-test-kserve-7b5c48c787-pkzjr\" (UID: \"e1adf61d-d995-4434-8a0a-34e9b312f1f4\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7b5c48c787-pkzjr"
Jul 28 10:39:14.809945 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:39:14.809920    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1adf61d-d995-4434-8a0a-34e9b312f1f4-home\") pod \"tokenizer-clean-path-test-kserve-7b5c48c787-pkzjr\" (UID: \"e1adf61d-d995-4434-8a0a-34e9b312f1f4\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7b5c48c787-pkzjr"
Jul 28 10:39:14.810170 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:39:14.810148    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1adf61d-d995-4434-8a0a-34e9b312f1f4-model-cache\") pod \"tokenizer-clean-path-test-kserve-7b5c48c787-pkzjr\" (UID: \"e1adf61d-d995-4434-8a0a-34e9b312f1f4\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7b5c48c787-pkzjr"
Jul 28 10:39:14.810680 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:39:14.810272    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1adf61d-d995-4434-8a0a-34e9b312f1f4-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-7b5c48c787-pkzjr\" (UID: \"e1adf61d-d995-4434-8a0a-34e9b312f1f4\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7b5c48c787-pkzjr"
Jul 28 10:39:14.810680 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:39:14.810630    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e1adf61d-d995-4434-8a0a-34e9b312f1f4-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-7b5c48c787-pkzjr\" (UID: \"e1adf61d-d995-4434-8a0a-34e9b312f1f4\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7b5c48c787-pkzjr"
Jul 28 10:39:14.812432 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:39:14.812409    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1adf61d-d995-4434-8a0a-34e9b312f1f4-dshm\") pod \"tokenizer-clean-path-test-kserve-7b5c48c787-pkzjr\" (UID: \"e1adf61d-d995-4434-8a0a-34e9b312f1f4\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7b5c48c787-pkzjr"
Jul 28 10:39:14.812768 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:39:14.812746    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1adf61d-d995-4434-8a0a-34e9b312f1f4-tls-certs\") pod \"tokenizer-clean-path-test-kserve-7b5c48c787-pkzjr\" (UID: \"e1adf61d-d995-4434-8a0a-34e9b312f1f4\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7b5c48c787-pkzjr"
Jul 28 10:39:14.821036 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:39:14.821012    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdgsc\" (UniqueName: \"kubernetes.io/projected/e1adf61d-d995-4434-8a0a-34e9b312f1f4-kube-api-access-hdgsc\") pod \"tokenizer-clean-path-test-kserve-7b5c48c787-pkzjr\" (UID: \"e1adf61d-d995-4434-8a0a-34e9b312f1f4\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7b5c48c787-pkzjr"
Jul 28 10:39:14.852894 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:39:14.852858    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65bqkkf"]
Jul 28 10:39:14.856439 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:39:14.856416    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65bqkkf"
Jul 28 10:39:14.858698 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:39:14.858673    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"tokenizer-clean-path-test-epp-sa-dockercfg-xnwhv\""
Jul 28 10:39:14.866858 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:39:14.866810    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:39:14.868296 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:39:14.868274    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65bqkkf"]
Jul 28 10:39:14.910321 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:39:14.910283    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b67fe68-f9d1-4a08-a36f-2eb8df32b4d1-tls-certs\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-866ccd65bqkkf\" (UID: \"1b67fe68-f9d1-4a08-a36f-2eb8df32b4d1\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65bqkkf"
Jul 28 10:39:14.910321 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:39:14.910323    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4stwm\" (UniqueName: \"kubernetes.io/projected/1b67fe68-f9d1-4a08-a36f-2eb8df32b4d1-kube-api-access-4stwm\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-866ccd65bqkkf\" (UID: \"1b67fe68-f9d1-4a08-a36f-2eb8df32b4d1\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65bqkkf"
Jul 28 10:39:14.924299 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:39:14.924262    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7b5c48c787-pkzjr"
Jul 28 10:39:14.930296 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:39:14.930264    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:39:15.011258 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:39:15.011197    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b67fe68-f9d1-4a08-a36f-2eb8df32b4d1-tls-certs\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-866ccd65bqkkf\" (UID: \"1b67fe68-f9d1-4a08-a36f-2eb8df32b4d1\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65bqkkf"
Jul 28 10:39:15.011258 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:39:15.011253    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4stwm\" (UniqueName: \"kubernetes.io/projected/1b67fe68-f9d1-4a08-a36f-2eb8df32b4d1-kube-api-access-4stwm\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-866ccd65bqkkf\" (UID: \"1b67fe68-f9d1-4a08-a36f-2eb8df32b4d1\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65bqkkf"
Jul 28 10:39:15.013800 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:39:15.013774    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b67fe68-f9d1-4a08-a36f-2eb8df32b4d1-tls-certs\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-866ccd65bqkkf\" (UID: \"1b67fe68-f9d1-4a08-a36f-2eb8df32b4d1\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65bqkkf"
Jul 28 10:39:15.020099 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:39:15.020075    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4stwm\" (UniqueName: \"kubernetes.io/projected/1b67fe68-f9d1-4a08-a36f-2eb8df32b4d1-kube-api-access-4stwm\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-866ccd65bqkkf\" (UID: \"1b67fe68-f9d1-4a08-a36f-2eb8df32b4d1\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65bqkkf"
Jul 28 10:39:15.064742 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:39:15.064717    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7b5c48c787-pkzjr"]
Jul 28 10:39:15.066510 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:39:15.066484    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1adf61d_d995_4434_8a0a_34e9b312f1f4.slice/crio-b102d6d42af7cf3a1ecb5dc0da48d7622f42afd8b9cdb96c54c0715539a41c77 WatchSource:0}: Error finding container b102d6d42af7cf3a1ecb5dc0da48d7622f42afd8b9cdb96c54c0715539a41c77: Status 404 returned error can't find the container with id b102d6d42af7cf3a1ecb5dc0da48d7622f42afd8b9cdb96c54c0715539a41c77
Jul 28 10:39:15.167837 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:39:15.167793    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65bqkkf"
Jul 28 10:39:15.312235 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:39:15.312200    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65bqkkf"]
Jul 28 10:39:15.314835 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:39:15.314809    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b67fe68_f9d1_4a08_a36f_2eb8df32b4d1.slice/crio-466696fc65802492d2be57a32abeca16776456b4c55ed46b9ebfeb6fe6085006 WatchSource:0}: Error finding container 466696fc65802492d2be57a32abeca16776456b4c55ed46b9ebfeb6fe6085006: Status 404 returned error can't find the container with id 466696fc65802492d2be57a32abeca16776456b4c55ed46b9ebfeb6fe6085006
Jul 28 10:39:16.067665 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:39:16.067619    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65bqkkf" event={"ID":"1b67fe68-f9d1-4a08-a36f-2eb8df32b4d1","Type":"ContainerStarted","Data":"4be33a7b154054ca03feb4ca45c4db19595b8f20fe365d8af672fb9ce681706e"}
Jul 28 10:39:16.068165 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:39:16.067682    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65bqkkf" event={"ID":"1b67fe68-f9d1-4a08-a36f-2eb8df32b4d1","Type":"ContainerStarted","Data":"466696fc65802492d2be57a32abeca16776456b4c55ed46b9ebfeb6fe6085006"}
Jul 28 10:39:16.068165 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:39:16.067741    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65bqkkf"
Jul 28 10:39:16.069476 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:39:16.069446    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7b5c48c787-pkzjr" event={"ID":"e1adf61d-d995-4434-8a0a-34e9b312f1f4","Type":"ContainerStarted","Data":"5aa62d4e335d8a26d037e9964e89aa761f02980ee3e7eff19d5106f3f7748118"}
Jul 28 10:39:16.069637 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:39:16.069484    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7b5c48c787-pkzjr" event={"ID":"e1adf61d-d995-4434-8a0a-34e9b312f1f4","Type":"ContainerStarted","Data":"b102d6d42af7cf3a1ecb5dc0da48d7622f42afd8b9cdb96c54c0715539a41c77"}
Jul 28 10:39:16.088812 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:39:16.088746    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65bqkkf" podStartSLOduration=2.088727404 podStartE2EDuration="2.088727404s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:39:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:39:16.086825375 +0000 UTC m=+1890.350221703" watchObservedRunningTime="2026-07-28 10:39:16.088727404 +0000 UTC m=+1890.352123733"
Jul 28 10:39:19.088291 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:39:19.088257    2579 generic.go:356] "Generic (PLEG): container finished" podID="e1adf61d-d995-4434-8a0a-34e9b312f1f4" containerID="5aa62d4e335d8a26d037e9964e89aa761f02980ee3e7eff19d5106f3f7748118" exitCode=0
Jul 28 10:39:19.088734 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:39:19.088330    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7b5c48c787-pkzjr" event={"ID":"e1adf61d-d995-4434-8a0a-34e9b312f1f4","Type":"ContainerDied","Data":"5aa62d4e335d8a26d037e9964e89aa761f02980ee3e7eff19d5106f3f7748118"}
Jul 28 10:39:20.095211 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:39:20.095175    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7b5c48c787-pkzjr" event={"ID":"e1adf61d-d995-4434-8a0a-34e9b312f1f4","Type":"ContainerStarted","Data":"07ec9f86ff239bf555663288d93ea596bb404b0a3ad65782710f60b8920f1518"}
Jul 28 10:39:20.116918 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:39:20.116865    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7b5c48c787-pkzjr" podStartSLOduration=2.549245119 podStartE2EDuration="6.116848329s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.56760321s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:39:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:39:20.115533206 +0000 UTC m=+1894.378929535" watchObservedRunningTime="2026-07-28 10:39:20.116848329 +0000 UTC m=+1894.380244658"
Jul 28 10:39:20.230399 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:39:20.230367    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:39:20.303573 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:39:20.303518    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:39:20.924788 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:39:20.924748    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7b5c48c787-pkzjr"
Jul 28 10:39:24.924725 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:39:24.924684    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7b5c48c787-pkzjr"
Jul 28 10:39:24.937605 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:39:24.937571    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7b5c48c787-pkzjr"
Jul 28 10:39:25.130516 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:39:25.130489    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7b5c48c787-pkzjr"
Jul 28 10:39:29.723976 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:39:29.723942    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:39:47.077878 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:39:47.077848    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65bqkkf"
Jul 28 10:40:26.721133 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:40:26.721087    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:40:26.723820 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:40:26.723789    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65bqkkf"]
Jul 28 10:40:26.727690 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:40:26.727669    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7b5c48c787-pkzjr"]
Jul 28 10:40:26.736121 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:40:26.736089    2579 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 28 10:40:26.736121 ip-10-0-141-50 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 10:40:26.736121 ip-10-0-141-50 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Jul 28 10:40:26.736121 ip-10-0-141-50 kubenswrapper[2579]:  > containerID="07ec9f86ff239bf555663288d93ea596bb404b0a3ad65782710f60b8920f1518" cmd=["/bin/sleep","15"]
Jul 28 10:40:26.736416 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:40:26.736135    2579 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7b5c48c787-pkzjr" message="" err=<
Jul 28 10:40:26.736416 ip-10-0-141-50 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 10:40:26.736416 ip-10-0-141-50 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Jul 28 10:40:26.736416 ip-10-0-141-50 kubenswrapper[2579]:  >
Jul 28 10:40:26.736416 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:40:26.736147    2579 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 28 10:40:26.736416 ip-10-0-141-50 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 10:40:26.736416 ip-10-0-141-50 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Jul 28 10:40:26.736416 ip-10-0-141-50 kubenswrapper[2579]:  > pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7b5c48c787-pkzjr" podUID="e1adf61d-d995-4434-8a0a-34e9b312f1f4" containerName="main" containerID="cri-o://07ec9f86ff239bf555663288d93ea596bb404b0a3ad65782710f60b8920f1518"
Jul 28 10:40:26.736416 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:40:26.736182    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7b5c48c787-pkzjr" podUID="e1adf61d-d995-4434-8a0a-34e9b312f1f4" containerName="main" containerID="cri-o://07ec9f86ff239bf555663288d93ea596bb404b0a3ad65782710f60b8920f1518" gracePeriod=60
Jul 28 10:40:27.077960 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:40:27.077919    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65bqkkf" podUID="1b67fe68-f9d1-4a08-a36f-2eb8df32b4d1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 10:40:27.103083 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:40:27.103057    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7b5c48c787-pkzjr"
Jul 28 10:40:27.233685 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:40:27.233643    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hdgsc\" (UniqueName: \"kubernetes.io/projected/e1adf61d-d995-4434-8a0a-34e9b312f1f4-kube-api-access-hdgsc\") pod \"e1adf61d-d995-4434-8a0a-34e9b312f1f4\" (UID: \"e1adf61d-d995-4434-8a0a-34e9b312f1f4\") "
Jul 28 10:40:27.233892 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:40:27.233706    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1adf61d-d995-4434-8a0a-34e9b312f1f4-model-cache\") pod \"e1adf61d-d995-4434-8a0a-34e9b312f1f4\" (UID: \"e1adf61d-d995-4434-8a0a-34e9b312f1f4\") "
Jul 28 10:40:27.233892 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:40:27.233737    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1adf61d-d995-4434-8a0a-34e9b312f1f4-dshm\") pod \"e1adf61d-d995-4434-8a0a-34e9b312f1f4\" (UID: \"e1adf61d-d995-4434-8a0a-34e9b312f1f4\") "
Jul 28 10:40:27.233892 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:40:27.233772    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1adf61d-d995-4434-8a0a-34e9b312f1f4-home\") pod \"e1adf61d-d995-4434-8a0a-34e9b312f1f4\" (UID: \"e1adf61d-d995-4434-8a0a-34e9b312f1f4\") "
Jul 28 10:40:27.233892 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:40:27.233806    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1adf61d-d995-4434-8a0a-34e9b312f1f4-tmp-dir\") pod \"e1adf61d-d995-4434-8a0a-34e9b312f1f4\" (UID: \"e1adf61d-d995-4434-8a0a-34e9b312f1f4\") "
Jul 28 10:40:27.234130 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:40:27.233950    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1adf61d-d995-4434-8a0a-34e9b312f1f4-tls-certs\") pod \"e1adf61d-d995-4434-8a0a-34e9b312f1f4\" (UID: \"e1adf61d-d995-4434-8a0a-34e9b312f1f4\") "
Jul 28 10:40:27.234130 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:40:27.233998    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e1adf61d-d995-4434-8a0a-34e9b312f1f4-kserve-provision-location\") pod \"e1adf61d-d995-4434-8a0a-34e9b312f1f4\" (UID: \"e1adf61d-d995-4434-8a0a-34e9b312f1f4\") "
Jul 28 10:40:27.234130 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:40:27.234035    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1adf61d-d995-4434-8a0a-34e9b312f1f4-home" (OuterVolumeSpecName: "home") pod "e1adf61d-d995-4434-8a0a-34e9b312f1f4" (UID: "e1adf61d-d995-4434-8a0a-34e9b312f1f4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:40:27.234130 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:40:27.234074    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1adf61d-d995-4434-8a0a-34e9b312f1f4-model-cache" (OuterVolumeSpecName: "model-cache") pod "e1adf61d-d995-4434-8a0a-34e9b312f1f4" (UID: "e1adf61d-d995-4434-8a0a-34e9b312f1f4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:40:27.234130 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:40:27.234081    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1adf61d-d995-4434-8a0a-34e9b312f1f4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e1adf61d-d995-4434-8a0a-34e9b312f1f4" (UID: "e1adf61d-d995-4434-8a0a-34e9b312f1f4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:40:27.234449 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:40:27.234288    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1adf61d-d995-4434-8a0a-34e9b312f1f4-home\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 10:40:27.234449 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:40:27.234315    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1adf61d-d995-4434-8a0a-34e9b312f1f4-tmp-dir\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 10:40:27.234449 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:40:27.234329    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1adf61d-d995-4434-8a0a-34e9b312f1f4-model-cache\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 10:40:27.236184 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:40:27.236155    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1adf61d-d995-4434-8a0a-34e9b312f1f4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e1adf61d-d995-4434-8a0a-34e9b312f1f4" (UID: "e1adf61d-d995-4434-8a0a-34e9b312f1f4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:40:27.236330 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:40:27.236215    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1adf61d-d995-4434-8a0a-34e9b312f1f4-kube-api-access-hdgsc" (OuterVolumeSpecName: "kube-api-access-hdgsc") pod "e1adf61d-d995-4434-8a0a-34e9b312f1f4" (UID: "e1adf61d-d995-4434-8a0a-34e9b312f1f4"). InnerVolumeSpecName "kube-api-access-hdgsc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:40:27.236330 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:40:27.236260    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1adf61d-d995-4434-8a0a-34e9b312f1f4-dshm" (OuterVolumeSpecName: "dshm") pod "e1adf61d-d995-4434-8a0a-34e9b312f1f4" (UID: "e1adf61d-d995-4434-8a0a-34e9b312f1f4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:40:27.299095 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:40:27.299035    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1adf61d-d995-4434-8a0a-34e9b312f1f4-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e1adf61d-d995-4434-8a0a-34e9b312f1f4" (UID: "e1adf61d-d995-4434-8a0a-34e9b312f1f4"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:40:27.334756 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:40:27.334719    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1adf61d-d995-4434-8a0a-34e9b312f1f4-tls-certs\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 10:40:27.334756 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:40:27.334753    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e1adf61d-d995-4434-8a0a-34e9b312f1f4-kserve-provision-location\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 10:40:27.334756 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:40:27.334763    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdgsc\" (UniqueName: \"kubernetes.io/projected/e1adf61d-d995-4434-8a0a-34e9b312f1f4-kube-api-access-hdgsc\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 10:40:27.334992 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:40:27.334772    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1adf61d-d995-4434-8a0a-34e9b312f1f4-dshm\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 10:40:27.436550 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:40:27.436504    2579 generic.go:356] "Generic (PLEG): container finished" podID="e1adf61d-d995-4434-8a0a-34e9b312f1f4" containerID="07ec9f86ff239bf555663288d93ea596bb404b0a3ad65782710f60b8920f1518" exitCode=0
Jul 28 10:40:27.436719 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:40:27.436577    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7b5c48c787-pkzjr" event={"ID":"e1adf61d-d995-4434-8a0a-34e9b312f1f4","Type":"ContainerDied","Data":"07ec9f86ff239bf555663288d93ea596bb404b0a3ad65782710f60b8920f1518"}
Jul 28 10:40:27.436719 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:40:27.436628    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7b5c48c787-pkzjr" event={"ID":"e1adf61d-d995-4434-8a0a-34e9b312f1f4","Type":"ContainerDied","Data":"b102d6d42af7cf3a1ecb5dc0da48d7622f42afd8b9cdb96c54c0715539a41c77"}
Jul 28 10:40:27.436719 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:40:27.436650    2579 scope.go:117] "RemoveContainer" containerID="07ec9f86ff239bf555663288d93ea596bb404b0a3ad65782710f60b8920f1518"
Jul 28 10:40:27.436719 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:40:27.436582    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7b5c48c787-pkzjr"
Jul 28 10:40:27.446999 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:40:27.446980    2579 scope.go:117] "RemoveContainer" containerID="5aa62d4e335d8a26d037e9964e89aa761f02980ee3e7eff19d5106f3f7748118"
Jul 28 10:40:27.455779 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:40:27.455757    2579 scope.go:117] "RemoveContainer" containerID="07ec9f86ff239bf555663288d93ea596bb404b0a3ad65782710f60b8920f1518"
Jul 28 10:40:27.456034 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:40:27.456012    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07ec9f86ff239bf555663288d93ea596bb404b0a3ad65782710f60b8920f1518\": container with ID starting with 07ec9f86ff239bf555663288d93ea596bb404b0a3ad65782710f60b8920f1518 not found: ID does not exist" containerID="07ec9f86ff239bf555663288d93ea596bb404b0a3ad65782710f60b8920f1518"
Jul 28 10:40:27.456097 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:40:27.456047    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07ec9f86ff239bf555663288d93ea596bb404b0a3ad65782710f60b8920f1518"} err="failed to get container status \"07ec9f86ff239bf555663288d93ea596bb404b0a3ad65782710f60b8920f1518\": rpc error: code = NotFound desc = could not find container \"07ec9f86ff239bf555663288d93ea596bb404b0a3ad65782710f60b8920f1518\": container with ID starting with 07ec9f86ff239bf555663288d93ea596bb404b0a3ad65782710f60b8920f1518 not found: ID does not exist"
Jul 28 10:40:27.456097 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:40:27.456065    2579 scope.go:117] "RemoveContainer" containerID="5aa62d4e335d8a26d037e9964e89aa761f02980ee3e7eff19d5106f3f7748118"
Jul 28 10:40:27.456329 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:40:27.456311    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5aa62d4e335d8a26d037e9964e89aa761f02980ee3e7eff19d5106f3f7748118\": container with ID starting with 5aa62d4e335d8a26d037e9964e89aa761f02980ee3e7eff19d5106f3f7748118 not found: ID does not exist" containerID="5aa62d4e335d8a26d037e9964e89aa761f02980ee3e7eff19d5106f3f7748118"
Jul 28 10:40:27.456382 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:40:27.456338    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5aa62d4e335d8a26d037e9964e89aa761f02980ee3e7eff19d5106f3f7748118"} err="failed to get container status \"5aa62d4e335d8a26d037e9964e89aa761f02980ee3e7eff19d5106f3f7748118\": rpc error: code = NotFound desc = could not find container \"5aa62d4e335d8a26d037e9964e89aa761f02980ee3e7eff19d5106f3f7748118\": container with ID starting with 5aa62d4e335d8a26d037e9964e89aa761f02980ee3e7eff19d5106f3f7748118 not found: ID does not exist"
Jul 28 10:40:27.462571 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:40:27.462526    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7b5c48c787-pkzjr"]
Jul 28 10:40:27.466243 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:40:27.466221    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7b5c48c787-pkzjr"]
Jul 28 10:40:28.091672 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:40:28.091620    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:40:28.276560 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:40:28.276518    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e1adf61d-d995-4434-8a0a-34e9b312f1f4" path="/var/lib/kubelet/pods/e1adf61d-d995-4434-8a0a-34e9b312f1f4/volumes"
Jul 28 10:40:37.078609 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:40:37.078569    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65bqkkf" podUID="1b67fe68-f9d1-4a08-a36f-2eb8df32b4d1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 10:40:41.744700 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:40:41.744596    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65bqkkf" podUID="1b67fe68-f9d1-4a08-a36f-2eb8df32b4d1" containerName="main" containerID="cri-o://4be33a7b154054ca03feb4ca45c4db19595b8f20fe365d8af672fb9ce681706e" gracePeriod=45
Jul 28 10:40:41.999676 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:40:41.999612    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65bqkkf"
Jul 28 10:40:42.048189 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:40:42.048157    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4stwm\" (UniqueName: \"kubernetes.io/projected/1b67fe68-f9d1-4a08-a36f-2eb8df32b4d1-kube-api-access-4stwm\") pod \"1b67fe68-f9d1-4a08-a36f-2eb8df32b4d1\" (UID: \"1b67fe68-f9d1-4a08-a36f-2eb8df32b4d1\") "
Jul 28 10:40:42.048367 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:40:42.048231    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b67fe68-f9d1-4a08-a36f-2eb8df32b4d1-tls-certs\") pod \"1b67fe68-f9d1-4a08-a36f-2eb8df32b4d1\" (UID: \"1b67fe68-f9d1-4a08-a36f-2eb8df32b4d1\") "
Jul 28 10:40:42.050312 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:40:42.050279    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b67fe68-f9d1-4a08-a36f-2eb8df32b4d1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1b67fe68-f9d1-4a08-a36f-2eb8df32b4d1" (UID: "1b67fe68-f9d1-4a08-a36f-2eb8df32b4d1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:40:42.050418 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:40:42.050343    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b67fe68-f9d1-4a08-a36f-2eb8df32b4d1-kube-api-access-4stwm" (OuterVolumeSpecName: "kube-api-access-4stwm") pod "1b67fe68-f9d1-4a08-a36f-2eb8df32b4d1" (UID: "1b67fe68-f9d1-4a08-a36f-2eb8df32b4d1"). InnerVolumeSpecName "kube-api-access-4stwm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:40:42.149782 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:40:42.149103    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b67fe68-f9d1-4a08-a36f-2eb8df32b4d1-tls-certs\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 10:40:42.154747 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:40:42.154716    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4stwm\" (UniqueName: \"kubernetes.io/projected/1b67fe68-f9d1-4a08-a36f-2eb8df32b4d1-kube-api-access-4stwm\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 10:40:42.511597 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:40:42.511562    2579 generic.go:356] "Generic (PLEG): container finished" podID="1b67fe68-f9d1-4a08-a36f-2eb8df32b4d1" containerID="4be33a7b154054ca03feb4ca45c4db19595b8f20fe365d8af672fb9ce681706e" exitCode=0
Jul 28 10:40:42.511768 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:40:42.511654    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65bqkkf" event={"ID":"1b67fe68-f9d1-4a08-a36f-2eb8df32b4d1","Type":"ContainerDied","Data":"4be33a7b154054ca03feb4ca45c4db19595b8f20fe365d8af672fb9ce681706e"}
Jul 28 10:40:42.511768 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:40:42.511682    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65bqkkf"
Jul 28 10:40:42.511768 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:40:42.511722    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65bqkkf" event={"ID":"1b67fe68-f9d1-4a08-a36f-2eb8df32b4d1","Type":"ContainerDied","Data":"466696fc65802492d2be57a32abeca16776456b4c55ed46b9ebfeb6fe6085006"}
Jul 28 10:40:42.511768 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:40:42.511744    2579 scope.go:117] "RemoveContainer" containerID="4be33a7b154054ca03feb4ca45c4db19595b8f20fe365d8af672fb9ce681706e"
Jul 28 10:40:42.520200 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:40:42.520179    2579 scope.go:117] "RemoveContainer" containerID="4be33a7b154054ca03feb4ca45c4db19595b8f20fe365d8af672fb9ce681706e"
Jul 28 10:40:42.520438 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:40:42.520417    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4be33a7b154054ca03feb4ca45c4db19595b8f20fe365d8af672fb9ce681706e\": container with ID starting with 4be33a7b154054ca03feb4ca45c4db19595b8f20fe365d8af672fb9ce681706e not found: ID does not exist" containerID="4be33a7b154054ca03feb4ca45c4db19595b8f20fe365d8af672fb9ce681706e"
Jul 28 10:40:42.520484 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:40:42.520447    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4be33a7b154054ca03feb4ca45c4db19595b8f20fe365d8af672fb9ce681706e"} err="failed to get container status \"4be33a7b154054ca03feb4ca45c4db19595b8f20fe365d8af672fb9ce681706e\": rpc error: code = NotFound desc = could not find container \"4be33a7b154054ca03feb4ca45c4db19595b8f20fe365d8af672fb9ce681706e\": container with ID starting with 4be33a7b154054ca03feb4ca45c4db19595b8f20fe365d8af672fb9ce681706e not found: ID does not exist"
Jul 28 10:40:42.527922 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:40:42.527900    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65bqkkf"]
Jul 28 10:40:42.532140 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:40:42.532119    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-866ccd65bqkkf"]
Jul 28 10:40:44.276034 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:40:44.275999    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b67fe68-f9d1-4a08-a36f-2eb8df32b4d1" path="/var/lib/kubelet/pods/1b67fe68-f9d1-4a08-a36f-2eb8df32b4d1/volumes"
Jul 28 10:40:46.740139 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:40:46.740052    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:40:46.843223 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:40:46.843182    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:40:53.234021 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:40:53.233985    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:40:59.163042 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:40:59.163003    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:40:59.255223 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:40:59.255189    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:41:02.697719 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:41:02.697681    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-57f9fb87d6-5lzst"]
Jul 28 10:41:02.698930 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:41:02.698902    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1adf61d-d995-4434-8a0a-34e9b312f1f4" containerName="storage-initializer"
Jul 28 10:41:02.699099 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:41:02.699085    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1adf61d-d995-4434-8a0a-34e9b312f1f4" containerName="storage-initializer"
Jul 28 10:41:02.699185 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:41:02.699175    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b67fe68-f9d1-4a08-a36f-2eb8df32b4d1" containerName="main"
Jul 28 10:41:02.699256 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:41:02.699248    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b67fe68-f9d1-4a08-a36f-2eb8df32b4d1" containerName="main"
Jul 28 10:41:02.699354 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:41:02.699345    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1adf61d-d995-4434-8a0a-34e9b312f1f4" containerName="main"
Jul 28 10:41:02.699423 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:41:02.699415    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1adf61d-d995-4434-8a0a-34e9b312f1f4" containerName="main"
Jul 28 10:41:02.699684 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:41:02.699671    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1adf61d-d995-4434-8a0a-34e9b312f1f4" containerName="main"
Jul 28 10:41:02.699785 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:41:02.699776    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b67fe68-f9d1-4a08-a36f-2eb8df32b4d1" containerName="main"
Jul 28 10:41:02.702981 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:41:02.702957    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-57f9fb87d6-5lzst"
Jul 28 10:41:02.703129 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:41:02.702984    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:41:02.705077 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:41:02.705054    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"seaweedfs-s3-creds\""
Jul 28 10:41:02.705252 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:41:02.705233    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"openshift-service-ca.crt\""
Jul 28 10:41:02.705784 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:41:02.705760    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"default-dockercfg-7r99c\""
Jul 28 10:41:02.705877 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:41:02.705819    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"tokenizer-migration-path-test-kserve-self-signed-certs\""
Jul 28 10:41:02.705877 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:41:02.705848    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"kube-root-ca.crt\""
Jul 28 10:41:02.713170 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:41:02.713143    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-57f9fb87d6-5lzst"]
Jul 28 10:41:02.827149 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:41:02.827110    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1c47956-511e-4b8c-bb2c-35477a816fcb-home\") pod \"tokenizer-migration-path-test-kserve-57f9fb87d6-5lzst\" (UID: \"d1c47956-511e-4b8c-bb2c-35477a816fcb\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-57f9fb87d6-5lzst"
Jul 28 10:41:02.827149 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:41:02.827152    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1c47956-511e-4b8c-bb2c-35477a816fcb-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-57f9fb87d6-5lzst\" (UID: \"d1c47956-511e-4b8c-bb2c-35477a816fcb\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-57f9fb87d6-5lzst"
Jul 28 10:41:02.827408 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:41:02.827173    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1c47956-511e-4b8c-bb2c-35477a816fcb-tls-certs\") pod \"tokenizer-migration-path-test-kserve-57f9fb87d6-5lzst\" (UID: \"d1c47956-511e-4b8c-bb2c-35477a816fcb\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-57f9fb87d6-5lzst"
Jul 28 10:41:02.827408 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:41:02.827233    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1c47956-511e-4b8c-bb2c-35477a816fcb-dshm\") pod \"tokenizer-migration-path-test-kserve-57f9fb87d6-5lzst\" (UID: \"d1c47956-511e-4b8c-bb2c-35477a816fcb\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-57f9fb87d6-5lzst"
Jul 28 10:41:02.827408 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:41:02.827320    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1c47956-511e-4b8c-bb2c-35477a816fcb-model-cache\") pod \"tokenizer-migration-path-test-kserve-57f9fb87d6-5lzst\" (UID: \"d1c47956-511e-4b8c-bb2c-35477a816fcb\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-57f9fb87d6-5lzst"
Jul 28 10:41:02.827408 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:41:02.827344    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9zlw\" (UniqueName: \"kubernetes.io/projected/d1c47956-511e-4b8c-bb2c-35477a816fcb-kube-api-access-k9zlw\") pod \"tokenizer-migration-path-test-kserve-57f9fb87d6-5lzst\" (UID: \"d1c47956-511e-4b8c-bb2c-35477a816fcb\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-57f9fb87d6-5lzst"
Jul 28 10:41:02.827408 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:41:02.827370    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d1c47956-511e-4b8c-bb2c-35477a816fcb-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-57f9fb87d6-5lzst\" (UID: \"d1c47956-511e-4b8c-bb2c-35477a816fcb\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-57f9fb87d6-5lzst"
Jul 28 10:41:02.928202 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:41:02.928168    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1c47956-511e-4b8c-bb2c-35477a816fcb-home\") pod \"tokenizer-migration-path-test-kserve-57f9fb87d6-5lzst\" (UID: \"d1c47956-511e-4b8c-bb2c-35477a816fcb\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-57f9fb87d6-5lzst"
Jul 28 10:41:02.928202 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:41:02.928209    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1c47956-511e-4b8c-bb2c-35477a816fcb-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-57f9fb87d6-5lzst\" (UID: \"d1c47956-511e-4b8c-bb2c-35477a816fcb\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-57f9fb87d6-5lzst"
Jul 28 10:41:02.928467 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:41:02.928225    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1c47956-511e-4b8c-bb2c-35477a816fcb-tls-certs\") pod \"tokenizer-migration-path-test-kserve-57f9fb87d6-5lzst\" (UID: \"d1c47956-511e-4b8c-bb2c-35477a816fcb\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-57f9fb87d6-5lzst"
Jul 28 10:41:02.928467 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:41:02.928245    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1c47956-511e-4b8c-bb2c-35477a816fcb-dshm\") pod \"tokenizer-migration-path-test-kserve-57f9fb87d6-5lzst\" (UID: \"d1c47956-511e-4b8c-bb2c-35477a816fcb\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-57f9fb87d6-5lzst"
Jul 28 10:41:02.928467 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:41:02.928271    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1c47956-511e-4b8c-bb2c-35477a816fcb-model-cache\") pod \"tokenizer-migration-path-test-kserve-57f9fb87d6-5lzst\" (UID: \"d1c47956-511e-4b8c-bb2c-35477a816fcb\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-57f9fb87d6-5lzst"
Jul 28 10:41:02.928664 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:41:02.928476    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9zlw\" (UniqueName: \"kubernetes.io/projected/d1c47956-511e-4b8c-bb2c-35477a816fcb-kube-api-access-k9zlw\") pod \"tokenizer-migration-path-test-kserve-57f9fb87d6-5lzst\" (UID: \"d1c47956-511e-4b8c-bb2c-35477a816fcb\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-57f9fb87d6-5lzst"
Jul 28 10:41:02.928664 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:41:02.928527    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d1c47956-511e-4b8c-bb2c-35477a816fcb-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-57f9fb87d6-5lzst\" (UID: \"d1c47956-511e-4b8c-bb2c-35477a816fcb\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-57f9fb87d6-5lzst"
Jul 28 10:41:02.928777 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:41:02.928664    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1c47956-511e-4b8c-bb2c-35477a816fcb-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-57f9fb87d6-5lzst\" (UID: \"d1c47956-511e-4b8c-bb2c-35477a816fcb\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-57f9fb87d6-5lzst"
Jul 28 10:41:02.928777 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:41:02.928723    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1c47956-511e-4b8c-bb2c-35477a816fcb-home\") pod \"tokenizer-migration-path-test-kserve-57f9fb87d6-5lzst\" (UID: \"d1c47956-511e-4b8c-bb2c-35477a816fcb\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-57f9fb87d6-5lzst"
Jul 28 10:41:02.928928 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:41:02.928908    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d1c47956-511e-4b8c-bb2c-35477a816fcb-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-57f9fb87d6-5lzst\" (UID: \"d1c47956-511e-4b8c-bb2c-35477a816fcb\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-57f9fb87d6-5lzst"
Jul 28 10:41:02.929059 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:41:02.929038    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1c47956-511e-4b8c-bb2c-35477a816fcb-model-cache\") pod \"tokenizer-migration-path-test-kserve-57f9fb87d6-5lzst\" (UID: \"d1c47956-511e-4b8c-bb2c-35477a816fcb\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-57f9fb87d6-5lzst"
Jul 28 10:41:02.931464 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:41:02.931441    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1c47956-511e-4b8c-bb2c-35477a816fcb-dshm\") pod \"tokenizer-migration-path-test-kserve-57f9fb87d6-5lzst\" (UID: \"d1c47956-511e-4b8c-bb2c-35477a816fcb\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-57f9fb87d6-5lzst"
Jul 28 10:41:02.931862 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:41:02.931840    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1c47956-511e-4b8c-bb2c-35477a816fcb-tls-certs\") pod \"tokenizer-migration-path-test-kserve-57f9fb87d6-5lzst\" (UID: \"d1c47956-511e-4b8c-bb2c-35477a816fcb\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-57f9fb87d6-5lzst"
Jul 28 10:41:02.940343 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:41:02.940314    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9zlw\" (UniqueName: \"kubernetes.io/projected/d1c47956-511e-4b8c-bb2c-35477a816fcb-kube-api-access-k9zlw\") pod \"tokenizer-migration-path-test-kserve-57f9fb87d6-5lzst\" (UID: \"d1c47956-511e-4b8c-bb2c-35477a816fcb\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-57f9fb87d6-5lzst"
Jul 28 10:41:02.962832 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:41:02.962739    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b92khft"]
Jul 28 10:41:02.965409 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:41:02.965380    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b92khft"
Jul 28 10:41:02.967404 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:41:02.967371    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"tokenizer-migration-path-test-epp-sa-dockercfg-q2vn4\""
Jul 28 10:41:02.977366 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:41:02.977275    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:41:02.978041 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:41:02.978017    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b92khft"]
Jul 28 10:41:03.014182 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:41:03.014139    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-57f9fb87d6-5lzst"
Jul 28 10:41:03.029796 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:41:03.029766    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drlf6\" (UniqueName: \"kubernetes.io/projected/e90d54e3-951f-45d7-ad02-b0f99bacca5a-kube-api-access-drlf6\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-c8b92khft\" (UID: \"e90d54e3-951f-45d7-ad02-b0f99bacca5a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b92khft"
Jul 28 10:41:03.029942 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:41:03.029818    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e90d54e3-951f-45d7-ad02-b0f99bacca5a-tls-certs\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-c8b92khft\" (UID: \"e90d54e3-951f-45d7-ad02-b0f99bacca5a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b92khft"
Jul 28 10:41:03.072708 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:41:03.072672    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:41:03.131179 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:41:03.131137    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drlf6\" (UniqueName: \"kubernetes.io/projected/e90d54e3-951f-45d7-ad02-b0f99bacca5a-kube-api-access-drlf6\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-c8b92khft\" (UID: \"e90d54e3-951f-45d7-ad02-b0f99bacca5a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b92khft"
Jul 28 10:41:03.131386 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:41:03.131227    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e90d54e3-951f-45d7-ad02-b0f99bacca5a-tls-certs\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-c8b92khft\" (UID: \"e90d54e3-951f-45d7-ad02-b0f99bacca5a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b92khft"
Jul 28 10:41:03.134042 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:41:03.134011    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e90d54e3-951f-45d7-ad02-b0f99bacca5a-tls-certs\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-c8b92khft\" (UID: \"e90d54e3-951f-45d7-ad02-b0f99bacca5a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b92khft"
Jul 28 10:41:03.139621 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:41:03.139596    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-drlf6\" (UniqueName: \"kubernetes.io/projected/e90d54e3-951f-45d7-ad02-b0f99bacca5a-kube-api-access-drlf6\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-c8b92khft\" (UID: \"e90d54e3-951f-45d7-ad02-b0f99bacca5a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b92khft"
Jul 28 10:41:03.157492 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:41:03.157464    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-57f9fb87d6-5lzst"]
Jul 28 10:41:03.159796 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:41:03.159768    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1c47956_511e_4b8c_bb2c_35477a816fcb.slice/crio-4b6f58fed59761e580bbdfe58557e49809f81d47972aaee2b4b0ee034c46bcd1 WatchSource:0}: Error finding container 4b6f58fed59761e580bbdfe58557e49809f81d47972aaee2b4b0ee034c46bcd1: Status 404 returned error can't find the container with id 4b6f58fed59761e580bbdfe58557e49809f81d47972aaee2b4b0ee034c46bcd1
Jul 28 10:41:03.278499 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:41:03.278459    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b92khft"
Jul 28 10:41:03.456772 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:41:03.456745    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b92khft"]
Jul 28 10:41:03.459190 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:41:03.459162    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode90d54e3_951f_45d7_ad02_b0f99bacca5a.slice/crio-63b42e3d8e647b02894bc19166c09e3da864477a5835e97949e60a9f20c7dae0 WatchSource:0}: Error finding container 63b42e3d8e647b02894bc19166c09e3da864477a5835e97949e60a9f20c7dae0: Status 404 returned error can't find the container with id 63b42e3d8e647b02894bc19166c09e3da864477a5835e97949e60a9f20c7dae0
Jul 28 10:41:03.616513 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:41:03.616470    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-57f9fb87d6-5lzst" event={"ID":"d1c47956-511e-4b8c-bb2c-35477a816fcb","Type":"ContainerStarted","Data":"76026b31d7dbe652517368cf993dfe64ab0daffc3a56d2f244998a2c6b309b4f"}
Jul 28 10:41:03.616733 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:41:03.616560    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-57f9fb87d6-5lzst" event={"ID":"d1c47956-511e-4b8c-bb2c-35477a816fcb","Type":"ContainerStarted","Data":"4b6f58fed59761e580bbdfe58557e49809f81d47972aaee2b4b0ee034c46bcd1"}
Jul 28 10:41:03.618125 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:41:03.618095    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b92khft" event={"ID":"e90d54e3-951f-45d7-ad02-b0f99bacca5a","Type":"ContainerStarted","Data":"7417c681cc6dc178fe1416ccaa2d7bc1d58efa256f29f0587f6c29af737e59ac"}
Jul 28 10:41:03.618231 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:41:03.618152    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b92khft" event={"ID":"e90d54e3-951f-45d7-ad02-b0f99bacca5a","Type":"ContainerStarted","Data":"63b42e3d8e647b02894bc19166c09e3da864477a5835e97949e60a9f20c7dae0"}
Jul 28 10:41:03.618231 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:41:03.618184    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b92khft"
Jul 28 10:41:03.651571 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:41:03.651498    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b92khft" podStartSLOduration=1.65148136 podStartE2EDuration="1.65148136s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:41:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:41:03.65006305 +0000 UTC m=+1997.913459383" watchObservedRunningTime="2026-07-28 10:41:03.65148136 +0000 UTC m=+1997.914877688"
Jul 28 10:41:07.642519 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:41:07.642475    2579 generic.go:356] "Generic (PLEG): container finished" podID="d1c47956-511e-4b8c-bb2c-35477a816fcb" containerID="76026b31d7dbe652517368cf993dfe64ab0daffc3a56d2f244998a2c6b309b4f" exitCode=0
Jul 28 10:41:07.642912 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:41:07.642564    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-57f9fb87d6-5lzst" event={"ID":"d1c47956-511e-4b8c-bb2c-35477a816fcb","Type":"ContainerDied","Data":"76026b31d7dbe652517368cf993dfe64ab0daffc3a56d2f244998a2c6b309b4f"}
Jul 28 10:41:08.649914 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:41:08.649880    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-57f9fb87d6-5lzst" event={"ID":"d1c47956-511e-4b8c-bb2c-35477a816fcb","Type":"ContainerStarted","Data":"16df56925928392aca732bcd7c83c89e7b99ff137f1384460cf6d11966ee5d9e"}
Jul 28 10:41:08.671659 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:41:08.671596    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-57f9fb87d6-5lzst" podStartSLOduration=3.178084236 podStartE2EDuration="6.671575334s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.493491098s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:41:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:41:08.670708087 +0000 UTC m=+2002.934104416" watchObservedRunningTime="2026-07-28 10:41:08.671575334 +0000 UTC m=+2002.934971665"
Jul 28 10:41:09.015076 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:41:09.014986    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-57f9fb87d6-5lzst"
Jul 28 10:41:13.014785 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:41:13.014728    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-57f9fb87d6-5lzst"
Jul 28 10:41:13.027584 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:41:13.027531    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-57f9fb87d6-5lzst"
Jul 28 10:41:13.687306 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:41:13.687273    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-57f9fb87d6-5lzst"
Jul 28 10:41:17.455322 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:41:17.455285    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:41:26.942608 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:41:26.942566    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:41:34.625564 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:41:34.625502    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b92khft"
Jul 28 10:41:40.321205 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:41:40.321165    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:41:40.321731 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:41:40.321270    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-57f9fb87d6-5lzst"]
Jul 28 10:41:40.324411 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:41:40.324372    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b92khft"]
Jul 28 10:41:40.331068 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:41:40.331034    2579 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 28 10:41:40.331068 ip-10-0-141-50 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 10:41:40.331068 ip-10-0-141-50 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Jul 28 10:41:40.331068 ip-10-0-141-50 kubenswrapper[2579]:  > containerID="16df56925928392aca732bcd7c83c89e7b99ff137f1384460cf6d11966ee5d9e" cmd=["/bin/sleep","15"]
Jul 28 10:41:40.331357 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:41:40.331093    2579 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-57f9fb87d6-5lzst" message="" err=<
Jul 28 10:41:40.331357 ip-10-0-141-50 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 10:41:40.331357 ip-10-0-141-50 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Jul 28 10:41:40.331357 ip-10-0-141-50 kubenswrapper[2579]:  >
Jul 28 10:41:40.331357 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:41:40.331111    2579 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 28 10:41:40.331357 ip-10-0-141-50 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 10:41:40.331357 ip-10-0-141-50 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Jul 28 10:41:40.331357 ip-10-0-141-50 kubenswrapper[2579]:  > pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-57f9fb87d6-5lzst" podUID="d1c47956-511e-4b8c-bb2c-35477a816fcb" containerName="main" containerID="cri-o://16df56925928392aca732bcd7c83c89e7b99ff137f1384460cf6d11966ee5d9e"
Jul 28 10:41:40.331357 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:41:40.331166    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-57f9fb87d6-5lzst" podUID="d1c47956-511e-4b8c-bb2c-35477a816fcb" containerName="main" containerID="cri-o://16df56925928392aca732bcd7c83c89e7b99ff137f1384460cf6d11966ee5d9e" gracePeriod=60
Jul 28 10:41:40.405468 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:41:40.405430    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:41:40.690211 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:41:40.690187    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-57f9fb87d6-5lzst"
Jul 28 10:41:40.727471 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:41:40.727435    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k9zlw\" (UniqueName: \"kubernetes.io/projected/d1c47956-511e-4b8c-bb2c-35477a816fcb-kube-api-access-k9zlw\") pod \"d1c47956-511e-4b8c-bb2c-35477a816fcb\" (UID: \"d1c47956-511e-4b8c-bb2c-35477a816fcb\") "
Jul 28 10:41:40.727683 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:41:40.727491    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1c47956-511e-4b8c-bb2c-35477a816fcb-tmp-dir\") pod \"d1c47956-511e-4b8c-bb2c-35477a816fcb\" (UID: \"d1c47956-511e-4b8c-bb2c-35477a816fcb\") "
Jul 28 10:41:40.727683 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:41:40.727597    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d1c47956-511e-4b8c-bb2c-35477a816fcb-kserve-provision-location\") pod \"d1c47956-511e-4b8c-bb2c-35477a816fcb\" (UID: \"d1c47956-511e-4b8c-bb2c-35477a816fcb\") "
Jul 28 10:41:40.727859 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:41:40.727819    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1c47956-511e-4b8c-bb2c-35477a816fcb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d1c47956-511e-4b8c-bb2c-35477a816fcb" (UID: "d1c47956-511e-4b8c-bb2c-35477a816fcb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:41:40.730107 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:41:40.730064    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1c47956-511e-4b8c-bb2c-35477a816fcb-kube-api-access-k9zlw" (OuterVolumeSpecName: "kube-api-access-k9zlw") pod "d1c47956-511e-4b8c-bb2c-35477a816fcb" (UID: "d1c47956-511e-4b8c-bb2c-35477a816fcb"). InnerVolumeSpecName "kube-api-access-k9zlw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:41:40.730377 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:41:40.730345    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1c47956-511e-4b8c-bb2c-35477a816fcb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d1c47956-511e-4b8c-bb2c-35477a816fcb" (UID: "d1c47956-511e-4b8c-bb2c-35477a816fcb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:41:40.735709 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:41:40.727630    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1c47956-511e-4b8c-bb2c-35477a816fcb-tls-certs\") pod \"d1c47956-511e-4b8c-bb2c-35477a816fcb\" (UID: \"d1c47956-511e-4b8c-bb2c-35477a816fcb\") "
Jul 28 10:41:40.735709 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:41:40.734377    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1c47956-511e-4b8c-bb2c-35477a816fcb-dshm\") pod \"d1c47956-511e-4b8c-bb2c-35477a816fcb\" (UID: \"d1c47956-511e-4b8c-bb2c-35477a816fcb\") "
Jul 28 10:41:40.735709 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:41:40.734430    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1c47956-511e-4b8c-bb2c-35477a816fcb-model-cache\") pod \"d1c47956-511e-4b8c-bb2c-35477a816fcb\" (UID: \"d1c47956-511e-4b8c-bb2c-35477a816fcb\") "
Jul 28 10:41:40.735709 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:41:40.734460    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1c47956-511e-4b8c-bb2c-35477a816fcb-home\") pod \"d1c47956-511e-4b8c-bb2c-35477a816fcb\" (UID: \"d1c47956-511e-4b8c-bb2c-35477a816fcb\") "
Jul 28 10:41:40.735709 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:41:40.734853    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1c47956-511e-4b8c-bb2c-35477a816fcb-home" (OuterVolumeSpecName: "home") pod "d1c47956-511e-4b8c-bb2c-35477a816fcb" (UID: "d1c47956-511e-4b8c-bb2c-35477a816fcb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:41:40.735709 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:41:40.734871    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1c47956-511e-4b8c-bb2c-35477a816fcb-model-cache" (OuterVolumeSpecName: "model-cache") pod "d1c47956-511e-4b8c-bb2c-35477a816fcb" (UID: "d1c47956-511e-4b8c-bb2c-35477a816fcb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:41:40.735709 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:41:40.735013    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1c47956-511e-4b8c-bb2c-35477a816fcb-model-cache\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 10:41:40.735709 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:41:40.735031    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1c47956-511e-4b8c-bb2c-35477a816fcb-home\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 10:41:40.735709 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:41:40.735047    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9zlw\" (UniqueName: \"kubernetes.io/projected/d1c47956-511e-4b8c-bb2c-35477a816fcb-kube-api-access-k9zlw\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 10:41:40.735709 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:41:40.735062    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1c47956-511e-4b8c-bb2c-35477a816fcb-tmp-dir\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 10:41:40.735709 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:41:40.735075    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1c47956-511e-4b8c-bb2c-35477a816fcb-tls-certs\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 10:41:40.738791 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:41:40.738769    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1c47956-511e-4b8c-bb2c-35477a816fcb-dshm" (OuterVolumeSpecName: "dshm") pod "d1c47956-511e-4b8c-bb2c-35477a816fcb" (UID: "d1c47956-511e-4b8c-bb2c-35477a816fcb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:41:40.799399 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:41:40.799345    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1c47956-511e-4b8c-bb2c-35477a816fcb-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d1c47956-511e-4b8c-bb2c-35477a816fcb" (UID: "d1c47956-511e-4b8c-bb2c-35477a816fcb"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:41:40.814416 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:41:40.814378    2579 generic.go:356] "Generic (PLEG): container finished" podID="d1c47956-511e-4b8c-bb2c-35477a816fcb" containerID="16df56925928392aca732bcd7c83c89e7b99ff137f1384460cf6d11966ee5d9e" exitCode=0
Jul 28 10:41:40.814638 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:41:40.814434    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-57f9fb87d6-5lzst"
Jul 28 10:41:40.814638 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:41:40.814470    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-57f9fb87d6-5lzst" event={"ID":"d1c47956-511e-4b8c-bb2c-35477a816fcb","Type":"ContainerDied","Data":"16df56925928392aca732bcd7c83c89e7b99ff137f1384460cf6d11966ee5d9e"}
Jul 28 10:41:40.814638 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:41:40.814522    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-57f9fb87d6-5lzst" event={"ID":"d1c47956-511e-4b8c-bb2c-35477a816fcb","Type":"ContainerDied","Data":"4b6f58fed59761e580bbdfe58557e49809f81d47972aaee2b4b0ee034c46bcd1"}
Jul 28 10:41:40.814638 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:41:40.814575    2579 scope.go:117] "RemoveContainer" containerID="16df56925928392aca732bcd7c83c89e7b99ff137f1384460cf6d11966ee5d9e"
Jul 28 10:41:40.824016 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:41:40.823986    2579 scope.go:117] "RemoveContainer" containerID="76026b31d7dbe652517368cf993dfe64ab0daffc3a56d2f244998a2c6b309b4f"
Jul 28 10:41:40.832855 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:41:40.832834    2579 scope.go:117] "RemoveContainer" containerID="16df56925928392aca732bcd7c83c89e7b99ff137f1384460cf6d11966ee5d9e"
Jul 28 10:41:40.833131 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:41:40.833111    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16df56925928392aca732bcd7c83c89e7b99ff137f1384460cf6d11966ee5d9e\": container with ID starting with 16df56925928392aca732bcd7c83c89e7b99ff137f1384460cf6d11966ee5d9e not found: ID does not exist" containerID="16df56925928392aca732bcd7c83c89e7b99ff137f1384460cf6d11966ee5d9e"
Jul 28 10:41:40.833190 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:41:40.833139    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16df56925928392aca732bcd7c83c89e7b99ff137f1384460cf6d11966ee5d9e"} err="failed to get container status \"16df56925928392aca732bcd7c83c89e7b99ff137f1384460cf6d11966ee5d9e\": rpc error: code = NotFound desc = could not find container \"16df56925928392aca732bcd7c83c89e7b99ff137f1384460cf6d11966ee5d9e\": container with ID starting with 16df56925928392aca732bcd7c83c89e7b99ff137f1384460cf6d11966ee5d9e not found: ID does not exist"
Jul 28 10:41:40.833190 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:41:40.833158    2579 scope.go:117] "RemoveContainer" containerID="76026b31d7dbe652517368cf993dfe64ab0daffc3a56d2f244998a2c6b309b4f"
Jul 28 10:41:40.833579 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:41:40.833467    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76026b31d7dbe652517368cf993dfe64ab0daffc3a56d2f244998a2c6b309b4f\": container with ID starting with 76026b31d7dbe652517368cf993dfe64ab0daffc3a56d2f244998a2c6b309b4f not found: ID does not exist" containerID="76026b31d7dbe652517368cf993dfe64ab0daffc3a56d2f244998a2c6b309b4f"
Jul 28 10:41:40.833867 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:41:40.833506    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76026b31d7dbe652517368cf993dfe64ab0daffc3a56d2f244998a2c6b309b4f"} err="failed to get container status \"76026b31d7dbe652517368cf993dfe64ab0daffc3a56d2f244998a2c6b309b4f\": rpc error: code = NotFound desc = could not find container \"76026b31d7dbe652517368cf993dfe64ab0daffc3a56d2f244998a2c6b309b4f\": container with ID starting with 76026b31d7dbe652517368cf993dfe64ab0daffc3a56d2f244998a2c6b309b4f not found: ID does not exist"
Jul 28 10:41:40.836276 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:41:40.836257    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-57f9fb87d6-5lzst"]
Jul 28 10:41:40.837125 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:41:40.837104    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d1c47956-511e-4b8c-bb2c-35477a816fcb-kserve-provision-location\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 10:41:40.837125 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:41:40.837124    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1c47956-511e-4b8c-bb2c-35477a816fcb-dshm\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 10:41:40.839949 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:41:40.839929    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-57f9fb87d6-5lzst"]
Jul 28 10:41:41.676731 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:41:41.676681    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-57f9fb87d6-5lzst" podUID="d1c47956-511e-4b8c-bb2c-35477a816fcb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Jul 28 10:41:42.275774 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:41:42.275745    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d1c47956-511e-4b8c-bb2c-35477a816fcb" path="/var/lib/kubelet/pods/d1c47956-511e-4b8c-bb2c-35477a816fcb/volumes"
Jul 28 10:41:44.625574 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:41:44.625509    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b92khft" podUID="e90d54e3-951f-45d7-ad02-b0f99bacca5a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 10:41:47.216037 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:41:47.216001    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:41:54.625877 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:41:54.625835    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b92khft" podUID="e90d54e3-951f-45d7-ad02-b0f99bacca5a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 10:41:55.342892 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:41:55.342852    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b92khft" podUID="e90d54e3-951f-45d7-ad02-b0f99bacca5a" containerName="main" containerID="cri-o://7417c681cc6dc178fe1416ccaa2d7bc1d58efa256f29f0587f6c29af737e59ac" gracePeriod=45
Jul 28 10:41:55.588068 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:41:55.588044    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b92khft"
Jul 28 10:41:55.660698 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:41:55.660600    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-drlf6\" (UniqueName: \"kubernetes.io/projected/e90d54e3-951f-45d7-ad02-b0f99bacca5a-kube-api-access-drlf6\") pod \"e90d54e3-951f-45d7-ad02-b0f99bacca5a\" (UID: \"e90d54e3-951f-45d7-ad02-b0f99bacca5a\") "
Jul 28 10:41:55.660698 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:41:55.660644    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e90d54e3-951f-45d7-ad02-b0f99bacca5a-tls-certs\") pod \"e90d54e3-951f-45d7-ad02-b0f99bacca5a\" (UID: \"e90d54e3-951f-45d7-ad02-b0f99bacca5a\") "
Jul 28 10:41:55.662791 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:41:55.662764    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e90d54e3-951f-45d7-ad02-b0f99bacca5a-kube-api-access-drlf6" (OuterVolumeSpecName: "kube-api-access-drlf6") pod "e90d54e3-951f-45d7-ad02-b0f99bacca5a" (UID: "e90d54e3-951f-45d7-ad02-b0f99bacca5a"). InnerVolumeSpecName "kube-api-access-drlf6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:41:55.662885 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:41:55.662825    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e90d54e3-951f-45d7-ad02-b0f99bacca5a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e90d54e3-951f-45d7-ad02-b0f99bacca5a" (UID: "e90d54e3-951f-45d7-ad02-b0f99bacca5a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:41:55.761811 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:41:55.761760    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drlf6\" (UniqueName: \"kubernetes.io/projected/e90d54e3-951f-45d7-ad02-b0f99bacca5a-kube-api-access-drlf6\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 10:41:55.761811 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:41:55.761804    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e90d54e3-951f-45d7-ad02-b0f99bacca5a-tls-certs\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 10:41:55.897314 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:41:55.897273    2579 generic.go:356] "Generic (PLEG): container finished" podID="e90d54e3-951f-45d7-ad02-b0f99bacca5a" containerID="7417c681cc6dc178fe1416ccaa2d7bc1d58efa256f29f0587f6c29af737e59ac" exitCode=0
Jul 28 10:41:55.897475 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:41:55.897330    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b92khft"
Jul 28 10:41:55.897475 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:41:55.897369    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b92khft" event={"ID":"e90d54e3-951f-45d7-ad02-b0f99bacca5a","Type":"ContainerDied","Data":"7417c681cc6dc178fe1416ccaa2d7bc1d58efa256f29f0587f6c29af737e59ac"}
Jul 28 10:41:55.897475 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:41:55.897427    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b92khft" event={"ID":"e90d54e3-951f-45d7-ad02-b0f99bacca5a","Type":"ContainerDied","Data":"63b42e3d8e647b02894bc19166c09e3da864477a5835e97949e60a9f20c7dae0"}
Jul 28 10:41:55.897475 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:41:55.897456    2579 scope.go:117] "RemoveContainer" containerID="7417c681cc6dc178fe1416ccaa2d7bc1d58efa256f29f0587f6c29af737e59ac"
Jul 28 10:41:55.906792 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:41:55.906769    2579 scope.go:117] "RemoveContainer" containerID="7417c681cc6dc178fe1416ccaa2d7bc1d58efa256f29f0587f6c29af737e59ac"
Jul 28 10:41:55.907064 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:41:55.907044    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7417c681cc6dc178fe1416ccaa2d7bc1d58efa256f29f0587f6c29af737e59ac\": container with ID starting with 7417c681cc6dc178fe1416ccaa2d7bc1d58efa256f29f0587f6c29af737e59ac not found: ID does not exist" containerID="7417c681cc6dc178fe1416ccaa2d7bc1d58efa256f29f0587f6c29af737e59ac"
Jul 28 10:41:55.907130 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:41:55.907077    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7417c681cc6dc178fe1416ccaa2d7bc1d58efa256f29f0587f6c29af737e59ac"} err="failed to get container status \"7417c681cc6dc178fe1416ccaa2d7bc1d58efa256f29f0587f6c29af737e59ac\": rpc error: code = NotFound desc = could not find container \"7417c681cc6dc178fe1416ccaa2d7bc1d58efa256f29f0587f6c29af737e59ac\": container with ID starting with 7417c681cc6dc178fe1416ccaa2d7bc1d58efa256f29f0587f6c29af737e59ac not found: ID does not exist"
Jul 28 10:41:55.919859 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:41:55.919795    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b92khft"]
Jul 28 10:41:55.925995 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:41:55.925973    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-c8b92khft"]
Jul 28 10:41:56.276569 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:41:56.276482    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e90d54e3-951f-45d7-ad02-b0f99bacca5a" path="/var/lib/kubelet/pods/e90d54e3-951f-45d7-ad02-b0f99bacca5a/volumes"
Jul 28 10:42:00.322669 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:42:00.322634    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:42:00.417532 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:42:00.417495    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:42:06.699671 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:42:06.699632    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:42:12.370699 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:42:12.370607    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:42:12.451074 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:42:12.451036    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:42:16.116193 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:42:16.116151    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:42:16.298655 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:42:16.298623    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schev59m4"]
Jul 28 10:42:16.298942 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:42:16.298929    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1c47956-511e-4b8c-bb2c-35477a816fcb" containerName="storage-initializer"
Jul 28 10:42:16.298942 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:42:16.298942    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1c47956-511e-4b8c-bb2c-35477a816fcb" containerName="storage-initializer"
Jul 28 10:42:16.299068 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:42:16.298953    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1c47956-511e-4b8c-bb2c-35477a816fcb" containerName="main"
Jul 28 10:42:16.299068 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:42:16.298959    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1c47956-511e-4b8c-bb2c-35477a816fcb" containerName="main"
Jul 28 10:42:16.299068 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:42:16.298968    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e90d54e3-951f-45d7-ad02-b0f99bacca5a" containerName="main"
Jul 28 10:42:16.299068 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:42:16.298973    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e90d54e3-951f-45d7-ad02-b0f99bacca5a" containerName="main"
Jul 28 10:42:16.299068 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:42:16.299030    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e90d54e3-951f-45d7-ad02-b0f99bacca5a" containerName="main"
Jul 28 10:42:16.299068 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:42:16.299040    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1c47956-511e-4b8c-bb2c-35477a816fcb" containerName="main"
Jul 28 10:42:16.303204 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:42:16.303148    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schev59m4"
Jul 28 10:42:16.305787 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:42:16.305767    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"kube-root-ca.crt\""
Jul 28 10:42:16.305877 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:42:16.305836    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"default-dockercfg-5lckb\""
Jul 28 10:42:16.305934 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:42:16.305917    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"llmisv5c7e67b6c51568d1d6d13829a9337f2a-kserve-self-signed-certs\""
Jul 28 10:42:16.305994 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:42:16.305975    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"openshift-service-ca.crt\""
Jul 28 10:42:16.306053 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:42:16.306040    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"llmisvc-router-managed-test-llm-e45d1f79-epp-sa-dockercfg-x6nvm\""
Jul 28 10:42:16.311788 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:42:16.311767    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schev59m4"]
Jul 28 10:42:16.328261 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:42:16.328239    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:42:16.440101 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:42:16.440066    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54c1e2fc-76d5-4985-8ae8-2d545f05abd8-tls-certs\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schev59m4\" (UID: \"54c1e2fc-76d5-4985-8ae8-2d545f05abd8\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schev59m4"
Jul 28 10:42:16.440285 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:42:16.440106    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlslj\" (UniqueName: \"kubernetes.io/projected/54c1e2fc-76d5-4985-8ae8-2d545f05abd8-kube-api-access-zlslj\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schev59m4\" (UID: \"54c1e2fc-76d5-4985-8ae8-2d545f05abd8\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schev59m4"
Jul 28 10:42:16.540862 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:42:16.540823    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54c1e2fc-76d5-4985-8ae8-2d545f05abd8-tls-certs\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schev59m4\" (UID: \"54c1e2fc-76d5-4985-8ae8-2d545f05abd8\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schev59m4"
Jul 28 10:42:16.541083 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:42:16.540867    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlslj\" (UniqueName: \"kubernetes.io/projected/54c1e2fc-76d5-4985-8ae8-2d545f05abd8-kube-api-access-zlslj\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schev59m4\" (UID: \"54c1e2fc-76d5-4985-8ae8-2d545f05abd8\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schev59m4"
Jul 28 10:42:16.543346 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:42:16.543324    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54c1e2fc-76d5-4985-8ae8-2d545f05abd8-tls-certs\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schev59m4\" (UID: \"54c1e2fc-76d5-4985-8ae8-2d545f05abd8\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schev59m4"
Jul 28 10:42:16.549001 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:42:16.548969    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlslj\" (UniqueName: \"kubernetes.io/projected/54c1e2fc-76d5-4985-8ae8-2d545f05abd8-kube-api-access-zlslj\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schev59m4\" (UID: \"54c1e2fc-76d5-4985-8ae8-2d545f05abd8\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schev59m4"
Jul 28 10:42:16.613255 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:42:16.613227    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schev59m4"
Jul 28 10:42:16.757450 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:42:16.757417    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schev59m4"]
Jul 28 10:42:16.758788 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:42:16.758759    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54c1e2fc_76d5_4985_8ae8_2d545f05abd8.slice/crio-e41682f610059ad0513ea1d17927362908cec19a8279ce201d1129e1e64166dc WatchSource:0}: Error finding container e41682f610059ad0513ea1d17927362908cec19a8279ce201d1129e1e64166dc: Status 404 returned error can't find the container with id e41682f610059ad0513ea1d17927362908cec19a8279ce201d1129e1e64166dc
Jul 28 10:42:17.001674 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:42:17.001628    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schev59m4" event={"ID":"54c1e2fc-76d5-4985-8ae8-2d545f05abd8","Type":"ContainerStarted","Data":"4e7f0dfa1130301b679d5b7d28763b9aef4090d637c48e00fed4c3a999af1087"}
Jul 28 10:42:17.001823 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:42:17.001688    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schev59m4" event={"ID":"54c1e2fc-76d5-4985-8ae8-2d545f05abd8","Type":"ContainerStarted","Data":"e41682f610059ad0513ea1d17927362908cec19a8279ce201d1129e1e64166dc"}
Jul 28 10:42:17.001823 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:42:17.001765    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schev59m4"
Jul 28 10:42:25.180748 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:42:25.180698    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:42:33.470955 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:42:33.470918    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:42:48.009369 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:42:48.009342    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schev59m4"
Jul 28 10:42:48.028672 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:42:48.028613    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schev59m4" podStartSLOduration=32.028595904 podStartE2EDuration="32.028595904s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:42:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:42:17.02192612 +0000 UTC m=+2071.285322451" watchObservedRunningTime="2026-07-28 10:42:48.028595904 +0000 UTC m=+2102.291992234"
Jul 28 10:42:49.322827 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:42:49.322779    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:42:49.425299 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:42:49.425261    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:42:49.441294 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:42:49.441248    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schev59m4"]
Jul 28 10:42:49.524593 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:42:49.524560    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:42:58.009730 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:42:58.009690    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schev59m4" podUID="54c1e2fc-76d5-4985-8ae8-2d545f05abd8" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 10:43:04.461700 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:43:04.461657    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schev59m4" podUID="54c1e2fc-76d5-4985-8ae8-2d545f05abd8" containerName="main" containerID="cri-o://4e7f0dfa1130301b679d5b7d28763b9aef4090d637c48e00fed4c3a999af1087" gracePeriod=45
Jul 28 10:43:04.709755 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:43:04.709725    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schev59m4"
Jul 28 10:43:04.838173 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:43:04.838138    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zlslj\" (UniqueName: \"kubernetes.io/projected/54c1e2fc-76d5-4985-8ae8-2d545f05abd8-kube-api-access-zlslj\") pod \"54c1e2fc-76d5-4985-8ae8-2d545f05abd8\" (UID: \"54c1e2fc-76d5-4985-8ae8-2d545f05abd8\") "
Jul 28 10:43:04.838378 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:43:04.838188    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54c1e2fc-76d5-4985-8ae8-2d545f05abd8-tls-certs\") pod \"54c1e2fc-76d5-4985-8ae8-2d545f05abd8\" (UID: \"54c1e2fc-76d5-4985-8ae8-2d545f05abd8\") "
Jul 28 10:43:04.840464 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:43:04.840423    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54c1e2fc-76d5-4985-8ae8-2d545f05abd8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "54c1e2fc-76d5-4985-8ae8-2d545f05abd8" (UID: "54c1e2fc-76d5-4985-8ae8-2d545f05abd8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:43:04.840464 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:43:04.840444    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54c1e2fc-76d5-4985-8ae8-2d545f05abd8-kube-api-access-zlslj" (OuterVolumeSpecName: "kube-api-access-zlslj") pod "54c1e2fc-76d5-4985-8ae8-2d545f05abd8" (UID: "54c1e2fc-76d5-4985-8ae8-2d545f05abd8"). InnerVolumeSpecName "kube-api-access-zlslj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:43:04.939009 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:43:04.938969    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zlslj\" (UniqueName: \"kubernetes.io/projected/54c1e2fc-76d5-4985-8ae8-2d545f05abd8-kube-api-access-zlslj\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 10:43:04.939009 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:43:04.939004    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54c1e2fc-76d5-4985-8ae8-2d545f05abd8-tls-certs\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 10:43:05.237123 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:43:05.237036    2579 generic.go:356] "Generic (PLEG): container finished" podID="54c1e2fc-76d5-4985-8ae8-2d545f05abd8" containerID="4e7f0dfa1130301b679d5b7d28763b9aef4090d637c48e00fed4c3a999af1087" exitCode=0
Jul 28 10:43:05.237123 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:43:05.237085    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schev59m4"
Jul 28 10:43:05.237123 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:43:05.237107    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schev59m4" event={"ID":"54c1e2fc-76d5-4985-8ae8-2d545f05abd8","Type":"ContainerDied","Data":"4e7f0dfa1130301b679d5b7d28763b9aef4090d637c48e00fed4c3a999af1087"}
Jul 28 10:43:05.237350 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:43:05.237150    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schev59m4" event={"ID":"54c1e2fc-76d5-4985-8ae8-2d545f05abd8","Type":"ContainerDied","Data":"e41682f610059ad0513ea1d17927362908cec19a8279ce201d1129e1e64166dc"}
Jul 28 10:43:05.237350 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:43:05.237172    2579 scope.go:117] "RemoveContainer" containerID="4e7f0dfa1130301b679d5b7d28763b9aef4090d637c48e00fed4c3a999af1087"
Jul 28 10:43:05.246732 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:43:05.246712    2579 scope.go:117] "RemoveContainer" containerID="4e7f0dfa1130301b679d5b7d28763b9aef4090d637c48e00fed4c3a999af1087"
Jul 28 10:43:05.246979 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:43:05.246956    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e7f0dfa1130301b679d5b7d28763b9aef4090d637c48e00fed4c3a999af1087\": container with ID starting with 4e7f0dfa1130301b679d5b7d28763b9aef4090d637c48e00fed4c3a999af1087 not found: ID does not exist" containerID="4e7f0dfa1130301b679d5b7d28763b9aef4090d637c48e00fed4c3a999af1087"
Jul 28 10:43:05.247021 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:43:05.246989    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e7f0dfa1130301b679d5b7d28763b9aef4090d637c48e00fed4c3a999af1087"} err="failed to get container status \"4e7f0dfa1130301b679d5b7d28763b9aef4090d637c48e00fed4c3a999af1087\": rpc error: code = NotFound desc = could not find container \"4e7f0dfa1130301b679d5b7d28763b9aef4090d637c48e00fed4c3a999af1087\": container with ID starting with 4e7f0dfa1130301b679d5b7d28763b9aef4090d637c48e00fed4c3a999af1087 not found: ID does not exist"
Jul 28 10:43:05.260452 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:43:05.260419    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schev59m4"]
Jul 28 10:43:05.263100 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:43:05.263078    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schev59m4"]
Jul 28 10:43:06.276172 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:43:06.276139    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="54c1e2fc-76d5-4985-8ae8-2d545f05abd8" path="/var/lib/kubelet/pods/54c1e2fc-76d5-4985-8ae8-2d545f05abd8/volumes"
Jul 28 10:43:09.257400 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:43:09.257356    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:43:09.352461 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:43:09.352422    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:43:16.088521 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:43:16.088473    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:43:20.689053 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:43:20.689018    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:43:20.920233 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:43:20.920197    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetthmp"]
Jul 28 10:43:20.920549 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:43:20.920524    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54c1e2fc-76d5-4985-8ae8-2d545f05abd8" containerName="main"
Jul 28 10:43:20.920653 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:43:20.920552    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="54c1e2fc-76d5-4985-8ae8-2d545f05abd8" containerName="main"
Jul 28 10:43:20.920653 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:43:20.920620    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="54c1e2fc-76d5-4985-8ae8-2d545f05abd8" containerName="main"
Jul 28 10:43:20.923717 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:43:20.923695    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetthmp"
Jul 28 10:43:20.925645 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:43:20.925623    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"kube-root-ca.crt\""
Jul 28 10:43:20.926191 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:43:20.926177    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"openshift-service-ca.crt\""
Jul 28 10:43:20.926338 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:43:20.926317    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvc-router-managed-test-llm-5b1e8f15-epp-sa-dockercfg-wlvvv\""
Jul 28 10:43:20.926396 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:43:20.926317    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs\""
Jul 28 10:43:20.926396 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:43:20.926376    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"default-dockercfg-m2p8x\""
Jul 28 10:43:20.933449 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:43:20.933424    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetthmp"]
Jul 28 10:43:20.946352 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:43:20.946278    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:43:21.072736 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:43:21.072702    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74d6ea3a-aad8-42ad-9ab6-498cfee55ad6-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetthmp\" (UID: \"74d6ea3a-aad8-42ad-9ab6-498cfee55ad6\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetthmp"
Jul 28 10:43:21.072930 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:43:21.072789    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mbnh\" (UniqueName: \"kubernetes.io/projected/74d6ea3a-aad8-42ad-9ab6-498cfee55ad6-kube-api-access-5mbnh\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetthmp\" (UID: \"74d6ea3a-aad8-42ad-9ab6-498cfee55ad6\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetthmp"
Jul 28 10:43:21.174000 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:43:21.173960    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mbnh\" (UniqueName: \"kubernetes.io/projected/74d6ea3a-aad8-42ad-9ab6-498cfee55ad6-kube-api-access-5mbnh\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetthmp\" (UID: \"74d6ea3a-aad8-42ad-9ab6-498cfee55ad6\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetthmp"
Jul 28 10:43:21.174224 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:43:21.174033    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74d6ea3a-aad8-42ad-9ab6-498cfee55ad6-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetthmp\" (UID: \"74d6ea3a-aad8-42ad-9ab6-498cfee55ad6\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetthmp"
Jul 28 10:43:21.176757 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:43:21.176731    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74d6ea3a-aad8-42ad-9ab6-498cfee55ad6-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetthmp\" (UID: \"74d6ea3a-aad8-42ad-9ab6-498cfee55ad6\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetthmp"
Jul 28 10:43:21.182821 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:43:21.182797    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mbnh\" (UniqueName: \"kubernetes.io/projected/74d6ea3a-aad8-42ad-9ab6-498cfee55ad6-kube-api-access-5mbnh\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetthmp\" (UID: \"74d6ea3a-aad8-42ad-9ab6-498cfee55ad6\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetthmp"
Jul 28 10:43:21.235687 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:43:21.235597    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetthmp"
Jul 28 10:43:21.372488 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:43:21.372466    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetthmp"]
Jul 28 10:43:21.374379 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:43:21.374353    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74d6ea3a_aad8_42ad_9ab6_498cfee55ad6.slice/crio-94ac3409998f08e27b9a58ea457127530768cf5119f03f4e7842d033f090cac2 WatchSource:0}: Error finding container 94ac3409998f08e27b9a58ea457127530768cf5119f03f4e7842d033f090cac2: Status 404 returned error can't find the container with id 94ac3409998f08e27b9a58ea457127530768cf5119f03f4e7842d033f090cac2
Jul 28 10:43:22.320400 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:43:22.320357    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetthmp" event={"ID":"74d6ea3a-aad8-42ad-9ab6-498cfee55ad6","Type":"ContainerStarted","Data":"96029f946d12d41c76761d239b25ce1b0f8880da4ba0d57ed085271a92f2ae86"}
Jul 28 10:43:22.320852 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:43:22.320413    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetthmp" event={"ID":"74d6ea3a-aad8-42ad-9ab6-498cfee55ad6","Type":"ContainerStarted","Data":"94ac3409998f08e27b9a58ea457127530768cf5119f03f4e7842d033f090cac2"}
Jul 28 10:43:22.320852 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:43:22.320502    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetthmp"
Jul 28 10:43:22.339873 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:43:22.339798    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetthmp" podStartSLOduration=2.33977881 podStartE2EDuration="2.33977881s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:43:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:43:22.337783808 +0000 UTC m=+2136.601180136" watchObservedRunningTime="2026-07-28 10:43:22.33977881 +0000 UTC m=+2136.603175140"
Jul 28 10:43:22.368344 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:43:22.368311    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:43:22.431371 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:43:22.431340    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:43:33.690354 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:43:33.690314    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:43:41.015812 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:43:41.015703    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:43:53.327701 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:43:53.327668    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetthmp"
Jul 28 10:44:01.372470 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:44:01.372431    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954bcs4"]
Jul 28 10:44:01.375970 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:44:01.375948    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954bcs4"
Jul 28 10:44:01.377880 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:44:01.377858    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvb11a9c9832b99b016bc8f8e0ea095712-kserve-self-signed-certs\""
Jul 28 10:44:01.387352 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:44:01.387326    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954bcs4"]
Jul 28 10:44:01.398251 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:44:01.398219    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:44:01.496773 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:44:01.496742    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhtnx\" (UniqueName: \"kubernetes.io/projected/d0d3c7f0-fda0-4625-ad38-58347c0ddff4-kube-api-access-zhtnx\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954bcs4\" (UID: \"d0d3c7f0-fda0-4625-ad38-58347c0ddff4\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954bcs4"
Jul 28 10:44:01.496963 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:44:01.496786    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0d3c7f0-fda0-4625-ad38-58347c0ddff4-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954bcs4\" (UID: \"d0d3c7f0-fda0-4625-ad38-58347c0ddff4\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954bcs4"
Jul 28 10:44:01.496963 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:44:01.496816    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0d3c7f0-fda0-4625-ad38-58347c0ddff4-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954bcs4\" (UID: \"d0d3c7f0-fda0-4625-ad38-58347c0ddff4\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954bcs4"
Jul 28 10:44:01.496963 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:44:01.496885    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0d3c7f0-fda0-4625-ad38-58347c0ddff4-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954bcs4\" (UID: \"d0d3c7f0-fda0-4625-ad38-58347c0ddff4\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954bcs4"
Jul 28 10:44:01.496963 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:44:01.496930    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0d3c7f0-fda0-4625-ad38-58347c0ddff4-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954bcs4\" (UID: \"d0d3c7f0-fda0-4625-ad38-58347c0ddff4\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954bcs4"
Jul 28 10:44:01.496963 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:44:01.496949    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0d3c7f0-fda0-4625-ad38-58347c0ddff4-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954bcs4\" (UID: \"d0d3c7f0-fda0-4625-ad38-58347c0ddff4\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954bcs4"
Jul 28 10:44:01.598092 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:44:01.598045    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhtnx\" (UniqueName: \"kubernetes.io/projected/d0d3c7f0-fda0-4625-ad38-58347c0ddff4-kube-api-access-zhtnx\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954bcs4\" (UID: \"d0d3c7f0-fda0-4625-ad38-58347c0ddff4\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954bcs4"
Jul 28 10:44:01.598092 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:44:01.598107    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0d3c7f0-fda0-4625-ad38-58347c0ddff4-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954bcs4\" (UID: \"d0d3c7f0-fda0-4625-ad38-58347c0ddff4\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954bcs4"
Jul 28 10:44:01.598374 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:44:01.598133    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0d3c7f0-fda0-4625-ad38-58347c0ddff4-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954bcs4\" (UID: \"d0d3c7f0-fda0-4625-ad38-58347c0ddff4\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954bcs4"
Jul 28 10:44:01.598374 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:44:01.598159    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0d3c7f0-fda0-4625-ad38-58347c0ddff4-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954bcs4\" (UID: \"d0d3c7f0-fda0-4625-ad38-58347c0ddff4\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954bcs4"
Jul 28 10:44:01.598374 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:44:01.598189    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0d3c7f0-fda0-4625-ad38-58347c0ddff4-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954bcs4\" (UID: \"d0d3c7f0-fda0-4625-ad38-58347c0ddff4\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954bcs4"
Jul 28 10:44:01.598374 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:44:01.598205    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0d3c7f0-fda0-4625-ad38-58347c0ddff4-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954bcs4\" (UID: \"d0d3c7f0-fda0-4625-ad38-58347c0ddff4\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954bcs4"
Jul 28 10:44:01.598717 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:44:01.598677    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0d3c7f0-fda0-4625-ad38-58347c0ddff4-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954bcs4\" (UID: \"d0d3c7f0-fda0-4625-ad38-58347c0ddff4\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954bcs4"
Jul 28 10:44:01.598717 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:44:01.598701    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0d3c7f0-fda0-4625-ad38-58347c0ddff4-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954bcs4\" (UID: \"d0d3c7f0-fda0-4625-ad38-58347c0ddff4\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954bcs4"
Jul 28 10:44:01.598865 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:44:01.598757    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0d3c7f0-fda0-4625-ad38-58347c0ddff4-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954bcs4\" (UID: \"d0d3c7f0-fda0-4625-ad38-58347c0ddff4\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954bcs4"
Jul 28 10:44:01.600483 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:44:01.600464    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0d3c7f0-fda0-4625-ad38-58347c0ddff4-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954bcs4\" (UID: \"d0d3c7f0-fda0-4625-ad38-58347c0ddff4\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954bcs4"
Jul 28 10:44:01.600657 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:44:01.600640    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0d3c7f0-fda0-4625-ad38-58347c0ddff4-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954bcs4\" (UID: \"d0d3c7f0-fda0-4625-ad38-58347c0ddff4\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954bcs4"
Jul 28 10:44:01.609728 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:44:01.609684    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhtnx\" (UniqueName: \"kubernetes.io/projected/d0d3c7f0-fda0-4625-ad38-58347c0ddff4-kube-api-access-zhtnx\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954bcs4\" (UID: \"d0d3c7f0-fda0-4625-ad38-58347c0ddff4\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954bcs4"
Jul 28 10:44:01.657447 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:44:01.657352    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:44:01.686461 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:44:01.686430    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954bcs4"
Jul 28 10:44:01.825152 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:44:01.825126    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954bcs4"]
Jul 28 10:44:01.826924 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:44:01.826898    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0d3c7f0_fda0_4625_ad38_58347c0ddff4.slice/crio-dbd1d7ed87f8458ea5732da10b6ed5eb2d24a159e7d12a47573388d786c8c9f4 WatchSource:0}: Error finding container dbd1d7ed87f8458ea5732da10b6ed5eb2d24a159e7d12a47573388d786c8c9f4: Status 404 returned error can't find the container with id dbd1d7ed87f8458ea5732da10b6ed5eb2d24a159e7d12a47573388d786c8c9f4
Jul 28 10:44:02.521914 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:44:02.521869    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954bcs4" event={"ID":"d0d3c7f0-fda0-4625-ad38-58347c0ddff4","Type":"ContainerStarted","Data":"50c8faa47b74b3ca758a215975c1cebdddb4c8c15a0326e27d0e3bbaf3ea62a1"}
Jul 28 10:44:02.522373 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:44:02.521966    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954bcs4" event={"ID":"d0d3c7f0-fda0-4625-ad38-58347c0ddff4","Type":"ContainerStarted","Data":"dbd1d7ed87f8458ea5732da10b6ed5eb2d24a159e7d12a47573388d786c8c9f4"}
Jul 28 10:44:02.548445 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:44:02.548389    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954bcs4" podStartSLOduration=1.548371567 podStartE2EDuration="1.548371567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:44:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:44:02.546552704 +0000 UTC m=+2176.809949024" watchObservedRunningTime="2026-07-28 10:44:02.548371567 +0000 UTC m=+2176.811767897"
Jul 28 10:44:02.686742 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:44:02.686698    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954bcs4"
Jul 28 10:44:11.686632 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:44:11.686576    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954bcs4"
Jul 28 10:44:11.699213 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:44:11.699184    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954bcs4"
Jul 28 10:44:12.587850 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:44:12.587816    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954bcs4"
Jul 28 10:44:15.591909 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:44:15.591871    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:44:34.791570 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:44:34.791517    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954bcs4"]
Jul 28 10:44:34.792519 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:44:34.792488    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:44:34.801422 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:44:34.801385    2579 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 28 10:44:34.801422 ip-10-0-141-50 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 10:44:34.801422 ip-10-0-141-50 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Jul 28 10:44:34.801422 ip-10-0-141-50 kubenswrapper[2579]:  > containerID="50c8faa47b74b3ca758a215975c1cebdddb4c8c15a0326e27d0e3bbaf3ea62a1" cmd=["/bin/sleep","15"]
Jul 28 10:44:34.801741 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:44:34.801443    2579 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954bcs4" message="" err=<
Jul 28 10:44:34.801741 ip-10-0-141-50 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 10:44:34.801741 ip-10-0-141-50 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Jul 28 10:44:34.801741 ip-10-0-141-50 kubenswrapper[2579]:  >
Jul 28 10:44:34.801741 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:44:34.801459    2579 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 28 10:44:34.801741 ip-10-0-141-50 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 10:44:34.801741 ip-10-0-141-50 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Jul 28 10:44:34.801741 ip-10-0-141-50 kubenswrapper[2579]:  > pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954bcs4" podUID="d0d3c7f0-fda0-4625-ad38-58347c0ddff4" containerName="main" containerID="cri-o://50c8faa47b74b3ca758a215975c1cebdddb4c8c15a0326e27d0e3bbaf3ea62a1"
Jul 28 10:44:34.801741 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:44:34.801516    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954bcs4" podUID="d0d3c7f0-fda0-4625-ad38-58347c0ddff4" containerName="main" containerID="cri-o://50c8faa47b74b3ca758a215975c1cebdddb4c8c15a0326e27d0e3bbaf3ea62a1" gracePeriod=60
Jul 28 10:44:34.851311 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:44:34.851280    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:44:35.149218 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:44:35.149186    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954bcs4"
Jul 28 10:44:35.198318 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:44:35.198285    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0d3c7f0-fda0-4625-ad38-58347c0ddff4-tmp-dir\") pod \"d0d3c7f0-fda0-4625-ad38-58347c0ddff4\" (UID: \"d0d3c7f0-fda0-4625-ad38-58347c0ddff4\") "
Jul 28 10:44:35.198509 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:44:35.198326    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0d3c7f0-fda0-4625-ad38-58347c0ddff4-dshm\") pod \"d0d3c7f0-fda0-4625-ad38-58347c0ddff4\" (UID: \"d0d3c7f0-fda0-4625-ad38-58347c0ddff4\") "
Jul 28 10:44:35.198509 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:44:35.198347    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0d3c7f0-fda0-4625-ad38-58347c0ddff4-tls-certs\") pod \"d0d3c7f0-fda0-4625-ad38-58347c0ddff4\" (UID: \"d0d3c7f0-fda0-4625-ad38-58347c0ddff4\") "
Jul 28 10:44:35.198509 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:44:35.198388    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0d3c7f0-fda0-4625-ad38-58347c0ddff4-model-cache\") pod \"d0d3c7f0-fda0-4625-ad38-58347c0ddff4\" (UID: \"d0d3c7f0-fda0-4625-ad38-58347c0ddff4\") "
Jul 28 10:44:35.198509 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:44:35.198410    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0d3c7f0-fda0-4625-ad38-58347c0ddff4-home\") pod \"d0d3c7f0-fda0-4625-ad38-58347c0ddff4\" (UID: \"d0d3c7f0-fda0-4625-ad38-58347c0ddff4\") "
Jul 28 10:44:35.198509 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:44:35.198435    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zhtnx\" (UniqueName: \"kubernetes.io/projected/d0d3c7f0-fda0-4625-ad38-58347c0ddff4-kube-api-access-zhtnx\") pod \"d0d3c7f0-fda0-4625-ad38-58347c0ddff4\" (UID: \"d0d3c7f0-fda0-4625-ad38-58347c0ddff4\") "
Jul 28 10:44:35.198829 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:44:35.198651    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0d3c7f0-fda0-4625-ad38-58347c0ddff4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d0d3c7f0-fda0-4625-ad38-58347c0ddff4" (UID: "d0d3c7f0-fda0-4625-ad38-58347c0ddff4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:44:35.198829 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:44:35.198791    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0d3c7f0-fda0-4625-ad38-58347c0ddff4-home" (OuterVolumeSpecName: "home") pod "d0d3c7f0-fda0-4625-ad38-58347c0ddff4" (UID: "d0d3c7f0-fda0-4625-ad38-58347c0ddff4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:44:35.198829 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:44:35.198812    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0d3c7f0-fda0-4625-ad38-58347c0ddff4-model-cache" (OuterVolumeSpecName: "model-cache") pod "d0d3c7f0-fda0-4625-ad38-58347c0ddff4" (UID: "d0d3c7f0-fda0-4625-ad38-58347c0ddff4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:44:35.200705 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:44:35.200678    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0d3c7f0-fda0-4625-ad38-58347c0ddff4-kube-api-access-zhtnx" (OuterVolumeSpecName: "kube-api-access-zhtnx") pod "d0d3c7f0-fda0-4625-ad38-58347c0ddff4" (UID: "d0d3c7f0-fda0-4625-ad38-58347c0ddff4"). InnerVolumeSpecName "kube-api-access-zhtnx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:44:35.200831 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:44:35.200703    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0d3c7f0-fda0-4625-ad38-58347c0ddff4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d0d3c7f0-fda0-4625-ad38-58347c0ddff4" (UID: "d0d3c7f0-fda0-4625-ad38-58347c0ddff4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:44:35.200831 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:44:35.200686    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0d3c7f0-fda0-4625-ad38-58347c0ddff4-dshm" (OuterVolumeSpecName: "dshm") pod "d0d3c7f0-fda0-4625-ad38-58347c0ddff4" (UID: "d0d3c7f0-fda0-4625-ad38-58347c0ddff4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:44:35.299155 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:44:35.299116    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0d3c7f0-fda0-4625-ad38-58347c0ddff4-model-cache\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 10:44:35.299155 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:44:35.299152    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0d3c7f0-fda0-4625-ad38-58347c0ddff4-home\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 10:44:35.299155 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:44:35.299163    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhtnx\" (UniqueName: \"kubernetes.io/projected/d0d3c7f0-fda0-4625-ad38-58347c0ddff4-kube-api-access-zhtnx\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 10:44:35.299440 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:44:35.299175    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0d3c7f0-fda0-4625-ad38-58347c0ddff4-tmp-dir\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 10:44:35.299440 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:44:35.299184    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0d3c7f0-fda0-4625-ad38-58347c0ddff4-dshm\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 10:44:35.299440 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:44:35.299192    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0d3c7f0-fda0-4625-ad38-58347c0ddff4-tls-certs\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 10:44:35.691315 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:44:35.691279    2579 generic.go:356] "Generic (PLEG): container finished" podID="d0d3c7f0-fda0-4625-ad38-58347c0ddff4" containerID="50c8faa47b74b3ca758a215975c1cebdddb4c8c15a0326e27d0e3bbaf3ea62a1" exitCode=0
Jul 28 10:44:35.691491 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:44:35.691325    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954bcs4"
Jul 28 10:44:35.691491 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:44:35.691366    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954bcs4" event={"ID":"d0d3c7f0-fda0-4625-ad38-58347c0ddff4","Type":"ContainerDied","Data":"50c8faa47b74b3ca758a215975c1cebdddb4c8c15a0326e27d0e3bbaf3ea62a1"}
Jul 28 10:44:35.691491 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:44:35.691421    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954bcs4" event={"ID":"d0d3c7f0-fda0-4625-ad38-58347c0ddff4","Type":"ContainerDied","Data":"dbd1d7ed87f8458ea5732da10b6ed5eb2d24a159e7d12a47573388d786c8c9f4"}
Jul 28 10:44:35.691491 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:44:35.691444    2579 scope.go:117] "RemoveContainer" containerID="50c8faa47b74b3ca758a215975c1cebdddb4c8c15a0326e27d0e3bbaf3ea62a1"
Jul 28 10:44:35.700549 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:44:35.700519    2579 scope.go:117] "RemoveContainer" containerID="50c8faa47b74b3ca758a215975c1cebdddb4c8c15a0326e27d0e3bbaf3ea62a1"
Jul 28 10:44:35.700815 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:44:35.700795    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50c8faa47b74b3ca758a215975c1cebdddb4c8c15a0326e27d0e3bbaf3ea62a1\": container with ID starting with 50c8faa47b74b3ca758a215975c1cebdddb4c8c15a0326e27d0e3bbaf3ea62a1 not found: ID does not exist" containerID="50c8faa47b74b3ca758a215975c1cebdddb4c8c15a0326e27d0e3bbaf3ea62a1"
Jul 28 10:44:35.700881 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:44:35.700828    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50c8faa47b74b3ca758a215975c1cebdddb4c8c15a0326e27d0e3bbaf3ea62a1"} err="failed to get container status \"50c8faa47b74b3ca758a215975c1cebdddb4c8c15a0326e27d0e3bbaf3ea62a1\": rpc error: code = NotFound desc = could not find container \"50c8faa47b74b3ca758a215975c1cebdddb4c8c15a0326e27d0e3bbaf3ea62a1\": container with ID starting with 50c8faa47b74b3ca758a215975c1cebdddb4c8c15a0326e27d0e3bbaf3ea62a1 not found: ID does not exist"
Jul 28 10:44:35.716050 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:44:35.716021    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954bcs4"]
Jul 28 10:44:35.719448 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:44:35.719422    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954bcs4"]
Jul 28 10:44:36.277584 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:44:36.277556    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d0d3c7f0-fda0-4625-ad38-58347c0ddff4" path="/var/lib/kubelet/pods/d0d3c7f0-fda0-4625-ad38-58347c0ddff4/volumes"
Jul 28 10:44:46.422667 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:44:46.422633    2579 scope.go:117] "RemoveContainer" containerID="85f889e8925da2d2e421040e5d83de8c06b30660c2aa9ce3478f1849ec9fed76"
Jul 28 10:44:54.523891 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:44:54.523857    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetthmp"]
Jul 28 10:44:54.530735 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:44:54.530704    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:44:54.595772 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:44:54.595742    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:45:03.326979 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:45:03.326941    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetthmp" podUID="74d6ea3a-aad8-42ad-9ab6-498cfee55ad6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 10:45:09.809020 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:45:09.808922    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetthmp" podUID="74d6ea3a-aad8-42ad-9ab6-498cfee55ad6" containerName="main" containerID="cri-o://96029f946d12d41c76761d239b25ce1b0f8880da4ba0d57ed085271a92f2ae86" gracePeriod=45
Jul 28 10:45:10.065661 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:45:10.065600    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetthmp"
Jul 28 10:45:10.180952 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:45:10.180915    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74d6ea3a-aad8-42ad-9ab6-498cfee55ad6-tls-certs\") pod \"74d6ea3a-aad8-42ad-9ab6-498cfee55ad6\" (UID: \"74d6ea3a-aad8-42ad-9ab6-498cfee55ad6\") "
Jul 28 10:45:10.181139 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:45:10.180981    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5mbnh\" (UniqueName: \"kubernetes.io/projected/74d6ea3a-aad8-42ad-9ab6-498cfee55ad6-kube-api-access-5mbnh\") pod \"74d6ea3a-aad8-42ad-9ab6-498cfee55ad6\" (UID: \"74d6ea3a-aad8-42ad-9ab6-498cfee55ad6\") "
Jul 28 10:45:10.183138 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:45:10.183118    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74d6ea3a-aad8-42ad-9ab6-498cfee55ad6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "74d6ea3a-aad8-42ad-9ab6-498cfee55ad6" (UID: "74d6ea3a-aad8-42ad-9ab6-498cfee55ad6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:45:10.183212 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:45:10.183118    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74d6ea3a-aad8-42ad-9ab6-498cfee55ad6-kube-api-access-5mbnh" (OuterVolumeSpecName: "kube-api-access-5mbnh") pod "74d6ea3a-aad8-42ad-9ab6-498cfee55ad6" (UID: "74d6ea3a-aad8-42ad-9ab6-498cfee55ad6"). InnerVolumeSpecName "kube-api-access-5mbnh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:45:10.281462 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:45:10.281433    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74d6ea3a-aad8-42ad-9ab6-498cfee55ad6-tls-certs\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 10:45:10.281462 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:45:10.281456    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mbnh\" (UniqueName: \"kubernetes.io/projected/74d6ea3a-aad8-42ad-9ab6-498cfee55ad6-kube-api-access-5mbnh\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 10:45:10.858466 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:45:10.858430    2579 generic.go:356] "Generic (PLEG): container finished" podID="74d6ea3a-aad8-42ad-9ab6-498cfee55ad6" containerID="96029f946d12d41c76761d239b25ce1b0f8880da4ba0d57ed085271a92f2ae86" exitCode=0
Jul 28 10:45:10.858901 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:45:10.858481    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetthmp"
Jul 28 10:45:10.858901 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:45:10.858518    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetthmp" event={"ID":"74d6ea3a-aad8-42ad-9ab6-498cfee55ad6","Type":"ContainerDied","Data":"96029f946d12d41c76761d239b25ce1b0f8880da4ba0d57ed085271a92f2ae86"}
Jul 28 10:45:10.858901 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:45:10.858595    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetthmp" event={"ID":"74d6ea3a-aad8-42ad-9ab6-498cfee55ad6","Type":"ContainerDied","Data":"94ac3409998f08e27b9a58ea457127530768cf5119f03f4e7842d033f090cac2"}
Jul 28 10:45:10.858901 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:45:10.858617    2579 scope.go:117] "RemoveContainer" containerID="96029f946d12d41c76761d239b25ce1b0f8880da4ba0d57ed085271a92f2ae86"
Jul 28 10:45:10.866922 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:45:10.866903    2579 scope.go:117] "RemoveContainer" containerID="96029f946d12d41c76761d239b25ce1b0f8880da4ba0d57ed085271a92f2ae86"
Jul 28 10:45:10.867186 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:45:10.867164    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96029f946d12d41c76761d239b25ce1b0f8880da4ba0d57ed085271a92f2ae86\": container with ID starting with 96029f946d12d41c76761d239b25ce1b0f8880da4ba0d57ed085271a92f2ae86 not found: ID does not exist" containerID="96029f946d12d41c76761d239b25ce1b0f8880da4ba0d57ed085271a92f2ae86"
Jul 28 10:45:10.867281 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:45:10.867191    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96029f946d12d41c76761d239b25ce1b0f8880da4ba0d57ed085271a92f2ae86"} err="failed to get container status \"96029f946d12d41c76761d239b25ce1b0f8880da4ba0d57ed085271a92f2ae86\": rpc error: code = NotFound desc = could not find container \"96029f946d12d41c76761d239b25ce1b0f8880da4ba0d57ed085271a92f2ae86\": container with ID starting with 96029f946d12d41c76761d239b25ce1b0f8880da4ba0d57ed085271a92f2ae86 not found: ID does not exist"
Jul 28 10:45:10.873733 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:45:10.873712    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetthmp"]
Jul 28 10:45:10.883930 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:45:10.883897    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetthmp"]
Jul 28 10:45:12.275691 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:45:12.275661    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="74d6ea3a-aad8-42ad-9ab6-498cfee55ad6" path="/var/lib/kubelet/pods/74d6ea3a-aad8-42ad-9ab6-498cfee55ad6/volumes"
Jul 28 10:45:14.550683 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:45:14.550641    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:45:14.681716 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:45:14.681679    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:45:14.782667 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:45:14.782608    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:45:21.101081 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:45:21.101040    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:45:26.652905 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:45:26.652873    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:45:26.756036 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:45:26.755922    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:45:33.022352 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:45:33.022317    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:45:33.302919 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:45:33.302884    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scherqhcm"]
Jul 28 10:45:33.303243 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:45:33.303228    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74d6ea3a-aad8-42ad-9ab6-498cfee55ad6" containerName="main"
Jul 28 10:45:33.303289 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:45:33.303244    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="74d6ea3a-aad8-42ad-9ab6-498cfee55ad6" containerName="main"
Jul 28 10:45:33.303289 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:45:33.303263    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0d3c7f0-fda0-4625-ad38-58347c0ddff4" containerName="main"
Jul 28 10:45:33.303289 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:45:33.303269    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0d3c7f0-fda0-4625-ad38-58347c0ddff4" containerName="main"
Jul 28 10:45:33.303384 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:45:33.303330    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0d3c7f0-fda0-4625-ad38-58347c0ddff4" containerName="main"
Jul 28 10:45:33.303384 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:45:33.303341    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="74d6ea3a-aad8-42ad-9ab6-498cfee55ad6" containerName="main"
Jul 28 10:45:33.306335 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:45:33.306308    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scherqhcm"
Jul 28 10:45:33.309253 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:45:33.309225    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"kube-root-ca.crt\""
Jul 28 10:45:33.309253 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:45:33.309240    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvc-router-managed-test-llm-4b931143-epp-sa-dockercfg-4pmwh\""
Jul 28 10:45:33.309437 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:45:33.309245    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"openshift-service-ca.crt\""
Jul 28 10:45:33.309437 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:45:33.309234    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"default-dockercfg-k49t4\""
Jul 28 10:45:33.309437 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:45:33.309323    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvca2d2d7d499abb359505529ebe02c136-kserve-self-signed-certs\""
Jul 28 10:45:33.322124 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:45:33.322098    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scherqhcm"]
Jul 28 10:45:33.472817 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:45:33.472784    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nxdr\" (UniqueName: \"kubernetes.io/projected/6bc286ba-bd26-4316-8fc9-ae99f09c3cc6-kube-api-access-2nxdr\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scherqhcm\" (UID: \"6bc286ba-bd26-4316-8fc9-ae99f09c3cc6\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scherqhcm"
Jul 28 10:45:33.473062 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:45:33.472874    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6bc286ba-bd26-4316-8fc9-ae99f09c3cc6-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scherqhcm\" (UID: \"6bc286ba-bd26-4316-8fc9-ae99f09c3cc6\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scherqhcm"
Jul 28 10:45:33.574016 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:45:33.573919    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nxdr\" (UniqueName: \"kubernetes.io/projected/6bc286ba-bd26-4316-8fc9-ae99f09c3cc6-kube-api-access-2nxdr\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scherqhcm\" (UID: \"6bc286ba-bd26-4316-8fc9-ae99f09c3cc6\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scherqhcm"
Jul 28 10:45:33.574016 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:45:33.573984    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6bc286ba-bd26-4316-8fc9-ae99f09c3cc6-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scherqhcm\" (UID: \"6bc286ba-bd26-4316-8fc9-ae99f09c3cc6\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scherqhcm"
Jul 28 10:45:33.576471 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:45:33.576445    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6bc286ba-bd26-4316-8fc9-ae99f09c3cc6-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scherqhcm\" (UID: \"6bc286ba-bd26-4316-8fc9-ae99f09c3cc6\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scherqhcm"
Jul 28 10:45:33.583962 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:45:33.583941    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nxdr\" (UniqueName: \"kubernetes.io/projected/6bc286ba-bd26-4316-8fc9-ae99f09c3cc6-kube-api-access-2nxdr\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scherqhcm\" (UID: \"6bc286ba-bd26-4316-8fc9-ae99f09c3cc6\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scherqhcm"
Jul 28 10:45:33.618916 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:45:33.618883    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scherqhcm"
Jul 28 10:45:33.754618 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:45:33.754588    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scherqhcm"]
Jul 28 10:45:33.756840 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:45:33.756813    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bc286ba_bd26_4316_8fc9_ae99f09c3cc6.slice/crio-2435a8ad62ccd0cd5be9e62a526bd803847222fe93342de25f24f5da2cead953 WatchSource:0}: Error finding container 2435a8ad62ccd0cd5be9e62a526bd803847222fe93342de25f24f5da2cead953: Status 404 returned error can't find the container with id 2435a8ad62ccd0cd5be9e62a526bd803847222fe93342de25f24f5da2cead953
Jul 28 10:45:33.974815 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:45:33.974703    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scherqhcm" event={"ID":"6bc286ba-bd26-4316-8fc9-ae99f09c3cc6","Type":"ContainerStarted","Data":"f53cc463e9d23f44bb6c90dc5aef0347a3b303e0fd5124c39eb4c90207293bab"}
Jul 28 10:45:33.974815 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:45:33.974760    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scherqhcm" event={"ID":"6bc286ba-bd26-4316-8fc9-ae99f09c3cc6","Type":"ContainerStarted","Data":"2435a8ad62ccd0cd5be9e62a526bd803847222fe93342de25f24f5da2cead953"}
Jul 28 10:45:33.975074 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:45:33.974843    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scherqhcm"
Jul 28 10:45:33.991912 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:45:33.991849    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scherqhcm" podStartSLOduration=0.991827003 podStartE2EDuration="991.827003ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:45:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:45:33.990656318 +0000 UTC m=+2268.254052869" watchObservedRunningTime="2026-07-28 10:45:33.991827003 +0000 UTC m=+2268.255223331"
Jul 28 10:45:34.864469 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:45:34.864434    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:45:41.450016 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:45:41.449977    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:46:04.981181 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:46:04.981150    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scherqhcm"
Jul 28 10:46:10.679638 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:46:10.679599    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-fv6wt"]
Jul 28 10:46:10.683171 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:46:10.683149    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-fv6wt"
Jul 28 10:46:10.685139 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:46:10.685117    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisv08544b88a8d963ffd553cc1f3ed82d16-kserve-self-signed-certs\""
Jul 28 10:46:10.692584 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:46:10.692561    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-fv6wt"]
Jul 28 10:46:10.771341 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:46:10.771304    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:46:10.786347 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:46:10.786321    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrn4z\" (UniqueName: \"kubernetes.io/projected/753197e9-c368-4498-8448-6c87b354e6f7-kube-api-access-hrn4z\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-fv6wt\" (UID: \"753197e9-c368-4498-8448-6c87b354e6f7\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-fv6wt"
Jul 28 10:46:10.786515 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:46:10.786354    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/753197e9-c368-4498-8448-6c87b354e6f7-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-fv6wt\" (UID: \"753197e9-c368-4498-8448-6c87b354e6f7\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-fv6wt"
Jul 28 10:46:10.786515 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:46:10.786376    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/753197e9-c368-4498-8448-6c87b354e6f7-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-fv6wt\" (UID: \"753197e9-c368-4498-8448-6c87b354e6f7\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-fv6wt"
Jul 28 10:46:10.786515 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:46:10.786397    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/753197e9-c368-4498-8448-6c87b354e6f7-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-fv6wt\" (UID: \"753197e9-c368-4498-8448-6c87b354e6f7\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-fv6wt"
Jul 28 10:46:10.786515 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:46:10.786495    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/753197e9-c368-4498-8448-6c87b354e6f7-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-fv6wt\" (UID: \"753197e9-c368-4498-8448-6c87b354e6f7\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-fv6wt"
Jul 28 10:46:10.786694 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:46:10.786528    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/753197e9-c368-4498-8448-6c87b354e6f7-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-fv6wt\" (UID: \"753197e9-c368-4498-8448-6c87b354e6f7\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-fv6wt"
Jul 28 10:46:10.887103 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:46:10.887068    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/753197e9-c368-4498-8448-6c87b354e6f7-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-fv6wt\" (UID: \"753197e9-c368-4498-8448-6c87b354e6f7\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-fv6wt"
Jul 28 10:46:10.887345 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:46:10.887142    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrn4z\" (UniqueName: \"kubernetes.io/projected/753197e9-c368-4498-8448-6c87b354e6f7-kube-api-access-hrn4z\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-fv6wt\" (UID: \"753197e9-c368-4498-8448-6c87b354e6f7\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-fv6wt"
Jul 28 10:46:10.887345 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:46:10.887162    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/753197e9-c368-4498-8448-6c87b354e6f7-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-fv6wt\" (UID: \"753197e9-c368-4498-8448-6c87b354e6f7\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-fv6wt"
Jul 28 10:46:10.887345 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:46:10.887181    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/753197e9-c368-4498-8448-6c87b354e6f7-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-fv6wt\" (UID: \"753197e9-c368-4498-8448-6c87b354e6f7\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-fv6wt"
Jul 28 10:46:10.887345 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:46:10.887200    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/753197e9-c368-4498-8448-6c87b354e6f7-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-fv6wt\" (UID: \"753197e9-c368-4498-8448-6c87b354e6f7\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-fv6wt"
Jul 28 10:46:10.887345 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:46:10.887224    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/753197e9-c368-4498-8448-6c87b354e6f7-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-fv6wt\" (UID: \"753197e9-c368-4498-8448-6c87b354e6f7\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-fv6wt"
Jul 28 10:46:10.887658 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:46:10.887578    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/753197e9-c368-4498-8448-6c87b354e6f7-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-fv6wt\" (UID: \"753197e9-c368-4498-8448-6c87b354e6f7\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-fv6wt"
Jul 28 10:46:10.887658 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:46:10.887585    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/753197e9-c368-4498-8448-6c87b354e6f7-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-fv6wt\" (UID: \"753197e9-c368-4498-8448-6c87b354e6f7\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-fv6wt"
Jul 28 10:46:10.887658 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:46:10.887632    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/753197e9-c368-4498-8448-6c87b354e6f7-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-fv6wt\" (UID: \"753197e9-c368-4498-8448-6c87b354e6f7\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-fv6wt"
Jul 28 10:46:10.889486 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:46:10.889456    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/753197e9-c368-4498-8448-6c87b354e6f7-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-fv6wt\" (UID: \"753197e9-c368-4498-8448-6c87b354e6f7\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-fv6wt"
Jul 28 10:46:10.889773 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:46:10.889755    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/753197e9-c368-4498-8448-6c87b354e6f7-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-fv6wt\" (UID: \"753197e9-c368-4498-8448-6c87b354e6f7\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-fv6wt"
Jul 28 10:46:10.896961 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:46:10.896935    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrn4z\" (UniqueName: \"kubernetes.io/projected/753197e9-c368-4498-8448-6c87b354e6f7-kube-api-access-hrn4z\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-fv6wt\" (UID: \"753197e9-c368-4498-8448-6c87b354e6f7\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-fv6wt"
Jul 28 10:46:10.971448 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:46:10.971362    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:46:10.993528 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:46:10.993497    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-fv6wt"
Jul 28 10:46:11.133412 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:46:11.133385    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-fv6wt"]
Jul 28 10:46:11.135145 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:46:11.135120    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod753197e9_c368_4498_8448_6c87b354e6f7.slice/crio-59e358e4471480cb03aabb0cfc02c98e6b0326ef543bee34cf6d56af55a6599e WatchSource:0}: Error finding container 59e358e4471480cb03aabb0cfc02c98e6b0326ef543bee34cf6d56af55a6599e: Status 404 returned error can't find the container with id 59e358e4471480cb03aabb0cfc02c98e6b0326ef543bee34cf6d56af55a6599e
Jul 28 10:46:11.162431 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:46:11.162395    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-fv6wt" event={"ID":"753197e9-c368-4498-8448-6c87b354e6f7","Type":"ContainerStarted","Data":"59e358e4471480cb03aabb0cfc02c98e6b0326ef543bee34cf6d56af55a6599e"}
Jul 28 10:46:12.168935 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:46:12.168901    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-fv6wt" event={"ID":"753197e9-c368-4498-8448-6c87b354e6f7","Type":"ContainerStarted","Data":"b2a86f7fd9faef0f45f8ba6a5ccb90ae9fc100fab5d1e3fe6196a31d8c80b227"}
Jul 28 10:46:12.186070 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:46:12.186017    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-fv6wt" podStartSLOduration=2.185997959 podStartE2EDuration="2.185997959s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:46:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:46:12.184526393 +0000 UTC m=+2306.447922723" watchObservedRunningTime="2026-07-28 10:46:12.185997959 +0000 UTC m=+2306.449394287"
Jul 28 10:46:12.994354 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:46:12.994306    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-fv6wt"
Jul 28 10:46:19.099465 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:46:19.099428    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:46:20.994286 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:46:20.994246    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-fv6wt"
Jul 28 10:46:21.007319 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:46:21.007288    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-fv6wt"
Jul 28 10:46:21.229277 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:46:21.229248    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-fv6wt"
Jul 28 10:46:44.381604 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:46:44.381512    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-fv6wt"]
Jul 28 10:46:44.392079 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:46:44.392042    2579 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 28 10:46:44.392079 ip-10-0-141-50 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 10:46:44.392079 ip-10-0-141-50 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Jul 28 10:46:44.392079 ip-10-0-141-50 kubenswrapper[2579]:  > containerID="b2a86f7fd9faef0f45f8ba6a5ccb90ae9fc100fab5d1e3fe6196a31d8c80b227" cmd=["/bin/sleep","15"]
Jul 28 10:46:44.392366 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:46:44.392095    2579 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-fv6wt" message="" err=<
Jul 28 10:46:44.392366 ip-10-0-141-50 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 10:46:44.392366 ip-10-0-141-50 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Jul 28 10:46:44.392366 ip-10-0-141-50 kubenswrapper[2579]:  >
Jul 28 10:46:44.392366 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:46:44.392107    2579 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 28 10:46:44.392366 ip-10-0-141-50 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 10:46:44.392366 ip-10-0-141-50 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Jul 28 10:46:44.392366 ip-10-0-141-50 kubenswrapper[2579]:  > pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-fv6wt" podUID="753197e9-c368-4498-8448-6c87b354e6f7" containerName="main" containerID="cri-o://b2a86f7fd9faef0f45f8ba6a5ccb90ae9fc100fab5d1e3fe6196a31d8c80b227"
Jul 28 10:46:44.392366 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:46:44.392161    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-fv6wt" podUID="753197e9-c368-4498-8448-6c87b354e6f7" containerName="main" containerID="cri-o://b2a86f7fd9faef0f45f8ba6a5ccb90ae9fc100fab5d1e3fe6196a31d8c80b227" gracePeriod=60
Jul 28 10:46:44.406921 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:46:44.406887    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:46:44.471428 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:46:44.471393    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:46:44.740355 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:46:44.740328    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-fv6wt"
Jul 28 10:46:44.889365 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:46:44.889333    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/753197e9-c368-4498-8448-6c87b354e6f7-model-cache\") pod \"753197e9-c368-4498-8448-6c87b354e6f7\" (UID: \"753197e9-c368-4498-8448-6c87b354e6f7\") "
Jul 28 10:46:44.889582 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:46:44.889377    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/753197e9-c368-4498-8448-6c87b354e6f7-tmp-dir\") pod \"753197e9-c368-4498-8448-6c87b354e6f7\" (UID: \"753197e9-c368-4498-8448-6c87b354e6f7\") "
Jul 28 10:46:44.889582 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:46:44.889395    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/753197e9-c368-4498-8448-6c87b354e6f7-home\") pod \"753197e9-c368-4498-8448-6c87b354e6f7\" (UID: \"753197e9-c368-4498-8448-6c87b354e6f7\") "
Jul 28 10:46:44.889582 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:46:44.889419    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/753197e9-c368-4498-8448-6c87b354e6f7-tls-certs\") pod \"753197e9-c368-4498-8448-6c87b354e6f7\" (UID: \"753197e9-c368-4498-8448-6c87b354e6f7\") "
Jul 28 10:46:44.889582 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:46:44.889485    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hrn4z\" (UniqueName: \"kubernetes.io/projected/753197e9-c368-4498-8448-6c87b354e6f7-kube-api-access-hrn4z\") pod \"753197e9-c368-4498-8448-6c87b354e6f7\" (UID: \"753197e9-c368-4498-8448-6c87b354e6f7\") "
Jul 28 10:46:44.889582 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:46:44.889533    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/753197e9-c368-4498-8448-6c87b354e6f7-dshm\") pod \"753197e9-c368-4498-8448-6c87b354e6f7\" (UID: \"753197e9-c368-4498-8448-6c87b354e6f7\") "
Jul 28 10:46:44.889870 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:46:44.889662    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/753197e9-c368-4498-8448-6c87b354e6f7-model-cache" (OuterVolumeSpecName: "model-cache") pod "753197e9-c368-4498-8448-6c87b354e6f7" (UID: "753197e9-c368-4498-8448-6c87b354e6f7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:46:44.889870 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:46:44.889723    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/753197e9-c368-4498-8448-6c87b354e6f7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "753197e9-c368-4498-8448-6c87b354e6f7" (UID: "753197e9-c368-4498-8448-6c87b354e6f7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:46:44.889870 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:46:44.889731    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/753197e9-c368-4498-8448-6c87b354e6f7-home" (OuterVolumeSpecName: "home") pod "753197e9-c368-4498-8448-6c87b354e6f7" (UID: "753197e9-c368-4498-8448-6c87b354e6f7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:46:44.889870 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:46:44.889823    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/753197e9-c368-4498-8448-6c87b354e6f7-model-cache\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 10:46:44.889870 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:46:44.889843    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/753197e9-c368-4498-8448-6c87b354e6f7-tmp-dir\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 10:46:44.889870 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:46:44.889856    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/753197e9-c368-4498-8448-6c87b354e6f7-home\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 10:46:44.891734 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:46:44.891714    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/753197e9-c368-4498-8448-6c87b354e6f7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "753197e9-c368-4498-8448-6c87b354e6f7" (UID: "753197e9-c368-4498-8448-6c87b354e6f7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:46:44.891842 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:46:44.891819    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/753197e9-c368-4498-8448-6c87b354e6f7-kube-api-access-hrn4z" (OuterVolumeSpecName: "kube-api-access-hrn4z") pod "753197e9-c368-4498-8448-6c87b354e6f7" (UID: "753197e9-c368-4498-8448-6c87b354e6f7"). InnerVolumeSpecName "kube-api-access-hrn4z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:46:44.891896 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:46:44.891825    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/753197e9-c368-4498-8448-6c87b354e6f7-dshm" (OuterVolumeSpecName: "dshm") pod "753197e9-c368-4498-8448-6c87b354e6f7" (UID: "753197e9-c368-4498-8448-6c87b354e6f7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:46:44.991291 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:46:44.991199    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/753197e9-c368-4498-8448-6c87b354e6f7-tls-certs\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 10:46:44.991291 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:46:44.991231    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrn4z\" (UniqueName: \"kubernetes.io/projected/753197e9-c368-4498-8448-6c87b354e6f7-kube-api-access-hrn4z\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 10:46:44.991291 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:46:44.991239    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/753197e9-c368-4498-8448-6c87b354e6f7-dshm\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 10:46:45.353829 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:46:45.353793    2579 generic.go:356] "Generic (PLEG): container finished" podID="753197e9-c368-4498-8448-6c87b354e6f7" containerID="b2a86f7fd9faef0f45f8ba6a5ccb90ae9fc100fab5d1e3fe6196a31d8c80b227" exitCode=0
Jul 28 10:46:45.354043 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:46:45.353844    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-fv6wt"
Jul 28 10:46:45.354043 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:46:45.353890    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-fv6wt" event={"ID":"753197e9-c368-4498-8448-6c87b354e6f7","Type":"ContainerDied","Data":"b2a86f7fd9faef0f45f8ba6a5ccb90ae9fc100fab5d1e3fe6196a31d8c80b227"}
Jul 28 10:46:45.354043 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:46:45.353951    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-fv6wt" event={"ID":"753197e9-c368-4498-8448-6c87b354e6f7","Type":"ContainerDied","Data":"59e358e4471480cb03aabb0cfc02c98e6b0326ef543bee34cf6d56af55a6599e"}
Jul 28 10:46:45.354043 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:46:45.353983    2579 scope.go:117] "RemoveContainer" containerID="b2a86f7fd9faef0f45f8ba6a5ccb90ae9fc100fab5d1e3fe6196a31d8c80b227"
Jul 28 10:46:45.363938 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:46:45.363911    2579 scope.go:117] "RemoveContainer" containerID="b2a86f7fd9faef0f45f8ba6a5ccb90ae9fc100fab5d1e3fe6196a31d8c80b227"
Jul 28 10:46:45.364264 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:46:45.364241    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2a86f7fd9faef0f45f8ba6a5ccb90ae9fc100fab5d1e3fe6196a31d8c80b227\": container with ID starting with b2a86f7fd9faef0f45f8ba6a5ccb90ae9fc100fab5d1e3fe6196a31d8c80b227 not found: ID does not exist" containerID="b2a86f7fd9faef0f45f8ba6a5ccb90ae9fc100fab5d1e3fe6196a31d8c80b227"
Jul 28 10:46:45.364313 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:46:45.364278    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2a86f7fd9faef0f45f8ba6a5ccb90ae9fc100fab5d1e3fe6196a31d8c80b227"} err="failed to get container status \"b2a86f7fd9faef0f45f8ba6a5ccb90ae9fc100fab5d1e3fe6196a31d8c80b227\": rpc error: code = NotFound desc = could not find container \"b2a86f7fd9faef0f45f8ba6a5ccb90ae9fc100fab5d1e3fe6196a31d8c80b227\": container with ID starting with b2a86f7fd9faef0f45f8ba6a5ccb90ae9fc100fab5d1e3fe6196a31d8c80b227 not found: ID does not exist"
Jul 28 10:46:45.375190 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:46:45.375161    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-fv6wt"]
Jul 28 10:46:45.378790 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:46:45.378756    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-fv6wt"]
Jul 28 10:46:46.277827 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:46:46.277790    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="753197e9-c368-4498-8448-6c87b354e6f7" path="/var/lib/kubelet/pods/753197e9-c368-4498-8448-6c87b354e6f7/volumes"
Jul 28 10:47:04.546184 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:47:04.546147    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:47:04.546950 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:47:04.546929    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scherqhcm"]
Jul 28 10:47:04.604114 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:47:04.604077    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:47:04.980906 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:47:04.980823    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scherqhcm" podUID="6bc286ba-bd26-4316-8fc9-ae99f09c3cc6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 10:47:14.980874 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:47:14.980831    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scherqhcm" podUID="6bc286ba-bd26-4316-8fc9-ae99f09c3cc6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 10:47:19.566745 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:47:19.566691    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scherqhcm" podUID="6bc286ba-bd26-4316-8fc9-ae99f09c3cc6" containerName="main" containerID="cri-o://f53cc463e9d23f44bb6c90dc5aef0347a3b303e0fd5124c39eb4c90207293bab" gracePeriod=45
Jul 28 10:47:19.814958 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:47:19.814929    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scherqhcm"
Jul 28 10:47:19.878000 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:47:19.877971    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6bc286ba-bd26-4316-8fc9-ae99f09c3cc6-tls-certs\") pod \"6bc286ba-bd26-4316-8fc9-ae99f09c3cc6\" (UID: \"6bc286ba-bd26-4316-8fc9-ae99f09c3cc6\") "
Jul 28 10:47:19.878200 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:47:19.878041    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2nxdr\" (UniqueName: \"kubernetes.io/projected/6bc286ba-bd26-4316-8fc9-ae99f09c3cc6-kube-api-access-2nxdr\") pod \"6bc286ba-bd26-4316-8fc9-ae99f09c3cc6\" (UID: \"6bc286ba-bd26-4316-8fc9-ae99f09c3cc6\") "
Jul 28 10:47:19.880167 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:47:19.880136    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6bc286ba-bd26-4316-8fc9-ae99f09c3cc6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6bc286ba-bd26-4316-8fc9-ae99f09c3cc6" (UID: "6bc286ba-bd26-4316-8fc9-ae99f09c3cc6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:47:19.880167 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:47:19.880136    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bc286ba-bd26-4316-8fc9-ae99f09c3cc6-kube-api-access-2nxdr" (OuterVolumeSpecName: "kube-api-access-2nxdr") pod "6bc286ba-bd26-4316-8fc9-ae99f09c3cc6" (UID: "6bc286ba-bd26-4316-8fc9-ae99f09c3cc6"). InnerVolumeSpecName "kube-api-access-2nxdr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:47:19.979510 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:47:19.979418    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6bc286ba-bd26-4316-8fc9-ae99f09c3cc6-tls-certs\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 10:47:19.979510 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:47:19.979450    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2nxdr\" (UniqueName: \"kubernetes.io/projected/6bc286ba-bd26-4316-8fc9-ae99f09c3cc6-kube-api-access-2nxdr\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 10:47:20.524381 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:47:20.524345    2579 generic.go:356] "Generic (PLEG): container finished" podID="6bc286ba-bd26-4316-8fc9-ae99f09c3cc6" containerID="f53cc463e9d23f44bb6c90dc5aef0347a3b303e0fd5124c39eb4c90207293bab" exitCode=0
Jul 28 10:47:20.524586 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:47:20.524393    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scherqhcm" event={"ID":"6bc286ba-bd26-4316-8fc9-ae99f09c3cc6","Type":"ContainerDied","Data":"f53cc463e9d23f44bb6c90dc5aef0347a3b303e0fd5124c39eb4c90207293bab"}
Jul 28 10:47:20.524586 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:47:20.524440    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scherqhcm" event={"ID":"6bc286ba-bd26-4316-8fc9-ae99f09c3cc6","Type":"ContainerDied","Data":"2435a8ad62ccd0cd5be9e62a526bd803847222fe93342de25f24f5da2cead953"}
Jul 28 10:47:20.524586 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:47:20.524395    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scherqhcm"
Jul 28 10:47:20.524586 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:47:20.524473    2579 scope.go:117] "RemoveContainer" containerID="f53cc463e9d23f44bb6c90dc5aef0347a3b303e0fd5124c39eb4c90207293bab"
Jul 28 10:47:20.532942 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:47:20.532925    2579 scope.go:117] "RemoveContainer" containerID="f53cc463e9d23f44bb6c90dc5aef0347a3b303e0fd5124c39eb4c90207293bab"
Jul 28 10:47:20.535782 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:47:20.535743    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f53cc463e9d23f44bb6c90dc5aef0347a3b303e0fd5124c39eb4c90207293bab\": container with ID starting with f53cc463e9d23f44bb6c90dc5aef0347a3b303e0fd5124c39eb4c90207293bab not found: ID does not exist" containerID="f53cc463e9d23f44bb6c90dc5aef0347a3b303e0fd5124c39eb4c90207293bab"
Jul 28 10:47:20.535900 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:47:20.535780    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f53cc463e9d23f44bb6c90dc5aef0347a3b303e0fd5124c39eb4c90207293bab"} err="failed to get container status \"f53cc463e9d23f44bb6c90dc5aef0347a3b303e0fd5124c39eb4c90207293bab\": rpc error: code = NotFound desc = could not find container \"f53cc463e9d23f44bb6c90dc5aef0347a3b303e0fd5124c39eb4c90207293bab\": container with ID starting with f53cc463e9d23f44bb6c90dc5aef0347a3b303e0fd5124c39eb4c90207293bab not found: ID does not exist"
Jul 28 10:47:20.539124 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:47:20.539088    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scherqhcm"]
Jul 28 10:47:20.542820 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:47:20.542802    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scherqhcm"]
Jul 28 10:47:22.275750 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:47:22.275706    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6bc286ba-bd26-4316-8fc9-ae99f09c3cc6" path="/var/lib/kubelet/pods/6bc286ba-bd26-4316-8fc9-ae99f09c3cc6/volumes"
Jul 28 10:47:24.588504 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:47:24.588474    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:47:24.687875 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:47:24.687831    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:47:28.833378 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:47:28.833345    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:47:29.020692 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:47:29.020657    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schej7xw5"]
Jul 28 10:47:29.021168 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:47:29.021144    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6bc286ba-bd26-4316-8fc9-ae99f09c3cc6" containerName="main"
Jul 28 10:47:29.021168 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:47:29.021168    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6bc286ba-bd26-4316-8fc9-ae99f09c3cc6" containerName="main"
Jul 28 10:47:29.021332 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:47:29.021192    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="753197e9-c368-4498-8448-6c87b354e6f7" containerName="main"
Jul 28 10:47:29.021332 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:47:29.021200    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="753197e9-c368-4498-8448-6c87b354e6f7" containerName="main"
Jul 28 10:47:29.021332 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:47:29.021308    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="753197e9-c368-4498-8448-6c87b354e6f7" containerName="main"
Jul 28 10:47:29.021332 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:47:29.021323    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="6bc286ba-bd26-4316-8fc9-ae99f09c3cc6" containerName="main"
Jul 28 10:47:29.024038 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:47:29.024020    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schej7xw5"
Jul 28 10:47:29.026778 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:47:29.026738    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"kube-root-ca.crt\""
Jul 28 10:47:29.026778 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:47:29.026744    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"llmisv3e414c2ba058a022dfd694dbcbac5b51-kserve-self-signed-certs\""
Jul 28 10:47:29.026972 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:47:29.026780    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"openshift-service-ca.crt\""
Jul 28 10:47:29.026972 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:47:29.026778    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"default-dockercfg-jws88\""
Jul 28 10:47:29.026972 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:47:29.026903    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"llmisvc-model-fb-opt-125m-with-7ca60146-epp-sa-dockercfg-f9kkg\""
Jul 28 10:47:29.034911 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:47:29.034879    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schej7xw5"]
Jul 28 10:47:29.055568 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:47:29.055523    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:47:29.151944 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:47:29.151849    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drlbg\" (UniqueName: \"kubernetes.io/projected/1825d742-598b-4e93-aadc-ae877b726eee-kube-api-access-drlbg\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schej7xw5\" (UID: \"1825d742-598b-4e93-aadc-ae877b726eee\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schej7xw5"
Jul 28 10:47:29.151944 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:47:29.151901    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1825d742-598b-4e93-aadc-ae877b726eee-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schej7xw5\" (UID: \"1825d742-598b-4e93-aadc-ae877b726eee\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schej7xw5"
Jul 28 10:47:29.252952 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:47:29.252919    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1825d742-598b-4e93-aadc-ae877b726eee-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schej7xw5\" (UID: \"1825d742-598b-4e93-aadc-ae877b726eee\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schej7xw5"
Jul 28 10:47:29.253163 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:47:29.253014    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drlbg\" (UniqueName: \"kubernetes.io/projected/1825d742-598b-4e93-aadc-ae877b726eee-kube-api-access-drlbg\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schej7xw5\" (UID: \"1825d742-598b-4e93-aadc-ae877b726eee\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schej7xw5"
Jul 28 10:47:29.255705 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:47:29.255645    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1825d742-598b-4e93-aadc-ae877b726eee-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schej7xw5\" (UID: \"1825d742-598b-4e93-aadc-ae877b726eee\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schej7xw5"
Jul 28 10:47:29.263266 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:47:29.263237    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-drlbg\" (UniqueName: \"kubernetes.io/projected/1825d742-598b-4e93-aadc-ae877b726eee-kube-api-access-drlbg\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schej7xw5\" (UID: \"1825d742-598b-4e93-aadc-ae877b726eee\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schej7xw5"
Jul 28 10:47:29.336657 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:47:29.336617    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schej7xw5"
Jul 28 10:47:29.471012 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:47:29.470987    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schej7xw5"]
Jul 28 10:47:29.473183 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:47:29.473155    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1825d742_598b_4e93_aadc_ae877b726eee.slice/crio-5be27574fac058b15bc41af239cee36fd7172eab53d784a86585b048f77e4d69 WatchSource:0}: Error finding container 5be27574fac058b15bc41af239cee36fd7172eab53d784a86585b048f77e4d69: Status 404 returned error can't find the container with id 5be27574fac058b15bc41af239cee36fd7172eab53d784a86585b048f77e4d69
Jul 28 10:47:29.570390 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:47:29.570355    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schej7xw5" event={"ID":"1825d742-598b-4e93-aadc-ae877b726eee","Type":"ContainerStarted","Data":"e66157bfd1276ac5bef8bd2aa402b9ad24cc9a40a03781d0d2fe3b80a32a9c2f"}
Jul 28 10:47:29.570583 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:47:29.570416    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schej7xw5" event={"ID":"1825d742-598b-4e93-aadc-ae877b726eee","Type":"ContainerStarted","Data":"5be27574fac058b15bc41af239cee36fd7172eab53d784a86585b048f77e4d69"}
Jul 28 10:47:29.570583 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:47:29.570475    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schej7xw5"
Jul 28 10:47:29.589693 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:47:29.589632    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schej7xw5" podStartSLOduration=0.589616041 podStartE2EDuration="589.616041ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:47:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:47:29.588894873 +0000 UTC m=+2383.852291204" watchObservedRunningTime="2026-07-28 10:47:29.589616041 +0000 UTC m=+2383.853012369"
Jul 28 10:47:32.085605 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:47:32.085569    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:47:37.647427 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:47:37.647390    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:47:37.709271 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:47:37.709227    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:47:41.715958 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:47:41.715917    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:48:00.577584 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:48:00.577551    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schej7xw5"
Jul 28 10:48:39.641078 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:48:39.641038    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:49:24.461240 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:49:24.461197    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:49:24.462210 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:49:24.462186    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schej7xw5"]
Jul 28 10:49:24.523106 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:49:24.523066    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:49:26.831493 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:49:26.831462    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-986gr"]
Jul 28 10:49:26.837627 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:49:26.837595    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:49:26.935073 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:49:26.935038    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:49:30.577718 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:49:30.577680    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schej7xw5" podUID="1825d742-598b-4e93-aadc-ae877b726eee" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 10:49:30.698600 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:49:30.698565    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-986gr" podUID="dee8a861-2e0c-482b-822c-fbbea81567c0" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 10:49:40.197427 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:49:40.197310    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schej7xw5" podUID="1825d742-598b-4e93-aadc-ae877b726eee" containerName="main" containerID="cri-o://e66157bfd1276ac5bef8bd2aa402b9ad24cc9a40a03781d0d2fe3b80a32a9c2f" gracePeriod=45
Jul 28 10:49:40.447938 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:49:40.447873    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schej7xw5"
Jul 28 10:49:40.579486 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:49:40.579449    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1825d742-598b-4e93-aadc-ae877b726eee-tls-certs\") pod \"1825d742-598b-4e93-aadc-ae877b726eee\" (UID: \"1825d742-598b-4e93-aadc-ae877b726eee\") "
Jul 28 10:49:40.579680 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:49:40.579558    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-drlbg\" (UniqueName: \"kubernetes.io/projected/1825d742-598b-4e93-aadc-ae877b726eee-kube-api-access-drlbg\") pod \"1825d742-598b-4e93-aadc-ae877b726eee\" (UID: \"1825d742-598b-4e93-aadc-ae877b726eee\") "
Jul 28 10:49:40.581741 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:49:40.581715    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1825d742-598b-4e93-aadc-ae877b726eee-kube-api-access-drlbg" (OuterVolumeSpecName: "kube-api-access-drlbg") pod "1825d742-598b-4e93-aadc-ae877b726eee" (UID: "1825d742-598b-4e93-aadc-ae877b726eee"). InnerVolumeSpecName "kube-api-access-drlbg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:49:40.581834 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:49:40.581796    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1825d742-598b-4e93-aadc-ae877b726eee-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1825d742-598b-4e93-aadc-ae877b726eee" (UID: "1825d742-598b-4e93-aadc-ae877b726eee"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:49:40.680899 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:49:40.680860    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1825d742-598b-4e93-aadc-ae877b726eee-tls-certs\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 10:49:40.680899 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:49:40.680893    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drlbg\" (UniqueName: \"kubernetes.io/projected/1825d742-598b-4e93-aadc-ae877b726eee-kube-api-access-drlbg\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 10:49:40.698716 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:49:40.698633    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-986gr" podUID="dee8a861-2e0c-482b-822c-fbbea81567c0" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 10:49:41.251742 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:49:41.251707    2579 generic.go:356] "Generic (PLEG): container finished" podID="1825d742-598b-4e93-aadc-ae877b726eee" containerID="e66157bfd1276ac5bef8bd2aa402b9ad24cc9a40a03781d0d2fe3b80a32a9c2f" exitCode=0
Jul 28 10:49:41.252174 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:49:41.251790    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schej7xw5" event={"ID":"1825d742-598b-4e93-aadc-ae877b726eee","Type":"ContainerDied","Data":"e66157bfd1276ac5bef8bd2aa402b9ad24cc9a40a03781d0d2fe3b80a32a9c2f"}
Jul 28 10:49:41.252174 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:49:41.251750    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schej7xw5"
Jul 28 10:49:41.252174 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:49:41.251849    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schej7xw5" event={"ID":"1825d742-598b-4e93-aadc-ae877b726eee","Type":"ContainerDied","Data":"5be27574fac058b15bc41af239cee36fd7172eab53d784a86585b048f77e4d69"}
Jul 28 10:49:41.252174 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:49:41.251879    2579 scope.go:117] "RemoveContainer" containerID="e66157bfd1276ac5bef8bd2aa402b9ad24cc9a40a03781d0d2fe3b80a32a9c2f"
Jul 28 10:49:41.261036 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:49:41.261016    2579 scope.go:117] "RemoveContainer" containerID="e66157bfd1276ac5bef8bd2aa402b9ad24cc9a40a03781d0d2fe3b80a32a9c2f"
Jul 28 10:49:41.261296 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:49:41.261278    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e66157bfd1276ac5bef8bd2aa402b9ad24cc9a40a03781d0d2fe3b80a32a9c2f\": container with ID starting with e66157bfd1276ac5bef8bd2aa402b9ad24cc9a40a03781d0d2fe3b80a32a9c2f not found: ID does not exist" containerID="e66157bfd1276ac5bef8bd2aa402b9ad24cc9a40a03781d0d2fe3b80a32a9c2f"
Jul 28 10:49:41.261341 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:49:41.261304    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e66157bfd1276ac5bef8bd2aa402b9ad24cc9a40a03781d0d2fe3b80a32a9c2f"} err="failed to get container status \"e66157bfd1276ac5bef8bd2aa402b9ad24cc9a40a03781d0d2fe3b80a32a9c2f\": rpc error: code = NotFound desc = could not find container \"e66157bfd1276ac5bef8bd2aa402b9ad24cc9a40a03781d0d2fe3b80a32a9c2f\": container with ID starting with e66157bfd1276ac5bef8bd2aa402b9ad24cc9a40a03781d0d2fe3b80a32a9c2f not found: ID does not exist"
Jul 28 10:49:41.272552 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:49:41.272510    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schej7xw5"]
Jul 28 10:49:41.279240 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:49:41.279215    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schej7xw5"]
Jul 28 10:49:41.852899 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:49:41.852859    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-986gr" podUID="dee8a861-2e0c-482b-822c-fbbea81567c0" containerName="main" containerID="cri-o://b084f0289ceef1c761395f7e18b1e76ef72b7c400c6f51f83a51c2bb5f224741" gracePeriod=45
Jul 28 10:49:42.113781 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:49:42.113719    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-986gr"
Jul 28 10:49:42.193208 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:49:42.193170    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-scf2w\" (UniqueName: \"kubernetes.io/projected/dee8a861-2e0c-482b-822c-fbbea81567c0-kube-api-access-scf2w\") pod \"dee8a861-2e0c-482b-822c-fbbea81567c0\" (UID: \"dee8a861-2e0c-482b-822c-fbbea81567c0\") "
Jul 28 10:49:42.193420 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:49:42.193217    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dee8a861-2e0c-482b-822c-fbbea81567c0-tls-certs\") pod \"dee8a861-2e0c-482b-822c-fbbea81567c0\" (UID: \"dee8a861-2e0c-482b-822c-fbbea81567c0\") "
Jul 28 10:49:42.195590 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:49:42.195533    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dee8a861-2e0c-482b-822c-fbbea81567c0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "dee8a861-2e0c-482b-822c-fbbea81567c0" (UID: "dee8a861-2e0c-482b-822c-fbbea81567c0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:49:42.195712 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:49:42.195559    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dee8a861-2e0c-482b-822c-fbbea81567c0-kube-api-access-scf2w" (OuterVolumeSpecName: "kube-api-access-scf2w") pod "dee8a861-2e0c-482b-822c-fbbea81567c0" (UID: "dee8a861-2e0c-482b-822c-fbbea81567c0"). InnerVolumeSpecName "kube-api-access-scf2w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:49:42.258033 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:49:42.257993    2579 generic.go:356] "Generic (PLEG): container finished" podID="dee8a861-2e0c-482b-822c-fbbea81567c0" containerID="b084f0289ceef1c761395f7e18b1e76ef72b7c400c6f51f83a51c2bb5f224741" exitCode=0
Jul 28 10:49:42.258525 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:49:42.258077    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-986gr"
Jul 28 10:49:42.258525 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:49:42.258083    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-986gr" event={"ID":"dee8a861-2e0c-482b-822c-fbbea81567c0","Type":"ContainerDied","Data":"b084f0289ceef1c761395f7e18b1e76ef72b7c400c6f51f83a51c2bb5f224741"}
Jul 28 10:49:42.258525 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:49:42.258159    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-986gr" event={"ID":"dee8a861-2e0c-482b-822c-fbbea81567c0","Type":"ContainerDied","Data":"bfb7d59f12370f428667595270df60098246aeaa64d96c488d9baea094b7b6d4"}
Jul 28 10:49:42.258525 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:49:42.258182    2579 scope.go:117] "RemoveContainer" containerID="b084f0289ceef1c761395f7e18b1e76ef72b7c400c6f51f83a51c2bb5f224741"
Jul 28 10:49:42.267431 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:49:42.267416    2579 scope.go:117] "RemoveContainer" containerID="b084f0289ceef1c761395f7e18b1e76ef72b7c400c6f51f83a51c2bb5f224741"
Jul 28 10:49:42.267748 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:49:42.267724    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b084f0289ceef1c761395f7e18b1e76ef72b7c400c6f51f83a51c2bb5f224741\": container with ID starting with b084f0289ceef1c761395f7e18b1e76ef72b7c400c6f51f83a51c2bb5f224741 not found: ID does not exist" containerID="b084f0289ceef1c761395f7e18b1e76ef72b7c400c6f51f83a51c2bb5f224741"
Jul 28 10:49:42.267844 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:49:42.267755    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b084f0289ceef1c761395f7e18b1e76ef72b7c400c6f51f83a51c2bb5f224741"} err="failed to get container status \"b084f0289ceef1c761395f7e18b1e76ef72b7c400c6f51f83a51c2bb5f224741\": rpc error: code = NotFound desc = could not find container \"b084f0289ceef1c761395f7e18b1e76ef72b7c400c6f51f83a51c2bb5f224741\": container with ID starting with b084f0289ceef1c761395f7e18b1e76ef72b7c400c6f51f83a51c2bb5f224741 not found: ID does not exist"
Jul 28 10:49:42.275799 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:49:42.275769    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1825d742-598b-4e93-aadc-ae877b726eee" path="/var/lib/kubelet/pods/1825d742-598b-4e93-aadc-ae877b726eee/volumes"
Jul 28 10:49:42.279585 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:49:42.279561    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-986gr"]
Jul 28 10:49:42.285952 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:49:42.285926    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-986gr"]
Jul 28 10:49:42.294403 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:49:42.294380    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-scf2w\" (UniqueName: \"kubernetes.io/projected/dee8a861-2e0c-482b-822c-fbbea81567c0-kube-api-access-scf2w\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 10:49:42.294403 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:49:42.294400    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dee8a861-2e0c-482b-822c-fbbea81567c0-tls-certs\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 10:49:44.275272 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:49:44.275239    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dee8a861-2e0c-482b-822c-fbbea81567c0" path="/var/lib/kubelet/pods/dee8a861-2e0c-482b-822c-fbbea81567c0/volumes"
Jul 28 10:49:44.381135 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:49:44.381102    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:49:44.481044 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:49:44.480974    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:49:47.016162 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:49:47.016112    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:49:47.129381 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:49:47.129345    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:49:50.509736 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:49:50.509698    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:49:56.686001 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:49:56.685965    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:49:56.782175 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:49:56.782140    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:49:58.801847 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:49:58.801810    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-84cbcdc776-7dr8g"]
Jul 28 10:49:58.802228 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:49:58.802118    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1825d742-598b-4e93-aadc-ae877b726eee" containerName="main"
Jul 28 10:49:58.802228 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:49:58.802129    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1825d742-598b-4e93-aadc-ae877b726eee" containerName="main"
Jul 28 10:49:58.802228 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:49:58.802145    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dee8a861-2e0c-482b-822c-fbbea81567c0" containerName="main"
Jul 28 10:49:58.802228 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:49:58.802151    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="dee8a861-2e0c-482b-822c-fbbea81567c0" containerName="main"
Jul 28 10:49:58.802228 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:49:58.802195    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="1825d742-598b-4e93-aadc-ae877b726eee" containerName="main"
Jul 28 10:49:58.802228 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:49:58.802205    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="dee8a861-2e0c-482b-822c-fbbea81567c0" containerName="main"
Jul 28 10:49:58.805215 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:49:58.805198    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-84cbcdc776-7dr8g"
Jul 28 10:49:58.807425 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:49:58.807400    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs\""
Jul 28 10:49:58.807590 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:49:58.807571    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"seaweedfs-s3-creds\""
Jul 28 10:49:58.807679 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:49:58.807641    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 10:49:58.807740 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:49:58.807725    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"default-dockercfg-27jsr\""
Jul 28 10:49:58.807868 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:49:58.807852    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"kube-root-ca.crt\""
Jul 28 10:49:58.807940 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:49:58.807912    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"openshift-service-ca.crt\""
Jul 28 10:49:58.815471 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:49:58.815452    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-84cbcdc776-7dr8g"]
Jul 28 10:49:58.833192 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:49:58.833158    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37f3723b-934d-4950-8853-b809bdb3764f-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-84cbcdc776-7dr8g\" (UID: \"37f3723b-934d-4950-8853-b809bdb3764f\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-84cbcdc776-7dr8g"
Jul 28 10:49:58.833335 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:49:58.833198    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/37f3723b-934d-4950-8853-b809bdb3764f-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-84cbcdc776-7dr8g\" (UID: \"37f3723b-934d-4950-8853-b809bdb3764f\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-84cbcdc776-7dr8g"
Jul 28 10:49:58.833335 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:49:58.833245    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37f3723b-934d-4950-8853-b809bdb3764f-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-84cbcdc776-7dr8g\" (UID: \"37f3723b-934d-4950-8853-b809bdb3764f\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-84cbcdc776-7dr8g"
Jul 28 10:49:58.833335 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:49:58.833329    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37f3723b-934d-4950-8853-b809bdb3764f-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-84cbcdc776-7dr8g\" (UID: \"37f3723b-934d-4950-8853-b809bdb3764f\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-84cbcdc776-7dr8g"
Jul 28 10:49:58.833475 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:49:58.833371    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37f3723b-934d-4950-8853-b809bdb3764f-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-84cbcdc776-7dr8g\" (UID: \"37f3723b-934d-4950-8853-b809bdb3764f\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-84cbcdc776-7dr8g"
Jul 28 10:49:58.833475 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:49:58.833405    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfwr7\" (UniqueName: \"kubernetes.io/projected/37f3723b-934d-4950-8853-b809bdb3764f-kube-api-access-kfwr7\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-84cbcdc776-7dr8g\" (UID: \"37f3723b-934d-4950-8853-b809bdb3764f\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-84cbcdc776-7dr8g"
Jul 28 10:49:58.833570 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:49:58.833485    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/37f3723b-934d-4950-8853-b809bdb3764f-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-84cbcdc776-7dr8g\" (UID: \"37f3723b-934d-4950-8853-b809bdb3764f\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-84cbcdc776-7dr8g"
Jul 28 10:49:58.833613 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:49:58.833569    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37f3723b-934d-4950-8853-b809bdb3764f-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-84cbcdc776-7dr8g\" (UID: \"37f3723b-934d-4950-8853-b809bdb3764f\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-84cbcdc776-7dr8g"
Jul 28 10:49:58.870750 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:49:58.870720    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:49:58.934848 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:49:58.934809    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/37f3723b-934d-4950-8853-b809bdb3764f-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-84cbcdc776-7dr8g\" (UID: \"37f3723b-934d-4950-8853-b809bdb3764f\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-84cbcdc776-7dr8g"
Jul 28 10:49:58.935010 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:49:58.934861    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37f3723b-934d-4950-8853-b809bdb3764f-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-84cbcdc776-7dr8g\" (UID: \"37f3723b-934d-4950-8853-b809bdb3764f\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-84cbcdc776-7dr8g"
Jul 28 10:49:58.935010 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:49:58.934979    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37f3723b-934d-4950-8853-b809bdb3764f-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-84cbcdc776-7dr8g\" (UID: \"37f3723b-934d-4950-8853-b809bdb3764f\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-84cbcdc776-7dr8g"
Jul 28 10:49:58.935116 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:49:58.935026    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/37f3723b-934d-4950-8853-b809bdb3764f-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-84cbcdc776-7dr8g\" (UID: \"37f3723b-934d-4950-8853-b809bdb3764f\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-84cbcdc776-7dr8g"
Jul 28 10:49:58.935116 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:49:58.935051    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37f3723b-934d-4950-8853-b809bdb3764f-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-84cbcdc776-7dr8g\" (UID: \"37f3723b-934d-4950-8853-b809bdb3764f\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-84cbcdc776-7dr8g"
Jul 28 10:49:58.935116 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:49:58.935098    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37f3723b-934d-4950-8853-b809bdb3764f-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-84cbcdc776-7dr8g\" (UID: \"37f3723b-934d-4950-8853-b809bdb3764f\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-84cbcdc776-7dr8g"
Jul 28 10:49:58.935264 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:49:58.935136    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37f3723b-934d-4950-8853-b809bdb3764f-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-84cbcdc776-7dr8g\" (UID: \"37f3723b-934d-4950-8853-b809bdb3764f\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-84cbcdc776-7dr8g"
Jul 28 10:49:58.935264 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:49:58.935164    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfwr7\" (UniqueName: \"kubernetes.io/projected/37f3723b-934d-4950-8853-b809bdb3764f-kube-api-access-kfwr7\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-84cbcdc776-7dr8g\" (UID: \"37f3723b-934d-4950-8853-b809bdb3764f\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-84cbcdc776-7dr8g"
Jul 28 10:49:58.935381 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:49:58.935269    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37f3723b-934d-4950-8853-b809bdb3764f-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-84cbcdc776-7dr8g\" (UID: \"37f3723b-934d-4950-8853-b809bdb3764f\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-84cbcdc776-7dr8g"
Jul 28 10:49:58.935381 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:49:58.935313    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/37f3723b-934d-4950-8853-b809bdb3764f-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-84cbcdc776-7dr8g\" (UID: \"37f3723b-934d-4950-8853-b809bdb3764f\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-84cbcdc776-7dr8g"
Jul 28 10:49:58.935575 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:49:58.935526    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37f3723b-934d-4950-8853-b809bdb3764f-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-84cbcdc776-7dr8g\" (UID: \"37f3723b-934d-4950-8853-b809bdb3764f\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-84cbcdc776-7dr8g"
Jul 28 10:49:58.935702 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:49:58.935598    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37f3723b-934d-4950-8853-b809bdb3764f-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-84cbcdc776-7dr8g\" (UID: \"37f3723b-934d-4950-8853-b809bdb3764f\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-84cbcdc776-7dr8g"
Jul 28 10:49:58.935833 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:49:58.935816    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/37f3723b-934d-4950-8853-b809bdb3764f-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-84cbcdc776-7dr8g\" (UID: \"37f3723b-934d-4950-8853-b809bdb3764f\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-84cbcdc776-7dr8g"
Jul 28 10:49:58.937331 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:49:58.937309    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37f3723b-934d-4950-8853-b809bdb3764f-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-84cbcdc776-7dr8g\" (UID: \"37f3723b-934d-4950-8853-b809bdb3764f\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-84cbcdc776-7dr8g"
Jul 28 10:49:58.937462 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:49:58.937446    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37f3723b-934d-4950-8853-b809bdb3764f-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-84cbcdc776-7dr8g\" (UID: \"37f3723b-934d-4950-8853-b809bdb3764f\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-84cbcdc776-7dr8g"
Jul 28 10:49:58.943563 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:49:58.943520    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfwr7\" (UniqueName: \"kubernetes.io/projected/37f3723b-934d-4950-8853-b809bdb3764f-kube-api-access-kfwr7\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-84cbcdc776-7dr8g\" (UID: \"37f3723b-934d-4950-8853-b809bdb3764f\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-84cbcdc776-7dr8g"
Jul 28 10:49:59.076345 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:49:59.076247    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:49:59.116108 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:49:59.116076    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-84cbcdc776-7dr8g"
Jul 28 10:49:59.254720 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:49:59.254628    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-84cbcdc776-7dr8g"]
Jul 28 10:49:59.257554 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:49:59.257513    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37f3723b_934d_4950_8853_b809bdb3764f.slice/crio-b7d4f1ab0048b6dff7c1137e092fd401a80af9cb09cc991ef78edcbf5e742b78 WatchSource:0}: Error finding container b7d4f1ab0048b6dff7c1137e092fd401a80af9cb09cc991ef78edcbf5e742b78: Status 404 returned error can't find the container with id b7d4f1ab0048b6dff7c1137e092fd401a80af9cb09cc991ef78edcbf5e742b78
Jul 28 10:49:59.338358 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:49:59.338327    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-84cbcdc776-7dr8g" event={"ID":"37f3723b-934d-4950-8853-b809bdb3764f","Type":"ContainerStarted","Data":"2dc050fd7a7283316952dc8b17119db1756fc7d782df539f961ef72c1e067f14"}
Jul 28 10:49:59.338486 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:49:59.338389    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-84cbcdc776-7dr8g" event={"ID":"37f3723b-934d-4950-8853-b809bdb3764f","Type":"ContainerStarted","Data":"b7d4f1ab0048b6dff7c1137e092fd401a80af9cb09cc991ef78edcbf5e742b78"}
Jul 28 10:50:05.372520 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:50:05.372486    2579 generic.go:356] "Generic (PLEG): container finished" podID="37f3723b-934d-4950-8853-b809bdb3764f" containerID="2dc050fd7a7283316952dc8b17119db1756fc7d782df539f961ef72c1e067f14" exitCode=0
Jul 28 10:50:05.372928 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:50:05.372567    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-84cbcdc776-7dr8g" event={"ID":"37f3723b-934d-4950-8853-b809bdb3764f","Type":"ContainerDied","Data":"2dc050fd7a7283316952dc8b17119db1756fc7d782df539f961ef72c1e067f14"}
Jul 28 10:50:06.381639 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:50:06.381591    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-84cbcdc776-7dr8g" event={"ID":"37f3723b-934d-4950-8853-b809bdb3764f","Type":"ContainerStarted","Data":"32367250f78d110bc1d287684b82b58d2789c584452f546fb2c25126da6ac509"}
Jul 28 10:50:06.403798 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:50:06.403723    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-84cbcdc776-7dr8g" podStartSLOduration=2.984483043 podStartE2EDuration="8.403701012s" totalImagesPullingTime="0s" totalInitContainerRuntime="5.419217969s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:49:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:50:06.401838437 +0000 UTC m=+2540.665234780" watchObservedRunningTime="2026-07-28 10:50:06.403701012 +0000 UTC m=+2540.667097344"
Jul 28 10:50:07.116761 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:50:07.116722    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-84cbcdc776-7dr8g"
Jul 28 10:50:08.570154 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:50:08.570116    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:50:08.987993 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:50:08.987891    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-rhmxf"]
Jul 28 10:50:08.992710 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:50:08.992676    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-rhmxf"
Jul 28 10:50:08.995358 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:50:08.995326    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-keda-deployment-b2150d0b\"/\"autoscale-keda-deploy-epp-sa-dockercfg-qwbc8\""
Jul 28 10:50:08.995510 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:50:08.995355    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-keda-deployment-b2150d0b\"/\"autoscale-keda-deploy-kserve-self-signed-certs\""
Jul 28 10:50:08.995510 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:50:08.995448    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-autoscaling-keda-deployment-b2150d0b\"/\"openshift-service-ca.crt\""
Jul 28 10:50:08.995655 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:50:08.995594    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-keda-deployment-b2150d0b\"/\"default-dockercfg-54svx\""
Jul 28 10:50:08.995724 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:50:08.995679    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-autoscaling-keda-deployment-b2150d0b\"/\"kube-root-ca.crt\""
Jul 28 10:50:09.003452 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:50:09.003419    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-rhmxf"]
Jul 28 10:50:09.017294 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:50:09.017252    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:50:09.117028 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:50:09.116968    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-84cbcdc776-7dr8g"
Jul 28 10:50:09.119268 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:50:09.119226    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-84cbcdc776-7dr8g" podUID="37f3723b-934d-4950-8853-b809bdb3764f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused"
Jul 28 10:50:09.124341 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:50:09.124313    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbb2t\" (UniqueName: \"kubernetes.io/projected/5b826d19-52f6-4b9b-81b8-31ef30f6d798-kube-api-access-bbb2t\") pod \"autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-rhmxf\" (UID: \"5b826d19-52f6-4b9b-81b8-31ef30f6d798\") " pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-rhmxf"
Jul 28 10:50:09.124473 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:50:09.124438    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b826d19-52f6-4b9b-81b8-31ef30f6d798-tls-certs\") pod \"autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-rhmxf\" (UID: \"5b826d19-52f6-4b9b-81b8-31ef30f6d798\") " pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-rhmxf"
Jul 28 10:50:09.225451 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:50:09.225403    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbb2t\" (UniqueName: \"kubernetes.io/projected/5b826d19-52f6-4b9b-81b8-31ef30f6d798-kube-api-access-bbb2t\") pod \"autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-rhmxf\" (UID: \"5b826d19-52f6-4b9b-81b8-31ef30f6d798\") " pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-rhmxf"
Jul 28 10:50:09.225694 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:50:09.225512    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b826d19-52f6-4b9b-81b8-31ef30f6d798-tls-certs\") pod \"autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-rhmxf\" (UID: \"5b826d19-52f6-4b9b-81b8-31ef30f6d798\") " pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-rhmxf"
Jul 28 10:50:09.227987 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:50:09.227962    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b826d19-52f6-4b9b-81b8-31ef30f6d798-tls-certs\") pod \"autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-rhmxf\" (UID: \"5b826d19-52f6-4b9b-81b8-31ef30f6d798\") " pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-rhmxf"
Jul 28 10:50:09.232839 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:50:09.232817    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbb2t\" (UniqueName: \"kubernetes.io/projected/5b826d19-52f6-4b9b-81b8-31ef30f6d798-kube-api-access-bbb2t\") pod \"autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-rhmxf\" (UID: \"5b826d19-52f6-4b9b-81b8-31ef30f6d798\") " pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-rhmxf"
Jul 28 10:50:09.307118 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:50:09.307076    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-rhmxf"
Jul 28 10:50:09.456678 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:50:09.456654    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-rhmxf"]
Jul 28 10:50:09.459952 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:50:09.459926    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b826d19_52f6_4b9b_81b8_31ef30f6d798.slice/crio-c3a0ca5c71fcb1426b514ae8cf51ba0349999d4bf34529cad7f31eeef61628e3 WatchSource:0}: Error finding container c3a0ca5c71fcb1426b514ae8cf51ba0349999d4bf34529cad7f31eeef61628e3: Status 404 returned error can't find the container with id c3a0ca5c71fcb1426b514ae8cf51ba0349999d4bf34529cad7f31eeef61628e3
Jul 28 10:50:10.404727 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:50:10.404689    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-rhmxf" event={"ID":"5b826d19-52f6-4b9b-81b8-31ef30f6d798","Type":"ContainerStarted","Data":"f10301e125a4448c6979ac9f2a3b93f5a4864dfa085dab863b5ea3f49df134d8"}
Jul 28 10:50:10.405141 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:50:10.404745    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-rhmxf" event={"ID":"5b826d19-52f6-4b9b-81b8-31ef30f6d798","Type":"ContainerStarted","Data":"c3a0ca5c71fcb1426b514ae8cf51ba0349999d4bf34529cad7f31eeef61628e3"}
Jul 28 10:50:10.405141 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:50:10.404810    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-rhmxf"
Jul 28 10:50:10.421936 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:50:10.421882    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-rhmxf" podStartSLOduration=2.421864646 podStartE2EDuration="2.421864646s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:50:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:50:10.420481541 +0000 UTC m=+2544.683877872" watchObservedRunningTime="2026-07-28 10:50:10.421864646 +0000 UTC m=+2544.685260975"
Jul 28 10:50:19.117626 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:50:19.117572    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-84cbcdc776-7dr8g" podUID="37f3723b-934d-4950-8853-b809bdb3764f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused"
Jul 28 10:50:21.027855 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:50:21.027815    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:50:28.036858 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:50:28.036821    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:50:29.116921 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:50:29.116880    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-84cbcdc776-7dr8g" podUID="37f3723b-934d-4950-8853-b809bdb3764f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused"
Jul 28 10:50:29.596858 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:50:29.596815    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:50:39.117518 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:50:39.117468    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-84cbcdc776-7dr8g" podUID="37f3723b-934d-4950-8853-b809bdb3764f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused"
Jul 28 10:50:41.411582 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:50:41.411525    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-rhmxf"
Jul 28 10:50:42.471308 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:50:42.471270    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:50:49.116906 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:50:49.116856    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-84cbcdc776-7dr8g" podUID="37f3723b-934d-4950-8853-b809bdb3764f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused"
Jul 28 10:50:59.116597 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:50:59.116531    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-84cbcdc776-7dr8g" podUID="37f3723b-934d-4950-8853-b809bdb3764f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused"
Jul 28 10:51:00.288418 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:51:00.288382    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:51:09.117529 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:51:09.117413    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-84cbcdc776-7dr8g" podUID="37f3723b-934d-4950-8853-b809bdb3764f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused"
Jul 28 10:51:09.140147 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:51:09.140108    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:51:19.117032 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:51:19.116976    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-84cbcdc776-7dr8g" podUID="37f3723b-934d-4950-8853-b809bdb3764f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused"
Jul 28 10:51:29.126511 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:51:29.126476    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-84cbcdc776-7dr8g"
Jul 28 10:51:29.134608 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:51:29.134575    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-84cbcdc776-7dr8g"
Jul 28 10:51:31.378927 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:51:31.378890    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-84cbcdc776-7dr8g"]
Jul 28 10:51:31.391429 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:51:31.391393    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:51:31.489127 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:51:31.489092    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:51:46.402213 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:51:46.402151    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-84cbcdc776-7dr8g" podUID="37f3723b-934d-4950-8853-b809bdb3764f" containerName="main" containerID="cri-o://32367250f78d110bc1d287684b82b58d2789c584452f546fb2c25126da6ac509" gracePeriod=45
Jul 28 10:51:47.052283 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:51:47.052254    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-84cbcdc776-7dr8g"
Jul 28 10:51:47.071456 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:51:47.071423    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/37f3723b-934d-4950-8853-b809bdb3764f-cabundle-cert\") pod \"37f3723b-934d-4950-8853-b809bdb3764f\" (UID: \"37f3723b-934d-4950-8853-b809bdb3764f\") "
Jul 28 10:51:47.071456 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:51:47.071457    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37f3723b-934d-4950-8853-b809bdb3764f-dshm\") pod \"37f3723b-934d-4950-8853-b809bdb3764f\" (UID: \"37f3723b-934d-4950-8853-b809bdb3764f\") "
Jul 28 10:51:47.071704 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:51:47.071479    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37f3723b-934d-4950-8853-b809bdb3764f-tmp-dir\") pod \"37f3723b-934d-4950-8853-b809bdb3764f\" (UID: \"37f3723b-934d-4950-8853-b809bdb3764f\") "
Jul 28 10:51:47.071704 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:51:47.071497    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37f3723b-934d-4950-8853-b809bdb3764f-tls-certs\") pod \"37f3723b-934d-4950-8853-b809bdb3764f\" (UID: \"37f3723b-934d-4950-8853-b809bdb3764f\") "
Jul 28 10:51:47.071704 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:51:47.071512    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/37f3723b-934d-4950-8853-b809bdb3764f-kserve-provision-location\") pod \"37f3723b-934d-4950-8853-b809bdb3764f\" (UID: \"37f3723b-934d-4950-8853-b809bdb3764f\") "
Jul 28 10:51:47.071704 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:51:47.071555    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37f3723b-934d-4950-8853-b809bdb3764f-model-cache\") pod \"37f3723b-934d-4950-8853-b809bdb3764f\" (UID: \"37f3723b-934d-4950-8853-b809bdb3764f\") "
Jul 28 10:51:47.071704 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:51:47.071583    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kfwr7\" (UniqueName: \"kubernetes.io/projected/37f3723b-934d-4950-8853-b809bdb3764f-kube-api-access-kfwr7\") pod \"37f3723b-934d-4950-8853-b809bdb3764f\" (UID: \"37f3723b-934d-4950-8853-b809bdb3764f\") "
Jul 28 10:51:47.071704 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:51:47.071670    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37f3723b-934d-4950-8853-b809bdb3764f-home\") pod \"37f3723b-934d-4950-8853-b809bdb3764f\" (UID: \"37f3723b-934d-4950-8853-b809bdb3764f\") "
Jul 28 10:51:47.072013 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:51:47.071823    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37f3723b-934d-4950-8853-b809bdb3764f-model-cache" (OuterVolumeSpecName: "model-cache") pod "37f3723b-934d-4950-8853-b809bdb3764f" (UID: "37f3723b-934d-4950-8853-b809bdb3764f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:51:47.072013 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:51:47.071881    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37f3723b-934d-4950-8853-b809bdb3764f-model-cache\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 10:51:47.072013 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:51:47.071903    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37f3723b-934d-4950-8853-b809bdb3764f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "37f3723b-934d-4950-8853-b809bdb3764f" (UID: "37f3723b-934d-4950-8853-b809bdb3764f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:51:47.072165 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:51:47.072106    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37f3723b-934d-4950-8853-b809bdb3764f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "37f3723b-934d-4950-8853-b809bdb3764f" (UID: "37f3723b-934d-4950-8853-b809bdb3764f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:51:47.072165 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:51:47.072159    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37f3723b-934d-4950-8853-b809bdb3764f-home" (OuterVolumeSpecName: "home") pod "37f3723b-934d-4950-8853-b809bdb3764f" (UID: "37f3723b-934d-4950-8853-b809bdb3764f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:51:47.073836 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:51:47.073814    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37f3723b-934d-4950-8853-b809bdb3764f-dshm" (OuterVolumeSpecName: "dshm") pod "37f3723b-934d-4950-8853-b809bdb3764f" (UID: "37f3723b-934d-4950-8853-b809bdb3764f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:51:47.073947 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:51:47.073838    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37f3723b-934d-4950-8853-b809bdb3764f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "37f3723b-934d-4950-8853-b809bdb3764f" (UID: "37f3723b-934d-4950-8853-b809bdb3764f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:51:47.073993 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:51:47.073952    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37f3723b-934d-4950-8853-b809bdb3764f-kube-api-access-kfwr7" (OuterVolumeSpecName: "kube-api-access-kfwr7") pod "37f3723b-934d-4950-8853-b809bdb3764f" (UID: "37f3723b-934d-4950-8853-b809bdb3764f"). InnerVolumeSpecName "kube-api-access-kfwr7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:51:47.129805 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:51:47.129757    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37f3723b-934d-4950-8853-b809bdb3764f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "37f3723b-934d-4950-8853-b809bdb3764f" (UID: "37f3723b-934d-4950-8853-b809bdb3764f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:51:47.172769 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:51:47.172692    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37f3723b-934d-4950-8853-b809bdb3764f-home\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 10:51:47.172769 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:51:47.172724    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/37f3723b-934d-4950-8853-b809bdb3764f-cabundle-cert\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 10:51:47.172769 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:51:47.172735    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37f3723b-934d-4950-8853-b809bdb3764f-dshm\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 10:51:47.172769 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:51:47.172763    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37f3723b-934d-4950-8853-b809bdb3764f-tmp-dir\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 10:51:47.172769 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:51:47.172773    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37f3723b-934d-4950-8853-b809bdb3764f-tls-certs\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 10:51:47.173028 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:51:47.172782    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/37f3723b-934d-4950-8853-b809bdb3764f-kserve-provision-location\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 10:51:47.173028 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:51:47.172791    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kfwr7\" (UniqueName: \"kubernetes.io/projected/37f3723b-934d-4950-8853-b809bdb3764f-kube-api-access-kfwr7\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 10:51:47.926446 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:51:47.926415    2579 generic.go:356] "Generic (PLEG): container finished" podID="37f3723b-934d-4950-8853-b809bdb3764f" containerID="32367250f78d110bc1d287684b82b58d2789c584452f546fb2c25126da6ac509" exitCode=0
Jul 28 10:51:47.926873 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:51:47.926474    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-84cbcdc776-7dr8g"
Jul 28 10:51:47.926873 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:51:47.926496    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-84cbcdc776-7dr8g" event={"ID":"37f3723b-934d-4950-8853-b809bdb3764f","Type":"ContainerDied","Data":"32367250f78d110bc1d287684b82b58d2789c584452f546fb2c25126da6ac509"}
Jul 28 10:51:47.926873 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:51:47.926574    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-84cbcdc776-7dr8g" event={"ID":"37f3723b-934d-4950-8853-b809bdb3764f","Type":"ContainerDied","Data":"b7d4f1ab0048b6dff7c1137e092fd401a80af9cb09cc991ef78edcbf5e742b78"}
Jul 28 10:51:47.926873 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:51:47.926599    2579 scope.go:117] "RemoveContainer" containerID="32367250f78d110bc1d287684b82b58d2789c584452f546fb2c25126da6ac509"
Jul 28 10:51:47.935806 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:51:47.935776    2579 scope.go:117] "RemoveContainer" containerID="2dc050fd7a7283316952dc8b17119db1756fc7d782df539f961ef72c1e067f14"
Jul 28 10:51:47.944327 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:51:47.944309    2579 scope.go:117] "RemoveContainer" containerID="32367250f78d110bc1d287684b82b58d2789c584452f546fb2c25126da6ac509"
Jul 28 10:51:47.944587 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:51:47.944568    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32367250f78d110bc1d287684b82b58d2789c584452f546fb2c25126da6ac509\": container with ID starting with 32367250f78d110bc1d287684b82b58d2789c584452f546fb2c25126da6ac509 not found: ID does not exist" containerID="32367250f78d110bc1d287684b82b58d2789c584452f546fb2c25126da6ac509"
Jul 28 10:51:47.944665 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:51:47.944601    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32367250f78d110bc1d287684b82b58d2789c584452f546fb2c25126da6ac509"} err="failed to get container status \"32367250f78d110bc1d287684b82b58d2789c584452f546fb2c25126da6ac509\": rpc error: code = NotFound desc = could not find container \"32367250f78d110bc1d287684b82b58d2789c584452f546fb2c25126da6ac509\": container with ID starting with 32367250f78d110bc1d287684b82b58d2789c584452f546fb2c25126da6ac509 not found: ID does not exist"
Jul 28 10:51:47.944665 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:51:47.944626    2579 scope.go:117] "RemoveContainer" containerID="2dc050fd7a7283316952dc8b17119db1756fc7d782df539f961ef72c1e067f14"
Jul 28 10:51:47.944863 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:51:47.944845    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2dc050fd7a7283316952dc8b17119db1756fc7d782df539f961ef72c1e067f14\": container with ID starting with 2dc050fd7a7283316952dc8b17119db1756fc7d782df539f961ef72c1e067f14 not found: ID does not exist" containerID="2dc050fd7a7283316952dc8b17119db1756fc7d782df539f961ef72c1e067f14"
Jul 28 10:51:47.944903 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:51:47.944870    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2dc050fd7a7283316952dc8b17119db1756fc7d782df539f961ef72c1e067f14"} err="failed to get container status \"2dc050fd7a7283316952dc8b17119db1756fc7d782df539f961ef72c1e067f14\": rpc error: code = NotFound desc = could not find container \"2dc050fd7a7283316952dc8b17119db1756fc7d782df539f961ef72c1e067f14\": container with ID starting with 2dc050fd7a7283316952dc8b17119db1756fc7d782df539f961ef72c1e067f14 not found: ID does not exist"
Jul 28 10:51:47.948071 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:51:47.948051    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-84cbcdc776-7dr8g"]
Jul 28 10:51:47.951595 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:51:47.951574    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-84cbcdc776-7dr8g"]
Jul 28 10:51:48.276252 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:51:48.276170    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="37f3723b-934d-4950-8853-b809bdb3764f" path="/var/lib/kubelet/pods/37f3723b-934d-4950-8853-b809bdb3764f/volumes"
Jul 28 10:51:51.248701 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:51:51.248668    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:51:51.349431 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:51:51.349399    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:51:51.450157 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:51:51.450114    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:51:51.549690 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:51:51.549658    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:51:51.651670 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:51:51.651635    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:51:53.918638 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:51:53.918600    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:51:53.971291 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:51:53.971254    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:51:54.613376 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:51:54.613336    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:51:57.714123 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:51:57.714076    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:52:03.242464 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:52:03.242426    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:52:03.307631 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:52:03.307598    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:52:16.373275 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:52:16.373243    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:52:16.535122 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:52:16.535089    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schekbp2p"]
Jul 28 10:52:16.535408 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:52:16.535396    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37f3723b-934d-4950-8853-b809bdb3764f" containerName="main"
Jul 28 10:52:16.535455 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:52:16.535410    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="37f3723b-934d-4950-8853-b809bdb3764f" containerName="main"
Jul 28 10:52:16.535455 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:52:16.535428    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37f3723b-934d-4950-8853-b809bdb3764f" containerName="storage-initializer"
Jul 28 10:52:16.535455 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:52:16.535433    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="37f3723b-934d-4950-8853-b809bdb3764f" containerName="storage-initializer"
Jul 28 10:52:16.535589 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:52:16.535494    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="37f3723b-934d-4950-8853-b809bdb3764f" containerName="main"
Jul 28 10:52:16.538372 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:52:16.538354    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schekbp2p"
Jul 28 10:52:16.540441 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:52:16.540419    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cc472ec3\"/\"openshift-service-ca.crt\""
Jul 28 10:52:16.541006 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:52:16.540982    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cc472ec3\"/\"llmisv8bf079eb6eda4debfb4ef5bb7817824c-kserve-self-signed-certs\""
Jul 28 10:52:16.541118 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:52:16.541049    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cc472ec3\"/\"kube-root-ca.crt\""
Jul 28 10:52:16.541118 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:52:16.541066    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cc472ec3\"/\"default-dockercfg-khtqv\""
Jul 28 10:52:16.541232 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:52:16.541125    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cc472ec3\"/\"llmisvc-model-pvc-router-manage-59b9d263-epp-sa-dockercfg-hvxvq\""
Jul 28 10:52:16.546280 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:52:16.546256    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schekbp2p"]
Jul 28 10:52:16.562185 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:52:16.562158    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:52:16.614640 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:52:16.614601    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24qk7\" (UniqueName: \"kubernetes.io/projected/3ef3b2bf-8132-4a9a-8215-0c7e39b337e9-kube-api-access-24qk7\") pod \"llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schekbp2p\" (UID: \"3ef3b2bf-8132-4a9a-8215-0c7e39b337e9\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schekbp2p"
Jul 28 10:52:16.614816 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:52:16.614665    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ef3b2bf-8132-4a9a-8215-0c7e39b337e9-tls-certs\") pod \"llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schekbp2p\" (UID: \"3ef3b2bf-8132-4a9a-8215-0c7e39b337e9\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schekbp2p"
Jul 28 10:52:16.715621 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:52:16.715508    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24qk7\" (UniqueName: \"kubernetes.io/projected/3ef3b2bf-8132-4a9a-8215-0c7e39b337e9-kube-api-access-24qk7\") pod \"llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schekbp2p\" (UID: \"3ef3b2bf-8132-4a9a-8215-0c7e39b337e9\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schekbp2p"
Jul 28 10:52:16.715621 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:52:16.715612    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ef3b2bf-8132-4a9a-8215-0c7e39b337e9-tls-certs\") pod \"llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schekbp2p\" (UID: \"3ef3b2bf-8132-4a9a-8215-0c7e39b337e9\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schekbp2p"
Jul 28 10:52:16.718158 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:52:16.718123    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ef3b2bf-8132-4a9a-8215-0c7e39b337e9-tls-certs\") pod \"llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schekbp2p\" (UID: \"3ef3b2bf-8132-4a9a-8215-0c7e39b337e9\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schekbp2p"
Jul 28 10:52:16.724414 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:52:16.724381    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-24qk7\" (UniqueName: \"kubernetes.io/projected/3ef3b2bf-8132-4a9a-8215-0c7e39b337e9-kube-api-access-24qk7\") pod \"llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schekbp2p\" (UID: \"3ef3b2bf-8132-4a9a-8215-0c7e39b337e9\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schekbp2p"
Jul 28 10:52:16.849342 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:52:16.849305    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schekbp2p"
Jul 28 10:52:16.988257 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:52:16.988231    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schekbp2p"]
Jul 28 10:52:16.990083 ip-10-0-141-50 kubenswrapper[2579]: W0728 10:52:16.990059    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ef3b2bf_8132_4a9a_8215_0c7e39b337e9.slice/crio-2f871282c1aee0d562364ac44394fdf6e93189c6b48ce5eb17d82b0c293cf055 WatchSource:0}: Error finding container 2f871282c1aee0d562364ac44394fdf6e93189c6b48ce5eb17d82b0c293cf055: Status 404 returned error can't find the container with id 2f871282c1aee0d562364ac44394fdf6e93189c6b48ce5eb17d82b0c293cf055
Jul 28 10:52:17.074027 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:52:17.073996    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schekbp2p" event={"ID":"3ef3b2bf-8132-4a9a-8215-0c7e39b337e9","Type":"ContainerStarted","Data":"6dfdb0793a9b0063438bb6e83ed2181e66430b02d200ec23021b78c5028356ae"}
Jul 28 10:52:17.074219 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:52:17.074045    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schekbp2p" event={"ID":"3ef3b2bf-8132-4a9a-8215-0c7e39b337e9","Type":"ContainerStarted","Data":"2f871282c1aee0d562364ac44394fdf6e93189c6b48ce5eb17d82b0c293cf055"}
Jul 28 10:52:17.074219 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:52:17.074086    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schekbp2p"
Jul 28 10:52:17.093025 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:52:17.092929    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schekbp2p" podStartSLOduration=1.092907706 podStartE2EDuration="1.092907706s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:52:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:52:17.091713495 +0000 UTC m=+2671.355109824" watchObservedRunningTime="2026-07-28 10:52:17.092907706 +0000 UTC m=+2671.356304035"
Jul 28 10:52:29.513964 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:52:29.513932    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:52:48.081469 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:52:48.081439    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schekbp2p"
Jul 28 10:54:25.760061 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:54:25.759967    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schekbp2p"]
Jul 28 10:54:25.807718 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:54:25.807677    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:54:25.874453 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:54:25.874420    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:54:28.081284 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:54:28.081243    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schekbp2p" podUID="3ef3b2bf-8132-4a9a-8215-0c7e39b337e9" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 10:54:38.081242 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:54:38.081206    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schekbp2p" podUID="3ef3b2bf-8132-4a9a-8215-0c7e39b337e9" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 10:54:40.780185 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:54:40.780119    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schekbp2p" podUID="3ef3b2bf-8132-4a9a-8215-0c7e39b337e9" containerName="main" containerID="cri-o://6dfdb0793a9b0063438bb6e83ed2181e66430b02d200ec23021b78c5028356ae" gracePeriod=45
Jul 28 10:54:41.035102 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:54:41.035075    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schekbp2p"
Jul 28 10:54:41.190375 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:54:41.190337    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-24qk7\" (UniqueName: \"kubernetes.io/projected/3ef3b2bf-8132-4a9a-8215-0c7e39b337e9-kube-api-access-24qk7\") pod \"3ef3b2bf-8132-4a9a-8215-0c7e39b337e9\" (UID: \"3ef3b2bf-8132-4a9a-8215-0c7e39b337e9\") "
Jul 28 10:54:41.190375 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:54:41.190386    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ef3b2bf-8132-4a9a-8215-0c7e39b337e9-tls-certs\") pod \"3ef3b2bf-8132-4a9a-8215-0c7e39b337e9\" (UID: \"3ef3b2bf-8132-4a9a-8215-0c7e39b337e9\") "
Jul 28 10:54:41.192700 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:54:41.192665    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ef3b2bf-8132-4a9a-8215-0c7e39b337e9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3ef3b2bf-8132-4a9a-8215-0c7e39b337e9" (UID: "3ef3b2bf-8132-4a9a-8215-0c7e39b337e9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:54:41.192700 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:54:41.192690    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ef3b2bf-8132-4a9a-8215-0c7e39b337e9-kube-api-access-24qk7" (OuterVolumeSpecName: "kube-api-access-24qk7") pod "3ef3b2bf-8132-4a9a-8215-0c7e39b337e9" (UID: "3ef3b2bf-8132-4a9a-8215-0c7e39b337e9"). InnerVolumeSpecName "kube-api-access-24qk7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:54:41.291193 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:54:41.291099    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24qk7\" (UniqueName: \"kubernetes.io/projected/3ef3b2bf-8132-4a9a-8215-0c7e39b337e9-kube-api-access-24qk7\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 10:54:41.291193 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:54:41.291131    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ef3b2bf-8132-4a9a-8215-0c7e39b337e9-tls-certs\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 10:54:41.766970 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:54:41.766881    2579 generic.go:356] "Generic (PLEG): container finished" podID="3ef3b2bf-8132-4a9a-8215-0c7e39b337e9" containerID="6dfdb0793a9b0063438bb6e83ed2181e66430b02d200ec23021b78c5028356ae" exitCode=0
Jul 28 10:54:41.766970 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:54:41.766928    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schekbp2p"
Jul 28 10:54:41.767154 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:54:41.766967    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schekbp2p" event={"ID":"3ef3b2bf-8132-4a9a-8215-0c7e39b337e9","Type":"ContainerDied","Data":"6dfdb0793a9b0063438bb6e83ed2181e66430b02d200ec23021b78c5028356ae"}
Jul 28 10:54:41.767154 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:54:41.767027    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schekbp2p" event={"ID":"3ef3b2bf-8132-4a9a-8215-0c7e39b337e9","Type":"ContainerDied","Data":"2f871282c1aee0d562364ac44394fdf6e93189c6b48ce5eb17d82b0c293cf055"}
Jul 28 10:54:41.767154 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:54:41.767049    2579 scope.go:117] "RemoveContainer" containerID="6dfdb0793a9b0063438bb6e83ed2181e66430b02d200ec23021b78c5028356ae"
Jul 28 10:54:41.776679 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:54:41.776658    2579 scope.go:117] "RemoveContainer" containerID="6dfdb0793a9b0063438bb6e83ed2181e66430b02d200ec23021b78c5028356ae"
Jul 28 10:54:41.776958 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:54:41.776940    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6dfdb0793a9b0063438bb6e83ed2181e66430b02d200ec23021b78c5028356ae\": container with ID starting with 6dfdb0793a9b0063438bb6e83ed2181e66430b02d200ec23021b78c5028356ae not found: ID does not exist" containerID="6dfdb0793a9b0063438bb6e83ed2181e66430b02d200ec23021b78c5028356ae"
Jul 28 10:54:41.777018 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:54:41.776965    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6dfdb0793a9b0063438bb6e83ed2181e66430b02d200ec23021b78c5028356ae"} err="failed to get container status \"6dfdb0793a9b0063438bb6e83ed2181e66430b02d200ec23021b78c5028356ae\": rpc error: code = NotFound desc = could not find container \"6dfdb0793a9b0063438bb6e83ed2181e66430b02d200ec23021b78c5028356ae\": container with ID starting with 6dfdb0793a9b0063438bb6e83ed2181e66430b02d200ec23021b78c5028356ae not found: ID does not exist"
Jul 28 10:54:41.788370 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:54:41.788341    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schekbp2p"]
Jul 28 10:54:41.791909 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:54:41.791877    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schekbp2p"]
Jul 28 10:54:42.276119 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:54:42.276076    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3ef3b2bf-8132-4a9a-8215-0c7e39b337e9" path="/var/lib/kubelet/pods/3ef3b2bf-8132-4a9a-8215-0c7e39b337e9/volumes"
Jul 28 10:54:45.663056 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:54:45.663012    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:54:45.757635 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:54:45.757602    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:54:45.862322 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:54:45.862281    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:54:45.962620 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:54:45.962511    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:54:46.029464 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:54:46.029426    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:54:48.153511 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:54:48.153478    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:54:48.254758 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:54:48.254723    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:54:48.969077 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:54:48.969042    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:54:51.507781 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:54:51.507728    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:54:51.608381 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:54:51.608348    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:54:51.727834 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:54:51.727797    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:54:51.828782 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:54:51.828750    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:54:51.884217 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:54:51.884178    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:54:57.631161 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:54:57.631123    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:54:57.697141 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:54:57.697108    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:55:19.156701 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:55:19.156667    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bgw9ln"]
Jul 28 10:55:19.157095 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:55:19.157016    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ef3b2bf-8132-4a9a-8215-0c7e39b337e9" containerName="main"
Jul 28 10:55:19.157095 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:55:19.157028    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ef3b2bf-8132-4a9a-8215-0c7e39b337e9" containerName="main"
Jul 28 10:55:19.157095 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:55:19.157087    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ef3b2bf-8132-4a9a-8215-0c7e39b337e9" containerName="main"
Jul 28 10:55:19.160173 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:55:19.160152    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bgw9ln"
Jul 28 10:55:19.162370 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:55:19.162345    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-48639af5\"/\"openshift-service-ca.crt\""
Jul 28 10:55:19.162504 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:55:19.162450    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-48639af5\"/\"kube-root-ca.crt\""
Jul 28 10:55:19.162504 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:55:19.162456    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-48639af5\"/\"llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs\""
Jul 28 10:55:19.162504 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:55:19.162492    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-48639af5\"/\"default-dockercfg-s8znt\""
Jul 28 10:55:19.173233 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:55:19.173201    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bgw9ln"]
Jul 28 10:55:19.179770 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:55:19.179745    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f2f8d3c-6c27-4e8b-92c4-d22da1bb1cbe-dshm\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bgw9ln\" (UID: \"9f2f8d3c-6c27-4e8b-92c4-d22da1bb1cbe\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bgw9ln"
Jul 28 10:55:19.179906 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:55:19.179808    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f2f8d3c-6c27-4e8b-92c4-d22da1bb1cbe-home\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bgw9ln\" (UID: \"9f2f8d3c-6c27-4e8b-92c4-d22da1bb1cbe\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bgw9ln"
Jul 28 10:55:19.179906 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:55:19.179829    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f2f8d3c-6c27-4e8b-92c4-d22da1bb1cbe-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bgw9ln\" (UID: \"9f2f8d3c-6c27-4e8b-92c4-d22da1bb1cbe\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bgw9ln"
Jul 28 10:55:19.179906 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:55:19.179845    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f2f8d3c-6c27-4e8b-92c4-d22da1bb1cbe-model-cache\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bgw9ln\" (UID: \"9f2f8d3c-6c27-4e8b-92c4-d22da1bb1cbe\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bgw9ln"
Jul 28 10:55:19.179906 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:55:19.179863    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f2f8d3c-6c27-4e8b-92c4-d22da1bb1cbe-tls-certs\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bgw9ln\" (UID: \"9f2f8d3c-6c27-4e8b-92c4-d22da1bb1cbe\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bgw9ln"
Jul 28 10:55:19.194766 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:55:19.194735    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:55:19.280928 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:55:19.280887    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f2f8d3c-6c27-4e8b-92c4-d22da1bb1cbe-dshm\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bgw9ln\" (UID: \"9f2f8d3c-6c27-4e8b-92c4-d22da1bb1cbe\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bgw9ln"
Jul 28 10:55:19.281140 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:55:19.280943    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f2f8d3c-6c27-4e8b-92c4-d22da1bb1cbe-home\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bgw9ln\" (UID: \"9f2f8d3c-6c27-4e8b-92c4-d22da1bb1cbe\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bgw9ln"
Jul 28 10:55:19.281140 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:55:19.280965    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f2f8d3c-6c27-4e8b-92c4-d22da1bb1cbe-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bgw9ln\" (UID: \"9f2f8d3c-6c27-4e8b-92c4-d22da1bb1cbe\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bgw9ln"
Jul 28 10:55:19.281140 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:55:19.280997    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f2f8d3c-6c27-4e8b-92c4-d22da1bb1cbe-model-cache\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bgw9ln\" (UID: \"9f2f8d3c-6c27-4e8b-92c4-d22da1bb1cbe\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bgw9ln"
Jul 28 10:55:19.281140 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:55:19.281026    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f2f8d3c-6c27-4e8b-92c4-d22da1bb1cbe-tls-certs\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bgw9ln\" (UID: \"9f2f8d3c-6c27-4e8b-92c4-d22da1bb1cbe\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bgw9ln"
Jul 28 10:55:19.281140 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:55:19.281070    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fnfl\" (UniqueName: \"kubernetes.io/projected/9f2f8d3c-6c27-4e8b-92c4-d22da1bb1cbe-kube-api-access-8fnfl\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bgw9ln\" (UID: \"9f2f8d3c-6c27-4e8b-92c4-d22da1bb1cbe\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bgw9ln"
Jul 28 10:55:19.281481 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:55:19.281453    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f2f8d3c-6c27-4e8b-92c4-d22da1bb1cbe-home\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bgw9ln\" (UID: \"9f2f8d3c-6c27-4e8b-92c4-d22da1bb1cbe\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bgw9ln"
Jul 28 10:55:19.281672 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:55:19.281508    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f2f8d3c-6c27-4e8b-92c4-d22da1bb1cbe-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bgw9ln\" (UID: \"9f2f8d3c-6c27-4e8b-92c4-d22da1bb1cbe\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bgw9ln"
Jul 28 10:55:19.281672 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:55:19.281586    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f2f8d3c-6c27-4e8b-92c4-d22da1bb1cbe-model-cache\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bgw9ln\" (UID: \"9f2f8d3c-6c27-4e8b-92c4-d22da1bb1cbe\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bgw9ln"
Jul 28 10:55:19.283396 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:55:19.283376    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f2f8d3c-6c27-4e8b-92c4-d22da1bb1cbe-dshm\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bgw9ln\" (UID: \"9f2f8d3c-6c27-4e8b-92c4-d22da1bb1cbe\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bgw9ln"
Jul 28 10:55:19.284024 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:55:19.284001    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f2f8d3c-6c27-4e8b-92c4-d22da1bb1cbe-tls-certs\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bgw9ln\" (UID: \"9f2f8d3c-6c27-4e8b-92c4-d22da1bb1cbe\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bgw9ln"
Jul 28 10:55:19.382440 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:55:19.382398    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fnfl\" (UniqueName: \"kubernetes.io/projected/9f2f8d3c-6c27-4e8b-92c4-d22da1bb1cbe-kube-api-access-8fnfl\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bgw9ln\" (UID: \"9f2f8d3c-6c27-4e8b-92c4-d22da1bb1cbe\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bgw9ln"
Jul 28 10:55:19.390301 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:55:19.390262    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fnfl\" (UniqueName: \"kubernetes.io/projected/9f2f8d3c-6c27-4e8b-92c4-d22da1bb1cbe-kube-api-access-8fnfl\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bgw9ln\" (UID: \"9f2f8d3c-6c27-4e8b-92c4-d22da1bb1cbe\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bgw9ln"
Jul 28 10:55:19.573498 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:55:19.573468    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:55:29.252600 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:55:29.252564    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:55:29.481853 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:55:29.481818    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:55:29.582316 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:55:29.582279    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:57:22.170725 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:57:22.170630    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bgw9ln" podUID="9f2f8d3c-6c27-4e8b-92c4-d22da1bb1cbe"
Jul 28 10:57:22.535049 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:57:22.535013    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bgw9ln"
Jul 28 10:58:32.998290 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:58:32.998258    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:58:57.279789 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:58:57.279677    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:59:18.561994 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:59:18.561960    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:59:18.660879 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:59:18.660825    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 10:59:25.536213 ip-10-0-141-50 kubenswrapper[2579]: E0728 10:59:25.536167    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bgw9ln" podUID="9f2f8d3c-6c27-4e8b-92c4-d22da1bb1cbe"
Jul 28 10:59:26.110366 ip-10-0-141-50 kubenswrapper[2579]: I0728 10:59:26.110332    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bgw9ln"
Jul 28 11:01:29.111278 ip-10-0-141-50 kubenswrapper[2579]: E0728 11:01:29.111212    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bgw9ln" podUID="9f2f8d3c-6c27-4e8b-92c4-d22da1bb1cbe"
Jul 28 11:01:29.693960 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:01:29.693920    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bgw9ln"
Jul 28 11:03:32.695453 ip-10-0-141-50 kubenswrapper[2579]: E0728 11:03:32.695403    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bgw9ln" podUID="9f2f8d3c-6c27-4e8b-92c4-d22da1bb1cbe"
Jul 28 11:03:33.296727 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:03:33.296695    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bgw9ln"
Jul 28 11:05:06.015594 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:05:06.015486    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-rhmxf"]
Jul 28 11:05:06.939166 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:05:06.939131    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:05:08.685144 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:05:08.685107    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:05:11.411753 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:05:11.411716    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-rhmxf" podUID="5b826d19-52f6-4b9b-81b8-31ef30f6d798" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 11:05:21.034705 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:05:21.034634    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-rhmxf" podUID="5b826d19-52f6-4b9b-81b8-31ef30f6d798" containerName="main" containerID="cri-o://f10301e125a4448c6979ac9f2a3b93f5a4864dfa085dab863b5ea3f49df134d8" gracePeriod=45
Jul 28 11:05:21.288214 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:05:21.288153    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-rhmxf"
Jul 28 11:05:21.343874 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:05:21.343839    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b826d19-52f6-4b9b-81b8-31ef30f6d798-tls-certs\") pod \"5b826d19-52f6-4b9b-81b8-31ef30f6d798\" (UID: \"5b826d19-52f6-4b9b-81b8-31ef30f6d798\") "
Jul 28 11:05:21.344063 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:05:21.343900    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bbb2t\" (UniqueName: \"kubernetes.io/projected/5b826d19-52f6-4b9b-81b8-31ef30f6d798-kube-api-access-bbb2t\") pod \"5b826d19-52f6-4b9b-81b8-31ef30f6d798\" (UID: \"5b826d19-52f6-4b9b-81b8-31ef30f6d798\") "
Jul 28 11:05:21.346158 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:05:21.346129    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b826d19-52f6-4b9b-81b8-31ef30f6d798-kube-api-access-bbb2t" (OuterVolumeSpecName: "kube-api-access-bbb2t") pod "5b826d19-52f6-4b9b-81b8-31ef30f6d798" (UID: "5b826d19-52f6-4b9b-81b8-31ef30f6d798"). InnerVolumeSpecName "kube-api-access-bbb2t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 11:05:21.346158 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:05:21.346148    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b826d19-52f6-4b9b-81b8-31ef30f6d798-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5b826d19-52f6-4b9b-81b8-31ef30f6d798" (UID: "5b826d19-52f6-4b9b-81b8-31ef30f6d798"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 11:05:21.445049 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:05:21.445010    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b826d19-52f6-4b9b-81b8-31ef30f6d798-tls-certs\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 11:05:21.445049 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:05:21.445043    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbb2t\" (UniqueName: \"kubernetes.io/projected/5b826d19-52f6-4b9b-81b8-31ef30f6d798-kube-api-access-bbb2t\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 11:05:21.816661 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:05:21.816630    2579 generic.go:356] "Generic (PLEG): container finished" podID="5b826d19-52f6-4b9b-81b8-31ef30f6d798" containerID="f10301e125a4448c6979ac9f2a3b93f5a4864dfa085dab863b5ea3f49df134d8" exitCode=0
Jul 28 11:05:21.816837 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:05:21.816674    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-rhmxf"
Jul 28 11:05:21.816837 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:05:21.816714    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-rhmxf" event={"ID":"5b826d19-52f6-4b9b-81b8-31ef30f6d798","Type":"ContainerDied","Data":"f10301e125a4448c6979ac9f2a3b93f5a4864dfa085dab863b5ea3f49df134d8"}
Jul 28 11:05:21.816837 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:05:21.816765    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-rhmxf" event={"ID":"5b826d19-52f6-4b9b-81b8-31ef30f6d798","Type":"ContainerDied","Data":"c3a0ca5c71fcb1426b514ae8cf51ba0349999d4bf34529cad7f31eeef61628e3"}
Jul 28 11:05:21.816837 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:05:21.816789    2579 scope.go:117] "RemoveContainer" containerID="f10301e125a4448c6979ac9f2a3b93f5a4864dfa085dab863b5ea3f49df134d8"
Jul 28 11:05:21.826075 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:05:21.826058    2579 scope.go:117] "RemoveContainer" containerID="f10301e125a4448c6979ac9f2a3b93f5a4864dfa085dab863b5ea3f49df134d8"
Jul 28 11:05:21.826317 ip-10-0-141-50 kubenswrapper[2579]: E0728 11:05:21.826299    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f10301e125a4448c6979ac9f2a3b93f5a4864dfa085dab863b5ea3f49df134d8\": container with ID starting with f10301e125a4448c6979ac9f2a3b93f5a4864dfa085dab863b5ea3f49df134d8 not found: ID does not exist" containerID="f10301e125a4448c6979ac9f2a3b93f5a4864dfa085dab863b5ea3f49df134d8"
Jul 28 11:05:21.826362 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:05:21.826325    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f10301e125a4448c6979ac9f2a3b93f5a4864dfa085dab863b5ea3f49df134d8"} err="failed to get container status \"f10301e125a4448c6979ac9f2a3b93f5a4864dfa085dab863b5ea3f49df134d8\": rpc error: code = NotFound desc = could not find container \"f10301e125a4448c6979ac9f2a3b93f5a4864dfa085dab863b5ea3f49df134d8\": container with ID starting with f10301e125a4448c6979ac9f2a3b93f5a4864dfa085dab863b5ea3f49df134d8 not found: ID does not exist"
Jul 28 11:05:21.838783 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:05:21.838760    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-rhmxf"]
Jul 28 11:05:21.840518 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:05:21.840501    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-rhmxf"]
Jul 28 11:05:22.275478 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:05:22.275400    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b826d19-52f6-4b9b-81b8-31ef30f6d798" path="/var/lib/kubelet/pods/5b826d19-52f6-4b9b-81b8-31ef30f6d798/volumes"
Jul 28 11:05:24.307577 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:05:24.307517    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:05:24.405448 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:05:24.405409    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:05:36.297424 ip-10-0-141-50 kubenswrapper[2579]: E0728 11:05:36.297373    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bgw9ln" podUID="9f2f8d3c-6c27-4e8b-92c4-d22da1bb1cbe"
Jul 28 11:05:36.891005 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:05:36.890972    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bgw9ln"
Jul 28 11:05:46.829017 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:05:46.828974    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:05:46.944182 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:05:46.944099    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:05:47.039971 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:05:47.039938    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:05:47.050999 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:05:47.050968    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"]
Jul 28 11:05:47.051305 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:05:47.051293    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b826d19-52f6-4b9b-81b8-31ef30f6d798" containerName="main"
Jul 28 11:05:47.051351 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:05:47.051306    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b826d19-52f6-4b9b-81b8-31ef30f6d798" containerName="main"
Jul 28 11:05:47.051386 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:05:47.051360    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b826d19-52f6-4b9b-81b8-31ef30f6d798" containerName="main"
Jul 28 11:05:47.055346 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:05:47.055329    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 28 11:05:47.062269 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:05:47.062246    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2\"/\"autoscale-hpa-lws-kserve-mn-dockercfg-79sbd\""
Jul 28 11:05:47.062401 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:05:47.062282    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2\"/\"openshift-service-ca.crt\""
Jul 28 11:05:47.062401 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:05:47.062299    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2\"/\"default-dockercfg-m7rhc\""
Jul 28 11:05:47.062401 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:05:47.062299    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2\"/\"autoscale-hpa-lws-kserve-self-signed-certs\""
Jul 28 11:05:47.062401 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:05:47.062355    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2\"/\"kube-root-ca.crt\""
Jul 28 11:05:47.075404 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:05:47.075369    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"]
Jul 28 11:05:47.160280 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:05:47.160238    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44f19bea-8b8e-4dce-8ac9-c801d3d1cd28-tmp-dir\") pod \"autoscale-hpa-lws-kserve-mn-0-1\" (UID: \"44f19bea-8b8e-4dce-8ac9-c801d3d1cd28\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 28 11:05:47.160493 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:05:47.160315    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44f19bea-8b8e-4dce-8ac9-c801d3d1cd28-tls-certs\") pod \"autoscale-hpa-lws-kserve-mn-0-1\" (UID: \"44f19bea-8b8e-4dce-8ac9-c801d3d1cd28\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 28 11:05:47.160493 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:05:47.160395    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44f19bea-8b8e-4dce-8ac9-c801d3d1cd28-dshm\") pod \"autoscale-hpa-lws-kserve-mn-0-1\" (UID: \"44f19bea-8b8e-4dce-8ac9-c801d3d1cd28\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 28 11:05:47.160493 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:05:47.160420    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44f19bea-8b8e-4dce-8ac9-c801d3d1cd28-model-cache\") pod \"autoscale-hpa-lws-kserve-mn-0-1\" (UID: \"44f19bea-8b8e-4dce-8ac9-c801d3d1cd28\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 28 11:05:47.160493 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:05:47.160446    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqlbx\" (UniqueName: \"kubernetes.io/projected/44f19bea-8b8e-4dce-8ac9-c801d3d1cd28-kube-api-access-pqlbx\") pod \"autoscale-hpa-lws-kserve-mn-0-1\" (UID: \"44f19bea-8b8e-4dce-8ac9-c801d3d1cd28\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 28 11:05:47.160493 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:05:47.160481    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44f19bea-8b8e-4dce-8ac9-c801d3d1cd28-home\") pod \"autoscale-hpa-lws-kserve-mn-0-1\" (UID: \"44f19bea-8b8e-4dce-8ac9-c801d3d1cd28\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 28 11:05:47.220879 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:05:47.220762    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:05:47.262044 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:05:47.261978    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44f19bea-8b8e-4dce-8ac9-c801d3d1cd28-dshm\") pod \"autoscale-hpa-lws-kserve-mn-0-1\" (UID: \"44f19bea-8b8e-4dce-8ac9-c801d3d1cd28\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 28 11:05:47.262274 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:05:47.262058    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44f19bea-8b8e-4dce-8ac9-c801d3d1cd28-model-cache\") pod \"autoscale-hpa-lws-kserve-mn-0-1\" (UID: \"44f19bea-8b8e-4dce-8ac9-c801d3d1cd28\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 28 11:05:47.262274 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:05:47.262097    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqlbx\" (UniqueName: \"kubernetes.io/projected/44f19bea-8b8e-4dce-8ac9-c801d3d1cd28-kube-api-access-pqlbx\") pod \"autoscale-hpa-lws-kserve-mn-0-1\" (UID: \"44f19bea-8b8e-4dce-8ac9-c801d3d1cd28\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 28 11:05:47.262274 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:05:47.262143    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44f19bea-8b8e-4dce-8ac9-c801d3d1cd28-home\") pod \"autoscale-hpa-lws-kserve-mn-0-1\" (UID: \"44f19bea-8b8e-4dce-8ac9-c801d3d1cd28\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 28 11:05:47.262274 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:05:47.262181    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44f19bea-8b8e-4dce-8ac9-c801d3d1cd28-tmp-dir\") pod \"autoscale-hpa-lws-kserve-mn-0-1\" (UID: \"44f19bea-8b8e-4dce-8ac9-c801d3d1cd28\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 28 11:05:47.262506 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:05:47.262289    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44f19bea-8b8e-4dce-8ac9-c801d3d1cd28-tls-certs\") pod \"autoscale-hpa-lws-kserve-mn-0-1\" (UID: \"44f19bea-8b8e-4dce-8ac9-c801d3d1cd28\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 28 11:05:47.263141 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:05:47.263116    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44f19bea-8b8e-4dce-8ac9-c801d3d1cd28-tmp-dir\") pod \"autoscale-hpa-lws-kserve-mn-0-1\" (UID: \"44f19bea-8b8e-4dce-8ac9-c801d3d1cd28\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 28 11:05:47.263346 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:05:47.263325    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44f19bea-8b8e-4dce-8ac9-c801d3d1cd28-home\") pod \"autoscale-hpa-lws-kserve-mn-0-1\" (UID: \"44f19bea-8b8e-4dce-8ac9-c801d3d1cd28\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 28 11:05:47.263502 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:05:47.263483    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44f19bea-8b8e-4dce-8ac9-c801d3d1cd28-model-cache\") pod \"autoscale-hpa-lws-kserve-mn-0-1\" (UID: \"44f19bea-8b8e-4dce-8ac9-c801d3d1cd28\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 28 11:05:47.264849 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:05:47.264823    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44f19bea-8b8e-4dce-8ac9-c801d3d1cd28-dshm\") pod \"autoscale-hpa-lws-kserve-mn-0-1\" (UID: \"44f19bea-8b8e-4dce-8ac9-c801d3d1cd28\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 28 11:05:47.264962 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:05:47.264942    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44f19bea-8b8e-4dce-8ac9-c801d3d1cd28-tls-certs\") pod \"autoscale-hpa-lws-kserve-mn-0-1\" (UID: \"44f19bea-8b8e-4dce-8ac9-c801d3d1cd28\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 28 11:05:47.271199 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:05:47.271166    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqlbx\" (UniqueName: \"kubernetes.io/projected/44f19bea-8b8e-4dce-8ac9-c801d3d1cd28-kube-api-access-pqlbx\") pod \"autoscale-hpa-lws-kserve-mn-0-1\" (UID: \"44f19bea-8b8e-4dce-8ac9-c801d3d1cd28\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 28 11:05:47.365177 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:05:47.365147    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 28 11:05:47.506074 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:05:47.506050    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"]
Jul 28 11:05:47.509478 ip-10-0-141-50 kubenswrapper[2579]: W0728 11:05:47.508177    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44f19bea_8b8e_4dce_8ac9_c801d3d1cd28.slice/crio-aba09d1cff0d6337f1002ed5b067d8f9a845265681691b53d6e57f2063711b63 WatchSource:0}: Error finding container aba09d1cff0d6337f1002ed5b067d8f9a845265681691b53d6e57f2063711b63: Status 404 returned error can't find the container with id aba09d1cff0d6337f1002ed5b067d8f9a845265681691b53d6e57f2063711b63
Jul 28 11:05:47.949812 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:05:47.949778    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1" event={"ID":"44f19bea-8b8e-4dce-8ac9-c801d3d1cd28","Type":"ContainerStarted","Data":"3d4880bf8c804d8eee536c6c4e412ca314aa328acaa7e8449b0766c91cc804a4"}
Jul 28 11:05:47.950230 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:05:47.949833    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1" event={"ID":"44f19bea-8b8e-4dce-8ac9-c801d3d1cd28","Type":"ContainerStarted","Data":"aba09d1cff0d6337f1002ed5b067d8f9a845265681691b53d6e57f2063711b63"}
Jul 28 11:05:47.976590 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:05:47.976507    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1" podStartSLOduration=0.976485191 podStartE2EDuration="976.485191ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 11:05:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 11:05:47.973021692 +0000 UTC m=+3482.236418023" watchObservedRunningTime="2026-07-28 11:05:47.976485191 +0000 UTC m=+3482.239881521"
Jul 28 11:05:58.589984 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:05:58.589950    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:06:15.278078 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:06:15.277974    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:06:18.538940 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:06:18.538910    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:06:46.409281 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:06:46.409245    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:07:39.892013 ip-10-0-141-50 kubenswrapper[2579]: E0728 11:07:39.891924    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bgw9ln" podUID="9f2f8d3c-6c27-4e8b-92c4-d22da1bb1cbe"
Jul 28 11:07:40.500509 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:07:40.500470    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bgw9ln"
Jul 28 11:09:02.707800 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:09:02.707767    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:09:43.501919 ip-10-0-141-50 kubenswrapper[2579]: E0728 11:09:43.501875    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bgw9ln" podUID="9f2f8d3c-6c27-4e8b-92c4-d22da1bb1cbe"
Jul 28 11:09:44.085901 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:09:44.085869    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bgw9ln"
Jul 28 11:10:08.528831 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:10:08.528790    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:10:08.628917 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:10:08.628888    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:10:10.371222 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:10:10.371175    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:10:10.993129 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:10:10.993093    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:10:11.094143 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:10:11.094111    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:10:11.743524 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:10:11.743485    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-vww8z"]
Jul 28 11:10:11.746708 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:10:11.746687    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-vww8z"
Jul 28 11:10:11.748776 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:10:11.748752    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7c3633d1\"/\"openshift-service-ca.crt\""
Jul 28 11:10:11.748891 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:10:11.748813    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7c3633d1\"/\"kube-root-ca.crt\""
Jul 28 11:10:11.749306 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:10:11.749290    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7c3633d1\"/\"default-dockercfg-lfhh2\""
Jul 28 11:10:11.766912 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:10:11.766882    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-vww8z"]
Jul 28 11:10:11.794345 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:10:11.794298    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqtsb\" (UniqueName: \"kubernetes.io/projected/1e388107-0346-474e-a105-c6500edd99e1-kube-api-access-dqtsb\") pod \"e2e-pvc-model-download-vww8z\" (UID: \"1e388107-0346-474e-a105-c6500edd99e1\") " pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-vww8z"
Jul 28 11:10:11.820578 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:10:11.820528    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:10:11.895120 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:10:11.895087    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqtsb\" (UniqueName: \"kubernetes.io/projected/1e388107-0346-474e-a105-c6500edd99e1-kube-api-access-dqtsb\") pod \"e2e-pvc-model-download-vww8z\" (UID: \"1e388107-0346-474e-a105-c6500edd99e1\") " pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-vww8z"
Jul 28 11:10:11.903590 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:10:11.903560    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqtsb\" (UniqueName: \"kubernetes.io/projected/1e388107-0346-474e-a105-c6500edd99e1-kube-api-access-dqtsb\") pod \"e2e-pvc-model-download-vww8z\" (UID: \"1e388107-0346-474e-a105-c6500edd99e1\") " pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-vww8z"
Jul 28 11:10:16.131172 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:10:16.131122    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-22e6f62e-463f-4729-96e2-d0f264811ec3\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-021b25aa65f0f0357\") pod \"e2e-pvc-model-download-vww8z\" (UID: \"1e388107-0346-474e-a105-c6500edd99e1\") " pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-vww8z"
Jul 28 11:10:16.135936 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:10:16.135915    2579 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-22e6f62e-463f-4729-96e2-d0f264811ec3\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-021b25aa65f0f0357\") pod \"e2e-pvc-model-download-vww8z\" (UID: \"1e388107-0346-474e-a105-c6500edd99e1\") device path: \"\"" pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-vww8z"
Jul 28 11:10:16.231714 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:10:16.231669    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-22e6f62e-463f-4729-96e2-d0f264811ec3\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-021b25aa65f0f0357\") pod \"e2e-pvc-model-download-vww8z\" (UID: \"1e388107-0346-474e-a105-c6500edd99e1\") " pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-vww8z"
Jul 28 11:10:16.231945 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:10:16.231808    2579 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-22e6f62e-463f-4729-96e2-d0f264811ec3\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-021b25aa65f0f0357\") pod \"e2e-pvc-model-download-vww8z\" (UID: \"1e388107-0346-474e-a105-c6500edd99e1\") DevicePath \"\"" pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-vww8z"
Jul 28 11:10:16.234203 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:10:16.234179    2579 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-22e6f62e-463f-4729-96e2-d0f264811ec3\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-021b25aa65f0f0357\") pod \"e2e-pvc-model-download-vww8z\" (UID: \"1e388107-0346-474e-a105-c6500edd99e1\") DevicePath \"csi-9e32376c9b88a190d7269997df22609eabb17b8df4fb89632b4fbfadc617357e\"" pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-vww8z"
Jul 28 11:10:16.505329 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:10:16.505298    2579 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-22e6f62e-463f-4729-96e2-d0f264811ec3\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-021b25aa65f0f0357\") pod \"e2e-pvc-model-download-vww8z\" (UID: \"1e388107-0346-474e-a105-c6500edd99e1\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/946f308e63c2899cb5f4f1333e39d10efbf22eb79d52dcb6acf9510bc19079c3/globalmount\"" pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-vww8z"
Jul 28 11:10:16.518735 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:10:16.518708    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-22e6f62e-463f-4729-96e2-d0f264811ec3\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-021b25aa65f0f0357\") pod \"e2e-pvc-model-download-vww8z\" (UID: \"1e388107-0346-474e-a105-c6500edd99e1\") " pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-vww8z"
Jul 28 11:10:16.556348 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:10:16.556308    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-vww8z"
Jul 28 11:10:16.689262 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:10:16.689234    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-vww8z"]
Jul 28 11:10:16.691380 ip-10-0-141-50 kubenswrapper[2579]: W0728 11:10:16.691351    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1e388107_0346_474e_a105_c6500edd99e1.slice/crio-079f1eb11df304596a37e3d7fb28bbf9ce4b64b2a4992b6e853906fa7600bf98 WatchSource:0}: Error finding container 079f1eb11df304596a37e3d7fb28bbf9ce4b64b2a4992b6e853906fa7600bf98: Status 404 returned error can't find the container with id 079f1eb11df304596a37e3d7fb28bbf9ce4b64b2a4992b6e853906fa7600bf98
Jul 28 11:10:17.243905 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:10:17.243860    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-vww8z" event={"ID":"1e388107-0346-474e-a105-c6500edd99e1","Type":"ContainerStarted","Data":"e418ed033c7c3fc7ea8463e37e359c6b13f9c9dd9237f9d2bf5472a5ac4f127b"}
Jul 28 11:10:17.244306 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:10:17.243913    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-vww8z" event={"ID":"1e388107-0346-474e-a105-c6500edd99e1","Type":"ContainerStarted","Data":"079f1eb11df304596a37e3d7fb28bbf9ce4b64b2a4992b6e853906fa7600bf98"}
Jul 28 11:10:17.263599 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:10:17.263531    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-vww8z" podStartSLOduration=9.263513561 podStartE2EDuration="9.263513561s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 11:10:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 11:10:17.261716589 +0000 UTC m=+3751.525112930" watchObservedRunningTime="2026-07-28 11:10:17.263513561 +0000 UTC m=+3751.526909896"
Jul 28 11:10:24.280739 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:10:24.280703    2579 generic.go:356] "Generic (PLEG): container finished" podID="1e388107-0346-474e-a105-c6500edd99e1" containerID="e418ed033c7c3fc7ea8463e37e359c6b13f9c9dd9237f9d2bf5472a5ac4f127b" exitCode=0
Jul 28 11:10:24.281205 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:10:24.280775    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-vww8z" event={"ID":"1e388107-0346-474e-a105-c6500edd99e1","Type":"ContainerDied","Data":"e418ed033c7c3fc7ea8463e37e359c6b13f9c9dd9237f9d2bf5472a5ac4f127b"}
Jul 28 11:10:25.438434 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:10:25.438410    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-vww8z"
Jul 28 11:10:25.517366 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:10:25.517338    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dqtsb\" (UniqueName: \"kubernetes.io/projected/1e388107-0346-474e-a105-c6500edd99e1-kube-api-access-dqtsb\") pod \"1e388107-0346-474e-a105-c6500edd99e1\" (UID: \"1e388107-0346-474e-a105-c6500edd99e1\") "
Jul 28 11:10:25.517530 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:10:25.517478    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-storage\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-021b25aa65f0f0357\") pod \"1e388107-0346-474e-a105-c6500edd99e1\" (UID: \"1e388107-0346-474e-a105-c6500edd99e1\") "
Jul 28 11:10:25.519517 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:10:25.519493    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e388107-0346-474e-a105-c6500edd99e1-kube-api-access-dqtsb" (OuterVolumeSpecName: "kube-api-access-dqtsb") pod "1e388107-0346-474e-a105-c6500edd99e1" (UID: "1e388107-0346-474e-a105-c6500edd99e1"). InnerVolumeSpecName "kube-api-access-dqtsb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 11:10:25.521805 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:10:25.521782    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-021b25aa65f0f0357" (OuterVolumeSpecName: "model-storage") pod "1e388107-0346-474e-a105-c6500edd99e1" (UID: "1e388107-0346-474e-a105-c6500edd99e1"). InnerVolumeSpecName "pvc-22e6f62e-463f-4729-96e2-d0f264811ec3". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Jul 28 11:10:25.618496 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:10:25.618409    2579 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-22e6f62e-463f-4729-96e2-d0f264811ec3\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-021b25aa65f0f0357\") on node \"ip-10-0-141-50.ec2.internal\" "
Jul 28 11:10:25.618496 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:10:25.618438    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqtsb\" (UniqueName: \"kubernetes.io/projected/1e388107-0346-474e-a105-c6500edd99e1-kube-api-access-dqtsb\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 11:10:25.736779 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:10:25.736759    2579 operation_generator.go:896] UnmountDevice succeeded for volume "pvc-22e6f62e-463f-4729-96e2-d0f264811ec3" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-021b25aa65f0f0357") on node "ip-10-0-141-50.ec2.internal"
Jul 28 11:10:25.820169 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:10:25.820123    2579 reconciler_common.go:299] "Volume detached for volume \"pvc-22e6f62e-463f-4729-96e2-d0f264811ec3\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-021b25aa65f0f0357\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"csi-9e32376c9b88a190d7269997df22609eabb17b8df4fb89632b4fbfadc617357e\""
Jul 28 11:10:26.290708 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:10:26.290680    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-vww8z"
Jul 28 11:10:26.290708 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:10:26.290696    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-vww8z" event={"ID":"1e388107-0346-474e-a105-c6500edd99e1","Type":"ContainerDied","Data":"079f1eb11df304596a37e3d7fb28bbf9ce4b64b2a4992b6e853906fa7600bf98"}
Jul 28 11:10:26.290918 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:10:26.290732    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="079f1eb11df304596a37e3d7fb28bbf9ce4b64b2a4992b6e853906fa7600bf98"
Jul 28 11:10:27.777412 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:10:27.777370    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:10:27.877796 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:10:27.877765    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:10:48.236530 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:10:48.236434    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:10:48.336220 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:10:48.336187    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:10:48.414230 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:10:48.414188    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:10:48.574640 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:10:48.574604    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schekxjwb"]
Jul 28 11:10:48.575101 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:10:48.575083    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e388107-0346-474e-a105-c6500edd99e1" containerName="storage-initializer"
Jul 28 11:10:48.575184 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:10:48.575103    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e388107-0346-474e-a105-c6500edd99e1" containerName="storage-initializer"
Jul 28 11:10:48.575244 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:10:48.575187    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e388107-0346-474e-a105-c6500edd99e1" containerName="storage-initializer"
Jul 28 11:10:48.578459 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:10:48.578433    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schekxjwb"
Jul 28 11:10:48.580792 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:10:48.580769    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7c3633d1\"/\"kube-root-ca.crt\""
Jul 28 11:10:48.581175 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:10:48.581155    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7c3633d1\"/\"llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs\""
Jul 28 11:10:48.581764 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:10:48.581739    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7c3633d1\"/\"llmisvc-model-pvc-router-manage-2577e794-epp-sa-dockercfg-ktw7b\""
Jul 28 11:10:48.581877 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:10:48.581859    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7c3633d1\"/\"default-dockercfg-lfhh2\""
Jul 28 11:10:48.582041 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:10:48.582025    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7c3633d1\"/\"openshift-service-ca.crt\""
Jul 28 11:10:48.594371 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:10:48.594333    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schekxjwb"]
Jul 28 11:10:48.605739 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:10:48.605707    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:10:48.723205 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:10:48.723168    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be54b512-094b-4f12-94ae-858699835a9e-tls-certs\") pod \"llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schekxjwb\" (UID: \"be54b512-094b-4f12-94ae-858699835a9e\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schekxjwb"
Jul 28 11:10:48.723415 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:10:48.723300    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpgbx\" (UniqueName: \"kubernetes.io/projected/be54b512-094b-4f12-94ae-858699835a9e-kube-api-access-xpgbx\") pod \"llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schekxjwb\" (UID: \"be54b512-094b-4f12-94ae-858699835a9e\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schekxjwb"
Jul 28 11:10:48.824223 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:10:48.824182    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpgbx\" (UniqueName: \"kubernetes.io/projected/be54b512-094b-4f12-94ae-858699835a9e-kube-api-access-xpgbx\") pod \"llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schekxjwb\" (UID: \"be54b512-094b-4f12-94ae-858699835a9e\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schekxjwb"
Jul 28 11:10:48.824440 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:10:48.824236    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be54b512-094b-4f12-94ae-858699835a9e-tls-certs\") pod \"llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schekxjwb\" (UID: \"be54b512-094b-4f12-94ae-858699835a9e\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schekxjwb"
Jul 28 11:10:48.826884 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:10:48.826811    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be54b512-094b-4f12-94ae-858699835a9e-tls-certs\") pod \"llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schekxjwb\" (UID: \"be54b512-094b-4f12-94ae-858699835a9e\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schekxjwb"
Jul 28 11:10:48.835713 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:10:48.835685    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpgbx\" (UniqueName: \"kubernetes.io/projected/be54b512-094b-4f12-94ae-858699835a9e-kube-api-access-xpgbx\") pod \"llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schekxjwb\" (UID: \"be54b512-094b-4f12-94ae-858699835a9e\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schekxjwb"
Jul 28 11:10:48.895691 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:10:48.895652    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schekxjwb"
Jul 28 11:10:49.033000 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:10:49.032969    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schekxjwb"]
Jul 28 11:10:49.034481 ip-10-0-141-50 kubenswrapper[2579]: W0728 11:10:49.034450    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe54b512_094b_4f12_94ae_858699835a9e.slice/crio-535445e3a45559eab8e4eef125022a57894ac965718c58909447b9554b114e11 WatchSource:0}: Error finding container 535445e3a45559eab8e4eef125022a57894ac965718c58909447b9554b114e11: Status 404 returned error can't find the container with id 535445e3a45559eab8e4eef125022a57894ac965718c58909447b9554b114e11
Jul 28 11:10:49.398833 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:10:49.398735    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schekxjwb" event={"ID":"be54b512-094b-4f12-94ae-858699835a9e","Type":"ContainerStarted","Data":"6aa883e93344f3d3ccc0ab7363ed32aad66175d0f8f5515da82e434c1db123fc"}
Jul 28 11:10:49.398833 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:10:49.398790    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schekxjwb" event={"ID":"be54b512-094b-4f12-94ae-858699835a9e","Type":"ContainerStarted","Data":"535445e3a45559eab8e4eef125022a57894ac965718c58909447b9554b114e11"}
Jul 28 11:10:49.399242 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:10:49.398866    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schekxjwb"
Jul 28 11:10:49.415991 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:10:49.415938    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schekxjwb" podStartSLOduration=1.4159216749999999 podStartE2EDuration="1.415921675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 11:10:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 11:10:49.414447975 +0000 UTC m=+3783.677844336" watchObservedRunningTime="2026-07-28 11:10:49.415921675 +0000 UTC m=+3783.679318021"
Jul 28 11:11:00.580744 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:11:00.580708    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:11:20.405428 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:11:20.405396    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schekxjwb"
Jul 28 11:11:47.086881 ip-10-0-141-50 kubenswrapper[2579]: E0728 11:11:47.086821    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bgw9ln" podUID="9f2f8d3c-6c27-4e8b-92c4-d22da1bb1cbe"
Jul 28 11:11:47.678102 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:11:47.678064    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bgw9ln"
Jul 28 11:13:38.664723 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:13:38.664631    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:13:50.678661 ip-10-0-141-50 kubenswrapper[2579]: E0728 11:13:50.678613    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bgw9ln" podUID="9f2f8d3c-6c27-4e8b-92c4-d22da1bb1cbe"
Jul 28 11:13:51.269816 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:13:51.269778    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bgw9ln"
Jul 28 11:13:59.496583 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:13:59.496520    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:15:54.270393 ip-10-0-141-50 kubenswrapper[2579]: E0728 11:15:54.270344    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bgw9ln" podUID="9f2f8d3c-6c27-4e8b-92c4-d22da1bb1cbe"
Jul 28 11:15:54.859068 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:15:54.859036    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bgw9ln"
Jul 28 11:16:13.652834 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:16:13.652793    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:16:16.268796 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:16:16.268754    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:17:57.859811 ip-10-0-141-50 kubenswrapper[2579]: E0728 11:17:57.859762    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bgw9ln" podUID="9f2f8d3c-6c27-4e8b-92c4-d22da1bb1cbe"
Jul 28 11:17:58.454003 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:17:58.453967    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bgw9ln"
Jul 28 11:18:45.146782 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:18:45.146747    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:18:45.206185 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:18:45.206153    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:20:01.455400 ip-10-0-141-50 kubenswrapper[2579]: E0728 11:20:01.455298    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bgw9ln" podUID="9f2f8d3c-6c27-4e8b-92c4-d22da1bb1cbe"
Jul 28 11:20:02.052096 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:20:02.052062    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bgw9ln"
Jul 28 11:20:42.080910 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:20:42.080857    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:20:42.170576 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:20:42.170520    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:20:43.395967 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:20:43.395932    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"]
Jul 28 11:20:43.404704 ip-10-0-141-50 kubenswrapper[2579]: E0728 11:20:43.404669    2579 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 28 11:20:43.404704 ip-10-0-141-50 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 11:20:43.404704 ip-10-0-141-50 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Jul 28 11:20:43.404704 ip-10-0-141-50 kubenswrapper[2579]:  > containerID="3d4880bf8c804d8eee536c6c4e412ca314aa328acaa7e8449b0766c91cc804a4" cmd=["/bin/sleep","15"]
Jul 28 11:20:43.405046 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:20:43.404720    2579 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1" message="" err=<
Jul 28 11:20:43.405046 ip-10-0-141-50 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 11:20:43.405046 ip-10-0-141-50 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Jul 28 11:20:43.405046 ip-10-0-141-50 kubenswrapper[2579]:  >
Jul 28 11:20:43.405046 ip-10-0-141-50 kubenswrapper[2579]: E0728 11:20:43.404732    2579 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 28 11:20:43.405046 ip-10-0-141-50 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 11:20:43.405046 ip-10-0-141-50 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Jul 28 11:20:43.405046 ip-10-0-141-50 kubenswrapper[2579]:  > pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1" podUID="44f19bea-8b8e-4dce-8ac9-c801d3d1cd28" containerName="main" containerID="cri-o://3d4880bf8c804d8eee536c6c4e412ca314aa328acaa7e8449b0766c91cc804a4"
Jul 28 11:20:43.405046 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:20:43.404765    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1" podUID="44f19bea-8b8e-4dce-8ac9-c801d3d1cd28" containerName="main" containerID="cri-o://3d4880bf8c804d8eee536c6c4e412ca314aa328acaa7e8449b0766c91cc804a4" gracePeriod=60
Jul 28 11:20:43.648610 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:20:43.648523    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 28 11:20:43.754481 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:20:43.754445    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44f19bea-8b8e-4dce-8ac9-c801d3d1cd28-home\") pod \"44f19bea-8b8e-4dce-8ac9-c801d3d1cd28\" (UID: \"44f19bea-8b8e-4dce-8ac9-c801d3d1cd28\") "
Jul 28 11:20:43.754771 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:20:43.754494    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44f19bea-8b8e-4dce-8ac9-c801d3d1cd28-dshm\") pod \"44f19bea-8b8e-4dce-8ac9-c801d3d1cd28\" (UID: \"44f19bea-8b8e-4dce-8ac9-c801d3d1cd28\") "
Jul 28 11:20:43.754771 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:20:43.754530    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44f19bea-8b8e-4dce-8ac9-c801d3d1cd28-tls-certs\") pod \"44f19bea-8b8e-4dce-8ac9-c801d3d1cd28\" (UID: \"44f19bea-8b8e-4dce-8ac9-c801d3d1cd28\") "
Jul 28 11:20:43.754771 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:20:43.754614    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44f19bea-8b8e-4dce-8ac9-c801d3d1cd28-model-cache\") pod \"44f19bea-8b8e-4dce-8ac9-c801d3d1cd28\" (UID: \"44f19bea-8b8e-4dce-8ac9-c801d3d1cd28\") "
Jul 28 11:20:43.754771 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:20:43.754642    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pqlbx\" (UniqueName: \"kubernetes.io/projected/44f19bea-8b8e-4dce-8ac9-c801d3d1cd28-kube-api-access-pqlbx\") pod \"44f19bea-8b8e-4dce-8ac9-c801d3d1cd28\" (UID: \"44f19bea-8b8e-4dce-8ac9-c801d3d1cd28\") "
Jul 28 11:20:43.754771 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:20:43.754685    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44f19bea-8b8e-4dce-8ac9-c801d3d1cd28-tmp-dir\") pod \"44f19bea-8b8e-4dce-8ac9-c801d3d1cd28\" (UID: \"44f19bea-8b8e-4dce-8ac9-c801d3d1cd28\") "
Jul 28 11:20:43.754771 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:20:43.754748    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44f19bea-8b8e-4dce-8ac9-c801d3d1cd28-home" (OuterVolumeSpecName: "home") pod "44f19bea-8b8e-4dce-8ac9-c801d3d1cd28" (UID: "44f19bea-8b8e-4dce-8ac9-c801d3d1cd28"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 11:20:43.755112 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:20:43.754916    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44f19bea-8b8e-4dce-8ac9-c801d3d1cd28-model-cache" (OuterVolumeSpecName: "model-cache") pod "44f19bea-8b8e-4dce-8ac9-c801d3d1cd28" (UID: "44f19bea-8b8e-4dce-8ac9-c801d3d1cd28"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 11:20:43.755112 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:20:43.754957    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44f19bea-8b8e-4dce-8ac9-c801d3d1cd28-home\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 11:20:43.755220 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:20:43.755175    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44f19bea-8b8e-4dce-8ac9-c801d3d1cd28-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "44f19bea-8b8e-4dce-8ac9-c801d3d1cd28" (UID: "44f19bea-8b8e-4dce-8ac9-c801d3d1cd28"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 11:20:43.757101 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:20:43.757070    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44f19bea-8b8e-4dce-8ac9-c801d3d1cd28-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "44f19bea-8b8e-4dce-8ac9-c801d3d1cd28" (UID: "44f19bea-8b8e-4dce-8ac9-c801d3d1cd28"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 11:20:43.757225 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:20:43.757206    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44f19bea-8b8e-4dce-8ac9-c801d3d1cd28-dshm" (OuterVolumeSpecName: "dshm") pod "44f19bea-8b8e-4dce-8ac9-c801d3d1cd28" (UID: "44f19bea-8b8e-4dce-8ac9-c801d3d1cd28"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 11:20:43.757298 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:20:43.757234    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44f19bea-8b8e-4dce-8ac9-c801d3d1cd28-kube-api-access-pqlbx" (OuterVolumeSpecName: "kube-api-access-pqlbx") pod "44f19bea-8b8e-4dce-8ac9-c801d3d1cd28" (UID: "44f19bea-8b8e-4dce-8ac9-c801d3d1cd28"). InnerVolumeSpecName "kube-api-access-pqlbx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 11:20:43.855869 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:20:43.855832    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44f19bea-8b8e-4dce-8ac9-c801d3d1cd28-tmp-dir\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 11:20:43.855869 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:20:43.855864    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44f19bea-8b8e-4dce-8ac9-c801d3d1cd28-dshm\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 11:20:43.855869 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:20:43.855877    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44f19bea-8b8e-4dce-8ac9-c801d3d1cd28-tls-certs\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 11:20:43.856114 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:20:43.855889    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44f19bea-8b8e-4dce-8ac9-c801d3d1cd28-model-cache\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 11:20:43.856114 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:20:43.855902    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqlbx\" (UniqueName: \"kubernetes.io/projected/44f19bea-8b8e-4dce-8ac9-c801d3d1cd28-kube-api-access-pqlbx\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 11:20:44.259387 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:20:44.259340    2579 generic.go:356] "Generic (PLEG): container finished" podID="44f19bea-8b8e-4dce-8ac9-c801d3d1cd28" containerID="3d4880bf8c804d8eee536c6c4e412ca314aa328acaa7e8449b0766c91cc804a4" exitCode=0
Jul 28 11:20:44.259612 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:20:44.259407    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 28 11:20:44.259612 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:20:44.259427    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1" event={"ID":"44f19bea-8b8e-4dce-8ac9-c801d3d1cd28","Type":"ContainerDied","Data":"3d4880bf8c804d8eee536c6c4e412ca314aa328acaa7e8449b0766c91cc804a4"}
Jul 28 11:20:44.259612 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:20:44.259517    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1" event={"ID":"44f19bea-8b8e-4dce-8ac9-c801d3d1cd28","Type":"ContainerDied","Data":"aba09d1cff0d6337f1002ed5b067d8f9a845265681691b53d6e57f2063711b63"}
Jul 28 11:20:44.259612 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:20:44.259577    2579 scope.go:117] "RemoveContainer" containerID="3d4880bf8c804d8eee536c6c4e412ca314aa328acaa7e8449b0766c91cc804a4"
Jul 28 11:20:44.269309 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:20:44.269289    2579 scope.go:117] "RemoveContainer" containerID="3d4880bf8c804d8eee536c6c4e412ca314aa328acaa7e8449b0766c91cc804a4"
Jul 28 11:20:44.269627 ip-10-0-141-50 kubenswrapper[2579]: E0728 11:20:44.269603    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d4880bf8c804d8eee536c6c4e412ca314aa328acaa7e8449b0766c91cc804a4\": container with ID starting with 3d4880bf8c804d8eee536c6c4e412ca314aa328acaa7e8449b0766c91cc804a4 not found: ID does not exist" containerID="3d4880bf8c804d8eee536c6c4e412ca314aa328acaa7e8449b0766c91cc804a4"
Jul 28 11:20:44.269697 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:20:44.269640    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d4880bf8c804d8eee536c6c4e412ca314aa328acaa7e8449b0766c91cc804a4"} err="failed to get container status \"3d4880bf8c804d8eee536c6c4e412ca314aa328acaa7e8449b0766c91cc804a4\": rpc error: code = NotFound desc = could not find container \"3d4880bf8c804d8eee536c6c4e412ca314aa328acaa7e8449b0766c91cc804a4\": container with ID starting with 3d4880bf8c804d8eee536c6c4e412ca314aa328acaa7e8449b0766c91cc804a4 not found: ID does not exist"
Jul 28 11:20:44.282601 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:20:44.282566    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"]
Jul 28 11:20:44.286011 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:20:44.285984    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"]
Jul 28 11:20:46.275898 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:20:46.275869    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="44f19bea-8b8e-4dce-8ac9-c801d3d1cd28" path="/var/lib/kubelet/pods/44f19bea-8b8e-4dce-8ac9-c801d3d1cd28/volumes"
Jul 28 11:21:01.968449 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:21:01.968389    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:21:02.072761 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:21:02.072726    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:21:43.429475 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:21:43.429445    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:21:43.567420 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:21:43.567386    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"]
Jul 28 11:21:43.567738 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:21:43.567724    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44f19bea-8b8e-4dce-8ac9-c801d3d1cd28" containerName="main"
Jul 28 11:21:43.567738 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:21:43.567738    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="44f19bea-8b8e-4dce-8ac9-c801d3d1cd28" containerName="main"
Jul 28 11:21:43.567835 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:21:43.567797    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="44f19bea-8b8e-4dce-8ac9-c801d3d1cd28" containerName="main"
Jul 28 11:21:43.570745 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:21:43.570729    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 28 11:21:43.573489 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:21:43.573463    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-autoscaling-keda-lws-e541a132\"/\"kube-root-ca.crt\""
Jul 28 11:21:43.573489 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:21:43.573486    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-autoscaling-keda-lws-e541a132\"/\"openshift-service-ca.crt\""
Jul 28 11:21:43.573702 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:21:43.573488    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-keda-lws-e541a132\"/\"default-dockercfg-xzt9g\""
Jul 28 11:21:43.573702 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:21:43.573510    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-keda-lws-e541a132\"/\"autoscale-keda-lws-kserve-mn-dockercfg-qgccn\""
Jul 28 11:21:43.573702 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:21:43.573529    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-keda-lws-e541a132\"/\"autoscale-keda-lws-kserve-self-signed-certs\""
Jul 28 11:21:43.582575 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:21:43.582529    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"]
Jul 28 11:21:43.676318 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:21:43.676278    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18ad311c-36fc-4d60-a0c3-a9282849c227-tmp-dir\") pod \"autoscale-keda-lws-kserve-mn-0-1\" (UID: \"18ad311c-36fc-4d60-a0c3-a9282849c227\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 28 11:21:43.676505 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:21:43.676328    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2bsd\" (UniqueName: \"kubernetes.io/projected/18ad311c-36fc-4d60-a0c3-a9282849c227-kube-api-access-j2bsd\") pod \"autoscale-keda-lws-kserve-mn-0-1\" (UID: \"18ad311c-36fc-4d60-a0c3-a9282849c227\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 28 11:21:43.676505 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:21:43.676351    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18ad311c-36fc-4d60-a0c3-a9282849c227-tls-certs\") pod \"autoscale-keda-lws-kserve-mn-0-1\" (UID: \"18ad311c-36fc-4d60-a0c3-a9282849c227\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 28 11:21:43.676505 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:21:43.676441    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18ad311c-36fc-4d60-a0c3-a9282849c227-home\") pod \"autoscale-keda-lws-kserve-mn-0-1\" (UID: \"18ad311c-36fc-4d60-a0c3-a9282849c227\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 28 11:21:43.676505 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:21:43.676472    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18ad311c-36fc-4d60-a0c3-a9282849c227-model-cache\") pod \"autoscale-keda-lws-kserve-mn-0-1\" (UID: \"18ad311c-36fc-4d60-a0c3-a9282849c227\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 28 11:21:43.676505 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:21:43.676496    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18ad311c-36fc-4d60-a0c3-a9282849c227-dshm\") pod \"autoscale-keda-lws-kserve-mn-0-1\" (UID: \"18ad311c-36fc-4d60-a0c3-a9282849c227\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 28 11:21:43.751771 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:21:43.751680    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-t5ptm"]
Jul 28 11:21:43.754967 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:21:43.754942    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-t5ptm"
Jul 28 11:21:43.757255 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:21:43.757228    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-keda-lws-e541a132\"/\"autoscale-keda-lws-epp-sa-dockercfg-sblft\""
Jul 28 11:21:43.762709 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:21:43.762687    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-t5ptm"]
Jul 28 11:21:43.777383 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:21:43.777354    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18ad311c-36fc-4d60-a0c3-a9282849c227-tmp-dir\") pod \"autoscale-keda-lws-kserve-mn-0-1\" (UID: \"18ad311c-36fc-4d60-a0c3-a9282849c227\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 28 11:21:43.777566 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:21:43.777410    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2bsd\" (UniqueName: \"kubernetes.io/projected/18ad311c-36fc-4d60-a0c3-a9282849c227-kube-api-access-j2bsd\") pod \"autoscale-keda-lws-kserve-mn-0-1\" (UID: \"18ad311c-36fc-4d60-a0c3-a9282849c227\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 28 11:21:43.777566 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:21:43.777431    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18ad311c-36fc-4d60-a0c3-a9282849c227-tls-certs\") pod \"autoscale-keda-lws-kserve-mn-0-1\" (UID: \"18ad311c-36fc-4d60-a0c3-a9282849c227\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 28 11:21:43.777566 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:21:43.777469    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18ad311c-36fc-4d60-a0c3-a9282849c227-home\") pod \"autoscale-keda-lws-kserve-mn-0-1\" (UID: \"18ad311c-36fc-4d60-a0c3-a9282849c227\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 28 11:21:43.777566 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:21:43.777492    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18ad311c-36fc-4d60-a0c3-a9282849c227-model-cache\") pod \"autoscale-keda-lws-kserve-mn-0-1\" (UID: \"18ad311c-36fc-4d60-a0c3-a9282849c227\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 28 11:21:43.777566 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:21:43.777524    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18ad311c-36fc-4d60-a0c3-a9282849c227-dshm\") pod \"autoscale-keda-lws-kserve-mn-0-1\" (UID: \"18ad311c-36fc-4d60-a0c3-a9282849c227\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 28 11:21:43.777853 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:21:43.777712    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18ad311c-36fc-4d60-a0c3-a9282849c227-tmp-dir\") pod \"autoscale-keda-lws-kserve-mn-0-1\" (UID: \"18ad311c-36fc-4d60-a0c3-a9282849c227\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 28 11:21:43.777853 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:21:43.777739    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18ad311c-36fc-4d60-a0c3-a9282849c227-model-cache\") pod \"autoscale-keda-lws-kserve-mn-0-1\" (UID: \"18ad311c-36fc-4d60-a0c3-a9282849c227\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 28 11:21:43.777853 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:21:43.777841    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18ad311c-36fc-4d60-a0c3-a9282849c227-home\") pod \"autoscale-keda-lws-kserve-mn-0-1\" (UID: \"18ad311c-36fc-4d60-a0c3-a9282849c227\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 28 11:21:43.779699 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:21:43.779677    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18ad311c-36fc-4d60-a0c3-a9282849c227-dshm\") pod \"autoscale-keda-lws-kserve-mn-0-1\" (UID: \"18ad311c-36fc-4d60-a0c3-a9282849c227\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 28 11:21:43.779804 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:21:43.779772    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18ad311c-36fc-4d60-a0c3-a9282849c227-tls-certs\") pod \"autoscale-keda-lws-kserve-mn-0-1\" (UID: \"18ad311c-36fc-4d60-a0c3-a9282849c227\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 28 11:21:43.785233 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:21:43.785205    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2bsd\" (UniqueName: \"kubernetes.io/projected/18ad311c-36fc-4d60-a0c3-a9282849c227-kube-api-access-j2bsd\") pod \"autoscale-keda-lws-kserve-mn-0-1\" (UID: \"18ad311c-36fc-4d60-a0c3-a9282849c227\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 28 11:21:43.878430 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:21:43.878393    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wt996\" (UniqueName: \"kubernetes.io/projected/8e4537ec-37c7-4e84-8769-a22d4f2451d4-kube-api-access-wt996\") pod \"autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-t5ptm\" (UID: \"8e4537ec-37c7-4e84-8769-a22d4f2451d4\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-t5ptm"
Jul 28 11:21:43.878612 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:21:43.878441    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e4537ec-37c7-4e84-8769-a22d4f2451d4-tls-certs\") pod \"autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-t5ptm\" (UID: \"8e4537ec-37c7-4e84-8769-a22d4f2451d4\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-t5ptm"
Jul 28 11:21:43.882353 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:21:43.882331    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 28 11:21:43.979188 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:21:43.979153    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e4537ec-37c7-4e84-8769-a22d4f2451d4-tls-certs\") pod \"autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-t5ptm\" (UID: \"8e4537ec-37c7-4e84-8769-a22d4f2451d4\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-t5ptm"
Jul 28 11:21:43.979353 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:21:43.979286    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wt996\" (UniqueName: \"kubernetes.io/projected/8e4537ec-37c7-4e84-8769-a22d4f2451d4-kube-api-access-wt996\") pod \"autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-t5ptm\" (UID: \"8e4537ec-37c7-4e84-8769-a22d4f2451d4\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-t5ptm"
Jul 28 11:21:43.981933 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:21:43.981899    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e4537ec-37c7-4e84-8769-a22d4f2451d4-tls-certs\") pod \"autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-t5ptm\" (UID: \"8e4537ec-37c7-4e84-8769-a22d4f2451d4\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-t5ptm"
Jul 28 11:21:43.987052 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:21:43.987020    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wt996\" (UniqueName: \"kubernetes.io/projected/8e4537ec-37c7-4e84-8769-a22d4f2451d4-kube-api-access-wt996\") pod \"autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-t5ptm\" (UID: \"8e4537ec-37c7-4e84-8769-a22d4f2451d4\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-t5ptm"
Jul 28 11:21:44.026926 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:21:44.026895    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"]
Jul 28 11:21:44.028594 ip-10-0-141-50 kubenswrapper[2579]: W0728 11:21:44.028565    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18ad311c_36fc_4d60_a0c3_a9282849c227.slice/crio-512f346ce2f1725bccfbc28ad0799780339ade486616f9554b0c74c20e10e1ba WatchSource:0}: Error finding container 512f346ce2f1725bccfbc28ad0799780339ade486616f9554b0c74c20e10e1ba: Status 404 returned error can't find the container with id 512f346ce2f1725bccfbc28ad0799780339ade486616f9554b0c74c20e10e1ba
Jul 28 11:21:44.066407 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:21:44.066380    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-t5ptm"
Jul 28 11:21:44.205849 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:21:44.205822    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-t5ptm"]
Jul 28 11:21:44.207919 ip-10-0-141-50 kubenswrapper[2579]: W0728 11:21:44.207892    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e4537ec_37c7_4e84_8769_a22d4f2451d4.slice/crio-744b29f821a2754e0bfff6c3f5f86ba0c8b39707c899690d5729a69d7cb42669 WatchSource:0}: Error finding container 744b29f821a2754e0bfff6c3f5f86ba0c8b39707c899690d5729a69d7cb42669: Status 404 returned error can't find the container with id 744b29f821a2754e0bfff6c3f5f86ba0c8b39707c899690d5729a69d7cb42669
Jul 28 11:21:44.563010 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:21:44.562966    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1" event={"ID":"18ad311c-36fc-4d60-a0c3-a9282849c227","Type":"ContainerStarted","Data":"40de22e48ca042538704fdcb2ffde247e283c09913c0796475b8f31db3e2008f"}
Jul 28 11:21:44.563522 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:21:44.563023    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1" event={"ID":"18ad311c-36fc-4d60-a0c3-a9282849c227","Type":"ContainerStarted","Data":"512f346ce2f1725bccfbc28ad0799780339ade486616f9554b0c74c20e10e1ba"}
Jul 28 11:21:44.564803 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:21:44.564778    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-t5ptm" event={"ID":"8e4537ec-37c7-4e84-8769-a22d4f2451d4","Type":"ContainerStarted","Data":"2dd7fd324381edb9375d89f28332895b8c0a6304d833dbae06bff3e177d00841"}
Jul 28 11:21:44.564919 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:21:44.564826    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-t5ptm"
Jul 28 11:21:44.564919 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:21:44.564843    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-t5ptm" event={"ID":"8e4537ec-37c7-4e84-8769-a22d4f2451d4","Type":"ContainerStarted","Data":"744b29f821a2754e0bfff6c3f5f86ba0c8b39707c899690d5729a69d7cb42669"}
Jul 28 11:21:44.581132 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:21:44.581086    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1" podStartSLOduration=1.581071157 podStartE2EDuration="1.581071157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 11:21:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 11:21:44.579404709 +0000 UTC m=+4438.842801069" watchObservedRunningTime="2026-07-28 11:21:44.581071157 +0000 UTC m=+4438.844467486"
Jul 28 11:21:44.595247 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:21:44.595200    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-t5ptm" podStartSLOduration=1.5951834150000002 podStartE2EDuration="1.595183415s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 11:21:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 11:21:44.593780061 +0000 UTC m=+4438.857176389" watchObservedRunningTime="2026-07-28 11:21:44.595183415 +0000 UTC m=+4438.858579744"
Jul 28 11:21:44.773074 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:21:44.773036    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:21:59.662300 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:21:59.662259    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:21:59.764586 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:21:59.764533    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:22:05.053094 ip-10-0-141-50 kubenswrapper[2579]: E0728 11:22:05.053042    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bgw9ln" podUID="9f2f8d3c-6c27-4e8b-92c4-d22da1bb1cbe"
Jul 28 11:22:05.667504 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:22:05.667473    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bgw9ln"
Jul 28 11:22:09.364794 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:22:09.364764    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:22:15.571427 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:22:15.571390    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-t5ptm"
Jul 28 11:23:18.479813 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:23:18.479775    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:24:08.668388 ip-10-0-141-50 kubenswrapper[2579]: E0728 11:24:08.668294    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bgw9ln" podUID="9f2f8d3c-6c27-4e8b-92c4-d22da1bb1cbe"
Jul 28 11:24:09.269309 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:24:09.269279    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bgw9ln"
Jul 28 11:25:14.930939 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:25:14.930886    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:25:15.017723 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:25:15.017686    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:25:26.180533 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:25:26.180499    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:25:26.234559 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:25:26.234511    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:25:31.141016 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:25:31.140973    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-7c3633d1_llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schekxjwb_be54b512-094b-4f12-94ae-858699835a9e/main/0.log"
Jul 28 11:25:32.306396 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:25:32.306367    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:25:32.407579 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:25:32.407521    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:25:34.536332 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:25:34.536294    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:25:34.636949 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:25:34.636912    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:25:34.740583 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:25:34.740531    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:25:40.599406 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:25:40.599337    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:25:40.663734 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:25:40.663702    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:25:41.495484 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:25:41.495404    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:25:41.600464 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:25:41.600423    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:25:41.696134 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:25:41.696100    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:25:45.825430 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:25:45.825399    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:25:45.928076 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:25:45.928043    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:25:46.027865 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:25:46.027831    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:25:47.784835 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:25:47.784805    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:25:47.886170 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:25:47.886139    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:25:51.108250 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:25:51.108211    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:25:51.395971 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:25:51.395889    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:25:51.502981 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:25:51.502951    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:25:52.551014 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:25:52.550978    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:25:52.650394 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:25:52.650361    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:25:56.521109 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:25:56.521071    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:25:56.626561 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:25:56.626508    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:25:56.726734 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:25:56.726697    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:25:57.301191 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:25:57.301163    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:25:57.402892 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:25:57.402864    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:25:57.502658 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:25:57.502621    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:25:57.601219 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:25:57.601129    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:25:57.705204 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:25:57.705172    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:25:57.806045 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:25:57.806012    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:25:57.906076 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:25:57.905993    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:25:58.594653 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:25:58.594613    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:25:58.695360 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:25:58.695328    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:25:58.795447 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:25:58.795416    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:25:59.700994 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:25:59.700961    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:25:59.801261 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:25:59.801233    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:25:59.901822 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:25:59.901789    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:26:01.223041 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:26:01.223006    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:26:01.325516 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:26:01.325485    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:26:03.234893 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:26:03.234860    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:26:03.338011 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:26:03.337977    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:26:03.736447 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:26:03.736410    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:26:03.839091 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:26:03.839054    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:26:04.775371 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:26:04.775337    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:26:04.877749 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:26:04.877707    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:26:05.364797 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:26:05.364768    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:26:05.467148 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:26:05.467111    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:26:06.001513 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:26:06.001468    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:26:06.082016 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:26:06.081984    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:26:07.185844 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:26:07.185808    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:26:07.260134 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:26:07.260103    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:26:12.270679 ip-10-0-141-50 kubenswrapper[2579]: E0728 11:26:12.270630    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bgw9ln" podUID="9f2f8d3c-6c27-4e8b-92c4-d22da1bb1cbe"
Jul 28 11:26:12.868087 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:26:12.868054    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bgw9ln"
Jul 28 11:26:34.636025 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:26:34.635991    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:26:34.905574 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:26:34.905477    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-tjbr8"]
Jul 28 11:26:34.908856 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:26:34.908832    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-tjbr8"
Jul 28 11:26:34.911912 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:26:34.911882    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-epp-sa-dockercfg-h447h\""
Jul 28 11:26:34.912056 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:26:34.911920    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"kube-root-ca.crt\""
Jul 28 11:26:34.912056 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:26:34.911952    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"openshift-service-ca.crt\""
Jul 28 11:26:34.912056 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:26:34.911893    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-kserve-self-signed-certs\""
Jul 28 11:26:34.912056 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:26:34.911882    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"default-dockercfg-65z7g\""
Jul 28 11:26:34.917618 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:26:34.917598    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-tjbr8"]
Jul 28 11:26:34.935116 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:26:34.935084    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:26:34.991265 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:26:34.991223    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60e50fe5-0b57-4c56-be9f-7b46d260f10e-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-6ff9c88bb8-tjbr8\" (UID: \"60e50fe5-0b57-4c56-be9f-7b46d260f10e\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-tjbr8"
Jul 28 11:26:34.991440 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:26:34.991305    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2gjg\" (UniqueName: \"kubernetes.io/projected/60e50fe5-0b57-4c56-be9f-7b46d260f10e-kube-api-access-t2gjg\") pod \"stop-feature-test-kserve-router-scheduler-6ff9c88bb8-tjbr8\" (UID: \"60e50fe5-0b57-4c56-be9f-7b46d260f10e\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-tjbr8"
Jul 28 11:26:35.092135 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:26:35.092104    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2gjg\" (UniqueName: \"kubernetes.io/projected/60e50fe5-0b57-4c56-be9f-7b46d260f10e-kube-api-access-t2gjg\") pod \"stop-feature-test-kserve-router-scheduler-6ff9c88bb8-tjbr8\" (UID: \"60e50fe5-0b57-4c56-be9f-7b46d260f10e\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-tjbr8"
Jul 28 11:26:35.092317 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:26:35.092174    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60e50fe5-0b57-4c56-be9f-7b46d260f10e-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-6ff9c88bb8-tjbr8\" (UID: \"60e50fe5-0b57-4c56-be9f-7b46d260f10e\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-tjbr8"
Jul 28 11:26:35.094636 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:26:35.094614    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60e50fe5-0b57-4c56-be9f-7b46d260f10e-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-6ff9c88bb8-tjbr8\" (UID: \"60e50fe5-0b57-4c56-be9f-7b46d260f10e\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-tjbr8"
Jul 28 11:26:35.100182 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:26:35.100154    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2gjg\" (UniqueName: \"kubernetes.io/projected/60e50fe5-0b57-4c56-be9f-7b46d260f10e-kube-api-access-t2gjg\") pod \"stop-feature-test-kserve-router-scheduler-6ff9c88bb8-tjbr8\" (UID: \"60e50fe5-0b57-4c56-be9f-7b46d260f10e\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-tjbr8"
Jul 28 11:26:35.221160 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:26:35.221064    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-tjbr8"
Jul 28 11:26:35.365873 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:26:35.365848    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-tjbr8"]
Jul 28 11:26:35.368441 ip-10-0-141-50 kubenswrapper[2579]: W0728 11:26:35.368407    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60e50fe5_0b57_4c56_be9f_7b46d260f10e.slice/crio-ce5ae414c3daecb128f0724ba760acf7818bb2a65aa536777041613eb92f42d3 WatchSource:0}: Error finding container ce5ae414c3daecb128f0724ba760acf7818bb2a65aa536777041613eb92f42d3: Status 404 returned error can't find the container with id ce5ae414c3daecb128f0724ba760acf7818bb2a65aa536777041613eb92f42d3
Jul 28 11:26:35.985301 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:26:35.985261    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-tjbr8" event={"ID":"60e50fe5-0b57-4c56-be9f-7b46d260f10e","Type":"ContainerStarted","Data":"158802b7482712be318e18644969a7469759a97f2594386f23254bd70c86b666"}
Jul 28 11:26:35.985797 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:26:35.985375    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-tjbr8"
Jul 28 11:26:35.985797 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:26:35.985403    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-tjbr8" event={"ID":"60e50fe5-0b57-4c56-be9f-7b46d260f10e","Type":"ContainerStarted","Data":"ce5ae414c3daecb128f0724ba760acf7818bb2a65aa536777041613eb92f42d3"}
Jul 28 11:26:36.004297 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:26:36.004216    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-tjbr8" podStartSLOduration=2.004194937 podStartE2EDuration="2.004194937s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 11:26:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 11:26:36.002942175 +0000 UTC m=+4730.266338517" watchObservedRunningTime="2026-07-28 11:26:36.004194937 +0000 UTC m=+4730.267591267"
Jul 28 11:26:59.930756 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:26:59.930716    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:27:06.992605 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:27:06.992569    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-tjbr8"
Jul 28 11:28:15.869021 ip-10-0-141-50 kubenswrapper[2579]: E0728 11:28:15.868963    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bgw9ln" podUID="9f2f8d3c-6c27-4e8b-92c4-d22da1bb1cbe"
Jul 28 11:28:16.516775 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:28:16.516744    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bgw9ln"
Jul 28 11:29:27.694695 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:29:27.694659    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:29:27.753983 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:29:27.753945    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-tjbr8"]
Jul 28 11:29:27.781695 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:29:27.781667    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:29:27.833786 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:29:27.833755    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:29:36.992259 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:29:36.992220    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-tjbr8" podUID="60e50fe5-0b57-4c56-be9f-7b46d260f10e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 11:29:41.289285 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:29:41.289246    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:29:42.773763 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:29:42.773719    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-tjbr8" podUID="60e50fe5-0b57-4c56-be9f-7b46d260f10e" containerName="main" containerID="cri-o://158802b7482712be318e18644969a7469759a97f2594386f23254bd70c86b666" gracePeriod=45
Jul 28 11:29:42.956305 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:29:42.956266    2579 generic.go:356] "Generic (PLEG): container finished" podID="60e50fe5-0b57-4c56-be9f-7b46d260f10e" containerID="158802b7482712be318e18644969a7469759a97f2594386f23254bd70c86b666" exitCode=0
Jul 28 11:29:42.956448 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:29:42.956338    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-tjbr8" event={"ID":"60e50fe5-0b57-4c56-be9f-7b46d260f10e","Type":"ContainerDied","Data":"158802b7482712be318e18644969a7469759a97f2594386f23254bd70c86b666"}
Jul 28 11:29:43.014161 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:29:43.014135    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-tjbr8"
Jul 28 11:29:43.046274 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:29:43.046242    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t2gjg\" (UniqueName: \"kubernetes.io/projected/60e50fe5-0b57-4c56-be9f-7b46d260f10e-kube-api-access-t2gjg\") pod \"60e50fe5-0b57-4c56-be9f-7b46d260f10e\" (UID: \"60e50fe5-0b57-4c56-be9f-7b46d260f10e\") "
Jul 28 11:29:43.046450 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:29:43.046352    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60e50fe5-0b57-4c56-be9f-7b46d260f10e-tls-certs\") pod \"60e50fe5-0b57-4c56-be9f-7b46d260f10e\" (UID: \"60e50fe5-0b57-4c56-be9f-7b46d260f10e\") "
Jul 28 11:29:43.048590 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:29:43.048560    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60e50fe5-0b57-4c56-be9f-7b46d260f10e-kube-api-access-t2gjg" (OuterVolumeSpecName: "kube-api-access-t2gjg") pod "60e50fe5-0b57-4c56-be9f-7b46d260f10e" (UID: "60e50fe5-0b57-4c56-be9f-7b46d260f10e"). InnerVolumeSpecName "kube-api-access-t2gjg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 11:29:43.048730 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:29:43.048673    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60e50fe5-0b57-4c56-be9f-7b46d260f10e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "60e50fe5-0b57-4c56-be9f-7b46d260f10e" (UID: "60e50fe5-0b57-4c56-be9f-7b46d260f10e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 11:29:43.147280 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:29:43.147241    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60e50fe5-0b57-4c56-be9f-7b46d260f10e-tls-certs\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 11:29:43.147280 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:29:43.147272    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2gjg\" (UniqueName: \"kubernetes.io/projected/60e50fe5-0b57-4c56-be9f-7b46d260f10e-kube-api-access-t2gjg\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 11:29:43.963210 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:29:43.963171    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-tjbr8" event={"ID":"60e50fe5-0b57-4c56-be9f-7b46d260f10e","Type":"ContainerDied","Data":"ce5ae414c3daecb128f0724ba760acf7818bb2a65aa536777041613eb92f42d3"}
Jul 28 11:29:43.963210 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:29:43.963209    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-tjbr8"
Jul 28 11:29:43.963752 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:29:43.963244    2579 scope.go:117] "RemoveContainer" containerID="158802b7482712be318e18644969a7469759a97f2594386f23254bd70c86b666"
Jul 28 11:29:43.986217 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:29:43.986184    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-tjbr8"]
Jul 28 11:29:43.990583 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:29:43.990556    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-tjbr8"]
Jul 28 11:29:44.281527 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:29:44.281431    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="60e50fe5-0b57-4c56-be9f-7b46d260f10e" path="/var/lib/kubelet/pods/60e50fe5-0b57-4c56-be9f-7b46d260f10e/volumes"
Jul 28 11:29:46.835244 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:29:46.835214    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:29:51.674393 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:29:51.674356    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:29:51.924639 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:29:51.924557    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-vbtgt"]
Jul 28 11:29:51.924900 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:29:51.924889    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60e50fe5-0b57-4c56-be9f-7b46d260f10e" containerName="main"
Jul 28 11:29:51.924945 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:29:51.924902    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="60e50fe5-0b57-4c56-be9f-7b46d260f10e" containerName="main"
Jul 28 11:29:51.924987 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:29:51.924974    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="60e50fe5-0b57-4c56-be9f-7b46d260f10e" containerName="main"
Jul 28 11:29:51.929271 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:29:51.929253    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-vbtgt"
Jul 28 11:29:51.931950 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:29:51.931929    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"openshift-service-ca.crt\""
Jul 28 11:29:51.932085 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:29:51.932022    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"kube-root-ca.crt\""
Jul 28 11:29:51.932085 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:29:51.932034    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-kserve-self-signed-certs\""
Jul 28 11:29:51.932085 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:29:51.932022    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-epp-sa-dockercfg-cxdfv\""
Jul 28 11:29:51.932253 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:29:51.932022    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"default-dockercfg-65z7g\""
Jul 28 11:29:51.936943 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:29:51.936930    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-vbtgt"]
Jul 28 11:29:51.979292 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:29:51.979258    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:29:52.029239 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:29:52.029202    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7847f977-5e86-457c-9389-42f490feb37f-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-6ff9c88bb8-vbtgt\" (UID: \"7847f977-5e86-457c-9389-42f490feb37f\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-vbtgt"
Jul 28 11:29:52.029405 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:29:52.029253    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znslm\" (UniqueName: \"kubernetes.io/projected/7847f977-5e86-457c-9389-42f490feb37f-kube-api-access-znslm\") pod \"stop-feature-test-kserve-router-scheduler-6ff9c88bb8-vbtgt\" (UID: \"7847f977-5e86-457c-9389-42f490feb37f\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-vbtgt"
Jul 28 11:29:52.130507 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:29:52.130464    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7847f977-5e86-457c-9389-42f490feb37f-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-6ff9c88bb8-vbtgt\" (UID: \"7847f977-5e86-457c-9389-42f490feb37f\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-vbtgt"
Jul 28 11:29:52.130740 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:29:52.130525    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znslm\" (UniqueName: \"kubernetes.io/projected/7847f977-5e86-457c-9389-42f490feb37f-kube-api-access-znslm\") pod \"stop-feature-test-kserve-router-scheduler-6ff9c88bb8-vbtgt\" (UID: \"7847f977-5e86-457c-9389-42f490feb37f\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-vbtgt"
Jul 28 11:29:52.133162 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:29:52.133136    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7847f977-5e86-457c-9389-42f490feb37f-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-6ff9c88bb8-vbtgt\" (UID: \"7847f977-5e86-457c-9389-42f490feb37f\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-vbtgt"
Jul 28 11:29:52.142069 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:29:52.142036    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-znslm\" (UniqueName: \"kubernetes.io/projected/7847f977-5e86-457c-9389-42f490feb37f-kube-api-access-znslm\") pod \"stop-feature-test-kserve-router-scheduler-6ff9c88bb8-vbtgt\" (UID: \"7847f977-5e86-457c-9389-42f490feb37f\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-vbtgt"
Jul 28 11:29:52.240394 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:29:52.240300    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-vbtgt"
Jul 28 11:29:52.381425 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:29:52.381399    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-vbtgt"]
Jul 28 11:29:52.383632 ip-10-0-141-50 kubenswrapper[2579]: W0728 11:29:52.383605    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7847f977_5e86_457c_9389_42f490feb37f.slice/crio-447699a122f30c2c17bbda722a2a2cc4999ac22338bd852fcf191168e6e8964f WatchSource:0}: Error finding container 447699a122f30c2c17bbda722a2a2cc4999ac22338bd852fcf191168e6e8964f: Status 404 returned error can't find the container with id 447699a122f30c2c17bbda722a2a2cc4999ac22338bd852fcf191168e6e8964f
Jul 28 11:29:53.009662 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:29:53.009625    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-vbtgt" event={"ID":"7847f977-5e86-457c-9389-42f490feb37f","Type":"ContainerStarted","Data":"72800939a34e98725a41e7d88fb6238f9a008ef1215619a852b9f12e4a7ea4c9"}
Jul 28 11:29:53.010161 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:29:53.009687    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-vbtgt" event={"ID":"7847f977-5e86-457c-9389-42f490feb37f","Type":"ContainerStarted","Data":"447699a122f30c2c17bbda722a2a2cc4999ac22338bd852fcf191168e6e8964f"}
Jul 28 11:29:53.010161 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:29:53.009838    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-vbtgt"
Jul 28 11:29:53.027885 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:29:53.027836    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-vbtgt" podStartSLOduration=2.027818513 podStartE2EDuration="2.027818513s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 11:29:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 11:29:53.026450088 +0000 UTC m=+4927.289846428" watchObservedRunningTime="2026-07-28 11:29:53.027818513 +0000 UTC m=+4927.291214841"
Jul 28 11:30:19.517489 ip-10-0-141-50 kubenswrapper[2579]: E0728 11:30:19.517396    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bgw9ln" podUID="9f2f8d3c-6c27-4e8b-92c4-d22da1bb1cbe"
Jul 28 11:30:20.140729 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:30:20.140695    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bgw9ln"
Jul 28 11:30:23.894500 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:30:23.894469    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:30:24.017281 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:30:24.017253    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-vbtgt"
Jul 28 11:30:58.991611 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:30:58.991575    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:30:59.092156 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:30:59.092119    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:32:23.141639 ip-10-0-141-50 kubenswrapper[2579]: E0728 11:32:23.141583    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bgw9ln" podUID="9f2f8d3c-6c27-4e8b-92c4-d22da1bb1cbe"
Jul 28 11:32:23.754165 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:32:23.754130    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bgw9ln"
Jul 28 11:32:26.071570 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:32:26.071522    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-vbtgt"]
Jul 28 11:32:26.085740 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:32:26.085703    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:32:28.125141 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:32:28.125106    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:32:34.016702 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:32:34.016659    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-vbtgt" podUID="7847f977-5e86-457c-9389-42f490feb37f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 11:32:41.090488 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:32:41.090396    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-vbtgt" podUID="7847f977-5e86-457c-9389-42f490feb37f" containerName="main" containerID="cri-o://72800939a34e98725a41e7d88fb6238f9a008ef1215619a852b9f12e4a7ea4c9" gracePeriod=45
Jul 28 11:32:41.346686 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:32:41.346619    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-vbtgt"
Jul 28 11:32:41.413375 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:32:41.413335    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-znslm\" (UniqueName: \"kubernetes.io/projected/7847f977-5e86-457c-9389-42f490feb37f-kube-api-access-znslm\") pod \"7847f977-5e86-457c-9389-42f490feb37f\" (UID: \"7847f977-5e86-457c-9389-42f490feb37f\") "
Jul 28 11:32:41.413566 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:32:41.413421    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7847f977-5e86-457c-9389-42f490feb37f-tls-certs\") pod \"7847f977-5e86-457c-9389-42f490feb37f\" (UID: \"7847f977-5e86-457c-9389-42f490feb37f\") "
Jul 28 11:32:41.415575 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:32:41.415520    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7847f977-5e86-457c-9389-42f490feb37f-kube-api-access-znslm" (OuterVolumeSpecName: "kube-api-access-znslm") pod "7847f977-5e86-457c-9389-42f490feb37f" (UID: "7847f977-5e86-457c-9389-42f490feb37f"). InnerVolumeSpecName "kube-api-access-znslm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 11:32:41.415575 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:32:41.415552    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7847f977-5e86-457c-9389-42f490feb37f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7847f977-5e86-457c-9389-42f490feb37f" (UID: "7847f977-5e86-457c-9389-42f490feb37f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 11:32:41.514118 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:32:41.514083    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7847f977-5e86-457c-9389-42f490feb37f-tls-certs\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 11:32:41.514118 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:32:41.514114    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-znslm\" (UniqueName: \"kubernetes.io/projected/7847f977-5e86-457c-9389-42f490feb37f-kube-api-access-znslm\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 11:32:41.858632 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:32:41.858592    2579 generic.go:356] "Generic (PLEG): container finished" podID="7847f977-5e86-457c-9389-42f490feb37f" containerID="72800939a34e98725a41e7d88fb6238f9a008ef1215619a852b9f12e4a7ea4c9" exitCode=0
Jul 28 11:32:41.858632 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:32:41.858645    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-vbtgt"
Jul 28 11:32:41.858929 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:32:41.858689    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-vbtgt" event={"ID":"7847f977-5e86-457c-9389-42f490feb37f","Type":"ContainerDied","Data":"72800939a34e98725a41e7d88fb6238f9a008ef1215619a852b9f12e4a7ea4c9"}
Jul 28 11:32:41.858929 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:32:41.858760    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-vbtgt" event={"ID":"7847f977-5e86-457c-9389-42f490feb37f","Type":"ContainerDied","Data":"447699a122f30c2c17bbda722a2a2cc4999ac22338bd852fcf191168e6e8964f"}
Jul 28 11:32:41.858929 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:32:41.858800    2579 scope.go:117] "RemoveContainer" containerID="72800939a34e98725a41e7d88fb6238f9a008ef1215619a852b9f12e4a7ea4c9"
Jul 28 11:32:41.871732 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:32:41.871701    2579 scope.go:117] "RemoveContainer" containerID="72800939a34e98725a41e7d88fb6238f9a008ef1215619a852b9f12e4a7ea4c9"
Jul 28 11:32:41.872058 ip-10-0-141-50 kubenswrapper[2579]: E0728 11:32:41.872034    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72800939a34e98725a41e7d88fb6238f9a008ef1215619a852b9f12e4a7ea4c9\": container with ID starting with 72800939a34e98725a41e7d88fb6238f9a008ef1215619a852b9f12e4a7ea4c9 not found: ID does not exist" containerID="72800939a34e98725a41e7d88fb6238f9a008ef1215619a852b9f12e4a7ea4c9"
Jul 28 11:32:41.872165 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:32:41.872063    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72800939a34e98725a41e7d88fb6238f9a008ef1215619a852b9f12e4a7ea4c9"} err="failed to get container status \"72800939a34e98725a41e7d88fb6238f9a008ef1215619a852b9f12e4a7ea4c9\": rpc error: code = NotFound desc = could not find container \"72800939a34e98725a41e7d88fb6238f9a008ef1215619a852b9f12e4a7ea4c9\": container with ID starting with 72800939a34e98725a41e7d88fb6238f9a008ef1215619a852b9f12e4a7ea4c9 not found: ID does not exist"
Jul 28 11:32:41.883055 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:32:41.883022    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-vbtgt"]
Jul 28 11:32:41.896642 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:32:41.893920    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-vbtgt"]
Jul 28 11:32:42.282897 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:32:42.282859    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7847f977-5e86-457c-9389-42f490feb37f" path="/var/lib/kubelet/pods/7847f977-5e86-457c-9389-42f490feb37f/volumes"
Jul 28 11:32:46.517485 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:32:46.517448    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:32:46.612962 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:32:46.612924    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:32:51.936202 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:32:51.936169    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:32:57.214755 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:32:57.214713    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:32:57.281882 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:32:57.281847    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:33:15.392783 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:33:15.392743    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-67d54cdd4c-vdg68"]
Jul 28 11:33:15.393257 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:33:15.393125    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7847f977-5e86-457c-9389-42f490feb37f" containerName="main"
Jul 28 11:33:15.393257 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:33:15.393138    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="7847f977-5e86-457c-9389-42f490feb37f" containerName="main"
Jul 28 11:33:15.393257 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:33:15.393201    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="7847f977-5e86-457c-9389-42f490feb37f" containerName="main"
Jul 28 11:33:15.396056 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:33:15.396037    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-67d54cdd4c-vdg68"
Jul 28 11:33:15.398640 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:33:15.398609    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"kube-root-ca.crt\""
Jul 28 11:33:15.398888 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:33:15.398870    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"lora-single-adapter-test-kserve-self-signed-certs\""
Jul 28 11:33:15.399108 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:33:15.399086    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"openshift-service-ca.crt\""
Jul 28 11:33:15.399272 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:33:15.399138    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 11:33:15.399384 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:33:15.399153    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"default-dockercfg-c2675\""
Jul 28 11:33:15.399384 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:33:15.399153    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"seaweedfs-s3-creds\""
Jul 28 11:33:15.407784 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:33:15.407759    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-67d54cdd4c-vdg68"]
Jul 28 11:33:15.444110 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:33:15.444075    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:33:15.511176 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:33:15.511142    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7gwv\" (UniqueName: \"kubernetes.io/projected/ddf1a404-943c-4200-8e26-bdcead4f81be-kube-api-access-q7gwv\") pod \"lora-single-adapter-test-kserve-67d54cdd4c-vdg68\" (UID: \"ddf1a404-943c-4200-8e26-bdcead4f81be\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-67d54cdd4c-vdg68"
Jul 28 11:33:15.511370 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:33:15.511188    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ddf1a404-943c-4200-8e26-bdcead4f81be-tls-certs\") pod \"lora-single-adapter-test-kserve-67d54cdd4c-vdg68\" (UID: \"ddf1a404-943c-4200-8e26-bdcead4f81be\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-67d54cdd4c-vdg68"
Jul 28 11:33:15.511370 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:33:15.511240    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ddf1a404-943c-4200-8e26-bdcead4f81be-cabundle-cert\") pod \"lora-single-adapter-test-kserve-67d54cdd4c-vdg68\" (UID: \"ddf1a404-943c-4200-8e26-bdcead4f81be\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-67d54cdd4c-vdg68"
Jul 28 11:33:15.511370 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:33:15.511281    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ddf1a404-943c-4200-8e26-bdcead4f81be-tmp-dir\") pod \"lora-single-adapter-test-kserve-67d54cdd4c-vdg68\" (UID: \"ddf1a404-943c-4200-8e26-bdcead4f81be\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-67d54cdd4c-vdg68"
Jul 28 11:33:15.511370 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:33:15.511316    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ddf1a404-943c-4200-8e26-bdcead4f81be-home\") pod \"lora-single-adapter-test-kserve-67d54cdd4c-vdg68\" (UID: \"ddf1a404-943c-4200-8e26-bdcead4f81be\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-67d54cdd4c-vdg68"
Jul 28 11:33:15.511370 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:33:15.511356    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ddf1a404-943c-4200-8e26-bdcead4f81be-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-67d54cdd4c-vdg68\" (UID: \"ddf1a404-943c-4200-8e26-bdcead4f81be\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-67d54cdd4c-vdg68"
Jul 28 11:33:15.511662 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:33:15.511416    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ddf1a404-943c-4200-8e26-bdcead4f81be-dshm\") pod \"lora-single-adapter-test-kserve-67d54cdd4c-vdg68\" (UID: \"ddf1a404-943c-4200-8e26-bdcead4f81be\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-67d54cdd4c-vdg68"
Jul 28 11:33:15.511662 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:33:15.511474    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ddf1a404-943c-4200-8e26-bdcead4f81be-model-cache\") pod \"lora-single-adapter-test-kserve-67d54cdd4c-vdg68\" (UID: \"ddf1a404-943c-4200-8e26-bdcead4f81be\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-67d54cdd4c-vdg68"
Jul 28 11:33:15.612311 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:33:15.612274    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ddf1a404-943c-4200-8e26-bdcead4f81be-tmp-dir\") pod \"lora-single-adapter-test-kserve-67d54cdd4c-vdg68\" (UID: \"ddf1a404-943c-4200-8e26-bdcead4f81be\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-67d54cdd4c-vdg68"
Jul 28 11:33:15.612311 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:33:15.612313    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ddf1a404-943c-4200-8e26-bdcead4f81be-home\") pod \"lora-single-adapter-test-kserve-67d54cdd4c-vdg68\" (UID: \"ddf1a404-943c-4200-8e26-bdcead4f81be\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-67d54cdd4c-vdg68"
Jul 28 11:33:15.612600 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:33:15.612357    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ddf1a404-943c-4200-8e26-bdcead4f81be-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-67d54cdd4c-vdg68\" (UID: \"ddf1a404-943c-4200-8e26-bdcead4f81be\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-67d54cdd4c-vdg68"
Jul 28 11:33:15.612600 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:33:15.612381    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ddf1a404-943c-4200-8e26-bdcead4f81be-dshm\") pod \"lora-single-adapter-test-kserve-67d54cdd4c-vdg68\" (UID: \"ddf1a404-943c-4200-8e26-bdcead4f81be\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-67d54cdd4c-vdg68"
Jul 28 11:33:15.612600 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:33:15.612416    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ddf1a404-943c-4200-8e26-bdcead4f81be-model-cache\") pod \"lora-single-adapter-test-kserve-67d54cdd4c-vdg68\" (UID: \"ddf1a404-943c-4200-8e26-bdcead4f81be\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-67d54cdd4c-vdg68"
Jul 28 11:33:15.612600 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:33:15.612438    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7gwv\" (UniqueName: \"kubernetes.io/projected/ddf1a404-943c-4200-8e26-bdcead4f81be-kube-api-access-q7gwv\") pod \"lora-single-adapter-test-kserve-67d54cdd4c-vdg68\" (UID: \"ddf1a404-943c-4200-8e26-bdcead4f81be\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-67d54cdd4c-vdg68"
Jul 28 11:33:15.612600 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:33:15.612494    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ddf1a404-943c-4200-8e26-bdcead4f81be-tls-certs\") pod \"lora-single-adapter-test-kserve-67d54cdd4c-vdg68\" (UID: \"ddf1a404-943c-4200-8e26-bdcead4f81be\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-67d54cdd4c-vdg68"
Jul 28 11:33:15.612600 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:33:15.612592    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ddf1a404-943c-4200-8e26-bdcead4f81be-cabundle-cert\") pod \"lora-single-adapter-test-kserve-67d54cdd4c-vdg68\" (UID: \"ddf1a404-943c-4200-8e26-bdcead4f81be\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-67d54cdd4c-vdg68"
Jul 28 11:33:15.612940 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:33:15.612756    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ddf1a404-943c-4200-8e26-bdcead4f81be-tmp-dir\") pod \"lora-single-adapter-test-kserve-67d54cdd4c-vdg68\" (UID: \"ddf1a404-943c-4200-8e26-bdcead4f81be\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-67d54cdd4c-vdg68"
Jul 28 11:33:15.612940 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:33:15.612849    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ddf1a404-943c-4200-8e26-bdcead4f81be-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-67d54cdd4c-vdg68\" (UID: \"ddf1a404-943c-4200-8e26-bdcead4f81be\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-67d54cdd4c-vdg68"
Jul 28 11:33:15.613115 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:33:15.613096    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ddf1a404-943c-4200-8e26-bdcead4f81be-model-cache\") pod \"lora-single-adapter-test-kserve-67d54cdd4c-vdg68\" (UID: \"ddf1a404-943c-4200-8e26-bdcead4f81be\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-67d54cdd4c-vdg68"
Jul 28 11:33:15.613170 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:33:15.613146    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ddf1a404-943c-4200-8e26-bdcead4f81be-home\") pod \"lora-single-adapter-test-kserve-67d54cdd4c-vdg68\" (UID: \"ddf1a404-943c-4200-8e26-bdcead4f81be\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-67d54cdd4c-vdg68"
Jul 28 11:33:15.613311 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:33:15.613291    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ddf1a404-943c-4200-8e26-bdcead4f81be-cabundle-cert\") pod \"lora-single-adapter-test-kserve-67d54cdd4c-vdg68\" (UID: \"ddf1a404-943c-4200-8e26-bdcead4f81be\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-67d54cdd4c-vdg68"
Jul 28 11:33:15.615002 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:33:15.614980    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ddf1a404-943c-4200-8e26-bdcead4f81be-dshm\") pod \"lora-single-adapter-test-kserve-67d54cdd4c-vdg68\" (UID: \"ddf1a404-943c-4200-8e26-bdcead4f81be\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-67d54cdd4c-vdg68"
Jul 28 11:33:15.615200 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:33:15.615183    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ddf1a404-943c-4200-8e26-bdcead4f81be-tls-certs\") pod \"lora-single-adapter-test-kserve-67d54cdd4c-vdg68\" (UID: \"ddf1a404-943c-4200-8e26-bdcead4f81be\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-67d54cdd4c-vdg68"
Jul 28 11:33:15.620205 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:33:15.620175    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7gwv\" (UniqueName: \"kubernetes.io/projected/ddf1a404-943c-4200-8e26-bdcead4f81be-kube-api-access-q7gwv\") pod \"lora-single-adapter-test-kserve-67d54cdd4c-vdg68\" (UID: \"ddf1a404-943c-4200-8e26-bdcead4f81be\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-67d54cdd4c-vdg68"
Jul 28 11:33:15.708478 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:33:15.708390    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-67d54cdd4c-vdg68"
Jul 28 11:33:15.849486 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:33:15.849461    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-67d54cdd4c-vdg68"]
Jul 28 11:33:15.851239 ip-10-0-141-50 kubenswrapper[2579]: W0728 11:33:15.851213    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddf1a404_943c_4200_8e26_bdcead4f81be.slice/crio-32ab2638e1229cbcb7dad4615830dd9f745b2b1a7e85eb04e14ad1c5756a3add WatchSource:0}: Error finding container 32ab2638e1229cbcb7dad4615830dd9f745b2b1a7e85eb04e14ad1c5756a3add: Status 404 returned error can't find the container with id 32ab2638e1229cbcb7dad4615830dd9f745b2b1a7e85eb04e14ad1c5756a3add
Jul 28 11:33:16.036756 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:33:16.036725    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-67d54cdd4c-vdg68" event={"ID":"ddf1a404-943c-4200-8e26-bdcead4f81be","Type":"ContainerStarted","Data":"850fa53bb05071cfda04e05c1bd5f8af23424e5a697088058918603139c66e08"}
Jul 28 11:33:16.036924 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:33:16.036778    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-67d54cdd4c-vdg68" event={"ID":"ddf1a404-943c-4200-8e26-bdcead4f81be","Type":"ContainerStarted","Data":"32ab2638e1229cbcb7dad4615830dd9f745b2b1a7e85eb04e14ad1c5756a3add"}
Jul 28 11:33:21.074423 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:33:21.074388    2579 generic.go:356] "Generic (PLEG): container finished" podID="ddf1a404-943c-4200-8e26-bdcead4f81be" containerID="850fa53bb05071cfda04e05c1bd5f8af23424e5a697088058918603139c66e08" exitCode=0
Jul 28 11:33:21.074857 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:33:21.074475    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-67d54cdd4c-vdg68" event={"ID":"ddf1a404-943c-4200-8e26-bdcead4f81be","Type":"ContainerDied","Data":"850fa53bb05071cfda04e05c1bd5f8af23424e5a697088058918603139c66e08"}
Jul 28 11:33:22.081643 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:33:22.081605    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-67d54cdd4c-vdg68" event={"ID":"ddf1a404-943c-4200-8e26-bdcead4f81be","Type":"ContainerStarted","Data":"6741a9830f93609ef49090c8a6b622f03f596a6d7d980ac04e9184ea258c3b4f"}
Jul 28 11:33:22.108672 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:33:22.108615    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-67d54cdd4c-vdg68" podStartSLOduration=1.9813678989999999 podStartE2EDuration="7.108593996s" totalImagesPullingTime="0s" totalInitContainerRuntime="5.127226097s" isStatefulPod=false podCreationTimestamp="2026-07-28 11:33:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 11:33:22.10710115 +0000 UTC m=+5136.370497492" watchObservedRunningTime="2026-07-28 11:33:22.108593996 +0000 UTC m=+5136.371990325"
Jul 28 11:33:22.708756 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:33:22.708714    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-67d54cdd4c-vdg68"
Jul 28 11:33:25.709238 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:33:25.709182    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-67d54cdd4c-vdg68"
Jul 28 11:33:25.710965 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:33:25.710934    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-67d54cdd4c-vdg68" podUID="ddf1a404-943c-4200-8e26-bdcead4f81be" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": dial tcp 10.134.0.72:8000: connect: connection refused"
Jul 28 11:33:35.709231 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:33:35.709181    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-67d54cdd4c-vdg68" podUID="ddf1a404-943c-4200-8e26-bdcead4f81be" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": dial tcp 10.134.0.72:8000: connect: connection refused"
Jul 28 11:33:45.709591 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:33:45.709516    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-67d54cdd4c-vdg68" podUID="ddf1a404-943c-4200-8e26-bdcead4f81be" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": dial tcp 10.134.0.72:8000: connect: connection refused"
Jul 28 11:33:55.709733 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:33:55.709683    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-67d54cdd4c-vdg68" podUID="ddf1a404-943c-4200-8e26-bdcead4f81be" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": dial tcp 10.134.0.72:8000: connect: connection refused"
Jul 28 11:34:05.709314 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:34:05.709257    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-67d54cdd4c-vdg68" podUID="ddf1a404-943c-4200-8e26-bdcead4f81be" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": dial tcp 10.134.0.72:8000: connect: connection refused"
Jul 28 11:34:15.709620 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:34:15.709565    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-67d54cdd4c-vdg68" podUID="ddf1a404-943c-4200-8e26-bdcead4f81be" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": dial tcp 10.134.0.72:8000: connect: connection refused"
Jul 28 11:34:25.709279 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:34:25.709230    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-67d54cdd4c-vdg68" podUID="ddf1a404-943c-4200-8e26-bdcead4f81be" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": dial tcp 10.134.0.72:8000: connect: connection refused"
Jul 28 11:34:26.755744 ip-10-0-141-50 kubenswrapper[2579]: E0728 11:34:26.755693    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bgw9ln" podUID="9f2f8d3c-6c27-4e8b-92c4-d22da1bb1cbe"
Jul 28 11:34:27.474548 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:34:27.474513    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bgw9ln"
Jul 28 11:34:35.709893 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:34:35.709843    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-67d54cdd4c-vdg68" podUID="ddf1a404-943c-4200-8e26-bdcead4f81be" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": dial tcp 10.134.0.72:8000: connect: connection refused"
Jul 28 11:34:45.719841 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:34:45.719760    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-67d54cdd4c-vdg68"
Jul 28 11:34:45.728352 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:34:45.728321    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-67d54cdd4c-vdg68"
Jul 28 11:34:53.246707 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:34:53.246667    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:35:41.626477 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:35:41.626441    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-3cbf5e9c_lora-single-adapter-test-kserve-67d54cdd4c-vdg68_ddf1a404-943c-4200-8e26-bdcead4f81be/storage-initializer/0.log"
Jul 28 11:35:41.638788 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:35:41.638753    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-3cbf5e9c_lora-single-adapter-test-kserve-67d54cdd4c-vdg68_ddf1a404-943c-4200-8e26-bdcead4f81be/main/0.log"
Jul 28 11:35:55.642579 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:35:55.642526    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-67d54cdd4c-vdg68"]
Jul 28 11:35:55.659611 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:35:55.659578    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:36:10.664948 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:36:10.664840    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-67d54cdd4c-vdg68" podUID="ddf1a404-943c-4200-8e26-bdcead4f81be" containerName="main" containerID="cri-o://6741a9830f93609ef49090c8a6b622f03f596a6d7d980ac04e9184ea258c3b4f" gracePeriod=45
Jul 28 11:36:11.327963 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:36:11.327934    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-67d54cdd4c-vdg68"
Jul 28 11:36:11.361297 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:36:11.361262    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ddf1a404-943c-4200-8e26-bdcead4f81be-cabundle-cert\") pod \"ddf1a404-943c-4200-8e26-bdcead4f81be\" (UID: \"ddf1a404-943c-4200-8e26-bdcead4f81be\") "
Jul 28 11:36:11.361472 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:36:11.361311    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ddf1a404-943c-4200-8e26-bdcead4f81be-dshm\") pod \"ddf1a404-943c-4200-8e26-bdcead4f81be\" (UID: \"ddf1a404-943c-4200-8e26-bdcead4f81be\") "
Jul 28 11:36:11.361472 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:36:11.361334    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ddf1a404-943c-4200-8e26-bdcead4f81be-tmp-dir\") pod \"ddf1a404-943c-4200-8e26-bdcead4f81be\" (UID: \"ddf1a404-943c-4200-8e26-bdcead4f81be\") "
Jul 28 11:36:11.361472 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:36:11.361349    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ddf1a404-943c-4200-8e26-bdcead4f81be-kserve-provision-location\") pod \"ddf1a404-943c-4200-8e26-bdcead4f81be\" (UID: \"ddf1a404-943c-4200-8e26-bdcead4f81be\") "
Jul 28 11:36:11.361472 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:36:11.361458    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ddf1a404-943c-4200-8e26-bdcead4f81be-tls-certs\") pod \"ddf1a404-943c-4200-8e26-bdcead4f81be\" (UID: \"ddf1a404-943c-4200-8e26-bdcead4f81be\") "
Jul 28 11:36:11.361732 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:36:11.361512    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ddf1a404-943c-4200-8e26-bdcead4f81be-home\") pod \"ddf1a404-943c-4200-8e26-bdcead4f81be\" (UID: \"ddf1a404-943c-4200-8e26-bdcead4f81be\") "
Jul 28 11:36:11.361732 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:36:11.361594    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ddf1a404-943c-4200-8e26-bdcead4f81be-model-cache\") pod \"ddf1a404-943c-4200-8e26-bdcead4f81be\" (UID: \"ddf1a404-943c-4200-8e26-bdcead4f81be\") "
Jul 28 11:36:11.361732 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:36:11.361649    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q7gwv\" (UniqueName: \"kubernetes.io/projected/ddf1a404-943c-4200-8e26-bdcead4f81be-kube-api-access-q7gwv\") pod \"ddf1a404-943c-4200-8e26-bdcead4f81be\" (UID: \"ddf1a404-943c-4200-8e26-bdcead4f81be\") "
Jul 28 11:36:11.361892 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:36:11.361732    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddf1a404-943c-4200-8e26-bdcead4f81be-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ddf1a404-943c-4200-8e26-bdcead4f81be" (UID: "ddf1a404-943c-4200-8e26-bdcead4f81be"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 11:36:11.361948 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:36:11.361928    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ddf1a404-943c-4200-8e26-bdcead4f81be-cabundle-cert\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 11:36:11.362081 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:36:11.362056    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddf1a404-943c-4200-8e26-bdcead4f81be-model-cache" (OuterVolumeSpecName: "model-cache") pod "ddf1a404-943c-4200-8e26-bdcead4f81be" (UID: "ddf1a404-943c-4200-8e26-bdcead4f81be"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 11:36:11.362395 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:36:11.362327    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddf1a404-943c-4200-8e26-bdcead4f81be-home" (OuterVolumeSpecName: "home") pod "ddf1a404-943c-4200-8e26-bdcead4f81be" (UID: "ddf1a404-943c-4200-8e26-bdcead4f81be"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 11:36:11.364336 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:36:11.364301    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddf1a404-943c-4200-8e26-bdcead4f81be-dshm" (OuterVolumeSpecName: "dshm") pod "ddf1a404-943c-4200-8e26-bdcead4f81be" (UID: "ddf1a404-943c-4200-8e26-bdcead4f81be"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 11:36:11.364554 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:36:11.364505    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ddf1a404-943c-4200-8e26-bdcead4f81be-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ddf1a404-943c-4200-8e26-bdcead4f81be" (UID: "ddf1a404-943c-4200-8e26-bdcead4f81be"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 11:36:11.364674 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:36:11.364570    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddf1a404-943c-4200-8e26-bdcead4f81be-kube-api-access-q7gwv" (OuterVolumeSpecName: "kube-api-access-q7gwv") pod "ddf1a404-943c-4200-8e26-bdcead4f81be" (UID: "ddf1a404-943c-4200-8e26-bdcead4f81be"). InnerVolumeSpecName "kube-api-access-q7gwv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 11:36:11.375382 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:36:11.375350    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddf1a404-943c-4200-8e26-bdcead4f81be-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ddf1a404-943c-4200-8e26-bdcead4f81be" (UID: "ddf1a404-943c-4200-8e26-bdcead4f81be"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 11:36:11.426845 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:36:11.426752    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddf1a404-943c-4200-8e26-bdcead4f81be-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ddf1a404-943c-4200-8e26-bdcead4f81be" (UID: "ddf1a404-943c-4200-8e26-bdcead4f81be"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 11:36:11.463283 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:36:11.463254    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ddf1a404-943c-4200-8e26-bdcead4f81be-home\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 11:36:11.463283 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:36:11.463280    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ddf1a404-943c-4200-8e26-bdcead4f81be-model-cache\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 11:36:11.463482 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:36:11.463292    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7gwv\" (UniqueName: \"kubernetes.io/projected/ddf1a404-943c-4200-8e26-bdcead4f81be-kube-api-access-q7gwv\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 11:36:11.463482 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:36:11.463300    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ddf1a404-943c-4200-8e26-bdcead4f81be-dshm\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 11:36:11.463482 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:36:11.463311    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ddf1a404-943c-4200-8e26-bdcead4f81be-tmp-dir\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 11:36:11.463482 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:36:11.463323    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ddf1a404-943c-4200-8e26-bdcead4f81be-kserve-provision-location\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 11:36:11.463482 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:36:11.463332    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ddf1a404-943c-4200-8e26-bdcead4f81be-tls-certs\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 11:36:12.033091 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:36:12.033056    2579 generic.go:356] "Generic (PLEG): container finished" podID="ddf1a404-943c-4200-8e26-bdcead4f81be" containerID="6741a9830f93609ef49090c8a6b622f03f596a6d7d980ac04e9184ea258c3b4f" exitCode=0
Jul 28 11:36:12.033509 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:36:12.033096    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-67d54cdd4c-vdg68" event={"ID":"ddf1a404-943c-4200-8e26-bdcead4f81be","Type":"ContainerDied","Data":"6741a9830f93609ef49090c8a6b622f03f596a6d7d980ac04e9184ea258c3b4f"}
Jul 28 11:36:12.033509 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:36:12.033109    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-67d54cdd4c-vdg68"
Jul 28 11:36:12.033509 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:36:12.033165    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-67d54cdd4c-vdg68" event={"ID":"ddf1a404-943c-4200-8e26-bdcead4f81be","Type":"ContainerDied","Data":"32ab2638e1229cbcb7dad4615830dd9f745b2b1a7e85eb04e14ad1c5756a3add"}
Jul 28 11:36:12.033509 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:36:12.033187    2579 scope.go:117] "RemoveContainer" containerID="6741a9830f93609ef49090c8a6b622f03f596a6d7d980ac04e9184ea258c3b4f"
Jul 28 11:36:12.042377 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:36:12.042353    2579 scope.go:117] "RemoveContainer" containerID="850fa53bb05071cfda04e05c1bd5f8af23424e5a697088058918603139c66e08"
Jul 28 11:36:12.051172 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:36:12.051152    2579 scope.go:117] "RemoveContainer" containerID="6741a9830f93609ef49090c8a6b622f03f596a6d7d980ac04e9184ea258c3b4f"
Jul 28 11:36:12.051452 ip-10-0-141-50 kubenswrapper[2579]: E0728 11:36:12.051432    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6741a9830f93609ef49090c8a6b622f03f596a6d7d980ac04e9184ea258c3b4f\": container with ID starting with 6741a9830f93609ef49090c8a6b622f03f596a6d7d980ac04e9184ea258c3b4f not found: ID does not exist" containerID="6741a9830f93609ef49090c8a6b622f03f596a6d7d980ac04e9184ea258c3b4f"
Jul 28 11:36:12.051550 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:36:12.051468    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6741a9830f93609ef49090c8a6b622f03f596a6d7d980ac04e9184ea258c3b4f"} err="failed to get container status \"6741a9830f93609ef49090c8a6b622f03f596a6d7d980ac04e9184ea258c3b4f\": rpc error: code = NotFound desc = could not find container \"6741a9830f93609ef49090c8a6b622f03f596a6d7d980ac04e9184ea258c3b4f\": container with ID starting with 6741a9830f93609ef49090c8a6b622f03f596a6d7d980ac04e9184ea258c3b4f not found: ID does not exist"
Jul 28 11:36:12.051550 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:36:12.051496    2579 scope.go:117] "RemoveContainer" containerID="850fa53bb05071cfda04e05c1bd5f8af23424e5a697088058918603139c66e08"
Jul 28 11:36:12.051857 ip-10-0-141-50 kubenswrapper[2579]: E0728 11:36:12.051834    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"850fa53bb05071cfda04e05c1bd5f8af23424e5a697088058918603139c66e08\": container with ID starting with 850fa53bb05071cfda04e05c1bd5f8af23424e5a697088058918603139c66e08 not found: ID does not exist" containerID="850fa53bb05071cfda04e05c1bd5f8af23424e5a697088058918603139c66e08"
Jul 28 11:36:12.051899 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:36:12.051868    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"850fa53bb05071cfda04e05c1bd5f8af23424e5a697088058918603139c66e08"} err="failed to get container status \"850fa53bb05071cfda04e05c1bd5f8af23424e5a697088058918603139c66e08\": rpc error: code = NotFound desc = could not find container \"850fa53bb05071cfda04e05c1bd5f8af23424e5a697088058918603139c66e08\": container with ID starting with 850fa53bb05071cfda04e05c1bd5f8af23424e5a697088058918603139c66e08 not found: ID does not exist"
Jul 28 11:36:12.056355 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:36:12.056332    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-67d54cdd4c-vdg68"]
Jul 28 11:36:12.060301 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:36:12.060280    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-67d54cdd4c-vdg68"]
Jul 28 11:36:12.278142 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:36:12.278035    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ddf1a404-943c-4200-8e26-bdcead4f81be" path="/var/lib/kubelet/pods/ddf1a404-943c-4200-8e26-bdcead4f81be/volumes"
Jul 28 11:36:13.612728 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:36:13.612696    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:36:13.716061 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:36:13.716031    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:36:13.814797 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:36:13.814764    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:36:20.029380 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:36:20.029340    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:36:20.130641 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:36:20.130607    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:36:30.475460 ip-10-0-141-50 kubenswrapper[2579]: E0728 11:36:30.475417    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bgw9ln" podUID="9f2f8d3c-6c27-4e8b-92c4-d22da1bb1cbe"
Jul 28 11:36:31.135289 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:36:31.135254    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bgw9ln"
Jul 28 11:36:36.826241 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:36:36.826203    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-t5ptm"]
Jul 28 11:36:36.833063 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:36:36.833031    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:36:36.928403 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:36:36.928366    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:36:37.321804 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:36:37.321764    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"]
Jul 28 11:36:37.330486 ip-10-0-141-50 kubenswrapper[2579]: E0728 11:36:37.330438    2579 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 28 11:36:37.330486 ip-10-0-141-50 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 11:36:37.330486 ip-10-0-141-50 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Jul 28 11:36:37.330486 ip-10-0-141-50 kubenswrapper[2579]:  > containerID="40de22e48ca042538704fdcb2ffde247e283c09913c0796475b8f31db3e2008f" cmd=["/bin/sleep","15"]
Jul 28 11:36:37.330787 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:36:37.330497    2579 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1" message="" err=<
Jul 28 11:36:37.330787 ip-10-0-141-50 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 11:36:37.330787 ip-10-0-141-50 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Jul 28 11:36:37.330787 ip-10-0-141-50 kubenswrapper[2579]:  >
Jul 28 11:36:37.330787 ip-10-0-141-50 kubenswrapper[2579]: E0728 11:36:37.330509    2579 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 28 11:36:37.330787 ip-10-0-141-50 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 11:36:37.330787 ip-10-0-141-50 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Jul 28 11:36:37.330787 ip-10-0-141-50 kubenswrapper[2579]:  > pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1" podUID="18ad311c-36fc-4d60-a0c3-a9282849c227" containerName="main" containerID="cri-o://40de22e48ca042538704fdcb2ffde247e283c09913c0796475b8f31db3e2008f"
Jul 28 11:36:37.330787 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:36:37.330564    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1" podUID="18ad311c-36fc-4d60-a0c3-a9282849c227" containerName="main" containerID="cri-o://40de22e48ca042538704fdcb2ffde247e283c09913c0796475b8f31db3e2008f" gracePeriod=60
Jul 28 11:36:37.576488 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:36:37.576417    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 28 11:36:37.697306 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:36:37.697273    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18ad311c-36fc-4d60-a0c3-a9282849c227-home\") pod \"18ad311c-36fc-4d60-a0c3-a9282849c227\" (UID: \"18ad311c-36fc-4d60-a0c3-a9282849c227\") "
Jul 28 11:36:37.697496 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:36:37.697343    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18ad311c-36fc-4d60-a0c3-a9282849c227-dshm\") pod \"18ad311c-36fc-4d60-a0c3-a9282849c227\" (UID: \"18ad311c-36fc-4d60-a0c3-a9282849c227\") "
Jul 28 11:36:37.697496 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:36:37.697375    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18ad311c-36fc-4d60-a0c3-a9282849c227-tmp-dir\") pod \"18ad311c-36fc-4d60-a0c3-a9282849c227\" (UID: \"18ad311c-36fc-4d60-a0c3-a9282849c227\") "
Jul 28 11:36:37.697496 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:36:37.697433    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18ad311c-36fc-4d60-a0c3-a9282849c227-tls-certs\") pod \"18ad311c-36fc-4d60-a0c3-a9282849c227\" (UID: \"18ad311c-36fc-4d60-a0c3-a9282849c227\") "
Jul 28 11:36:37.697496 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:36:37.697480    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j2bsd\" (UniqueName: \"kubernetes.io/projected/18ad311c-36fc-4d60-a0c3-a9282849c227-kube-api-access-j2bsd\") pod \"18ad311c-36fc-4d60-a0c3-a9282849c227\" (UID: \"18ad311c-36fc-4d60-a0c3-a9282849c227\") "
Jul 28 11:36:37.697748 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:36:37.697521    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18ad311c-36fc-4d60-a0c3-a9282849c227-model-cache\") pod \"18ad311c-36fc-4d60-a0c3-a9282849c227\" (UID: \"18ad311c-36fc-4d60-a0c3-a9282849c227\") "
Jul 28 11:36:37.697748 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:36:37.697602    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18ad311c-36fc-4d60-a0c3-a9282849c227-home" (OuterVolumeSpecName: "home") pod "18ad311c-36fc-4d60-a0c3-a9282849c227" (UID: "18ad311c-36fc-4d60-a0c3-a9282849c227"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 11:36:37.697858 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:36:37.697773    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18ad311c-36fc-4d60-a0c3-a9282849c227-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "18ad311c-36fc-4d60-a0c3-a9282849c227" (UID: "18ad311c-36fc-4d60-a0c3-a9282849c227"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 11:36:37.697915 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:36:37.697891    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18ad311c-36fc-4d60-a0c3-a9282849c227-home\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 11:36:37.697915 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:36:37.697897    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18ad311c-36fc-4d60-a0c3-a9282849c227-model-cache" (OuterVolumeSpecName: "model-cache") pod "18ad311c-36fc-4d60-a0c3-a9282849c227" (UID: "18ad311c-36fc-4d60-a0c3-a9282849c227"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 11:36:37.698030 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:36:37.697915    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18ad311c-36fc-4d60-a0c3-a9282849c227-tmp-dir\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 11:36:37.699883 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:36:37.699832    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18ad311c-36fc-4d60-a0c3-a9282849c227-kube-api-access-j2bsd" (OuterVolumeSpecName: "kube-api-access-j2bsd") pod "18ad311c-36fc-4d60-a0c3-a9282849c227" (UID: "18ad311c-36fc-4d60-a0c3-a9282849c227"). InnerVolumeSpecName "kube-api-access-j2bsd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 11:36:37.699883 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:36:37.699861    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18ad311c-36fc-4d60-a0c3-a9282849c227-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "18ad311c-36fc-4d60-a0c3-a9282849c227" (UID: "18ad311c-36fc-4d60-a0c3-a9282849c227"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 11:36:37.700032 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:36:37.699891    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18ad311c-36fc-4d60-a0c3-a9282849c227-dshm" (OuterVolumeSpecName: "dshm") pod "18ad311c-36fc-4d60-a0c3-a9282849c227" (UID: "18ad311c-36fc-4d60-a0c3-a9282849c227"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 11:36:37.798757 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:36:37.798719    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18ad311c-36fc-4d60-a0c3-a9282849c227-model-cache\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 11:36:37.798757 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:36:37.798749    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18ad311c-36fc-4d60-a0c3-a9282849c227-dshm\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 11:36:37.798757 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:36:37.798762    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18ad311c-36fc-4d60-a0c3-a9282849c227-tls-certs\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 11:36:37.799017 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:36:37.798775    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2bsd\" (UniqueName: \"kubernetes.io/projected/18ad311c-36fc-4d60-a0c3-a9282849c227-kube-api-access-j2bsd\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 11:36:38.173299 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:36:38.173262    2579 generic.go:356] "Generic (PLEG): container finished" podID="18ad311c-36fc-4d60-a0c3-a9282849c227" containerID="40de22e48ca042538704fdcb2ffde247e283c09913c0796475b8f31db3e2008f" exitCode=0
Jul 28 11:36:38.173789 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:36:38.173308    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 28 11:36:38.173789 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:36:38.173304    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1" event={"ID":"18ad311c-36fc-4d60-a0c3-a9282849c227","Type":"ContainerDied","Data":"40de22e48ca042538704fdcb2ffde247e283c09913c0796475b8f31db3e2008f"}
Jul 28 11:36:38.173789 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:36:38.173434    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1" event={"ID":"18ad311c-36fc-4d60-a0c3-a9282849c227","Type":"ContainerDied","Data":"512f346ce2f1725bccfbc28ad0799780339ade486616f9554b0c74c20e10e1ba"}
Jul 28 11:36:38.173789 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:36:38.173463    2579 scope.go:117] "RemoveContainer" containerID="40de22e48ca042538704fdcb2ffde247e283c09913c0796475b8f31db3e2008f"
Jul 28 11:36:38.182767 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:36:38.182746    2579 scope.go:117] "RemoveContainer" containerID="40de22e48ca042538704fdcb2ffde247e283c09913c0796475b8f31db3e2008f"
Jul 28 11:36:38.183044 ip-10-0-141-50 kubenswrapper[2579]: E0728 11:36:38.183022    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40de22e48ca042538704fdcb2ffde247e283c09913c0796475b8f31db3e2008f\": container with ID starting with 40de22e48ca042538704fdcb2ffde247e283c09913c0796475b8f31db3e2008f not found: ID does not exist" containerID="40de22e48ca042538704fdcb2ffde247e283c09913c0796475b8f31db3e2008f"
Jul 28 11:36:38.183098 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:36:38.183053    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40de22e48ca042538704fdcb2ffde247e283c09913c0796475b8f31db3e2008f"} err="failed to get container status \"40de22e48ca042538704fdcb2ffde247e283c09913c0796475b8f31db3e2008f\": rpc error: code = NotFound desc = could not find container \"40de22e48ca042538704fdcb2ffde247e283c09913c0796475b8f31db3e2008f\": container with ID starting with 40de22e48ca042538704fdcb2ffde247e283c09913c0796475b8f31db3e2008f not found: ID does not exist"
Jul 28 11:36:38.196361 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:36:38.196306    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"]
Jul 28 11:36:38.198407 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:36:38.198385    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"]
Jul 28 11:36:38.275735 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:36:38.275701    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="18ad311c-36fc-4d60-a0c3-a9282849c227" path="/var/lib/kubelet/pods/18ad311c-36fc-4d60-a0c3-a9282849c227/volumes"
Jul 28 11:36:45.570807 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:36:45.570769    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-t5ptm" podUID="8e4537ec-37c7-4e84-8769-a22d4f2451d4" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 11:36:51.848927 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:36:51.848878    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-t5ptm" podUID="8e4537ec-37c7-4e84-8769-a22d4f2451d4" containerName="main" containerID="cri-o://2dd7fd324381edb9375d89f28332895b8c0a6304d833dbae06bff3e177d00841" gracePeriod=45
Jul 28 11:36:52.100209 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:36:52.100138    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-t5ptm"
Jul 28 11:36:52.122415 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:36:52.122381    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e4537ec-37c7-4e84-8769-a22d4f2451d4-tls-certs\") pod \"8e4537ec-37c7-4e84-8769-a22d4f2451d4\" (UID: \"8e4537ec-37c7-4e84-8769-a22d4f2451d4\") "
Jul 28 11:36:52.122625 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:36:52.122416    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wt996\" (UniqueName: \"kubernetes.io/projected/8e4537ec-37c7-4e84-8769-a22d4f2451d4-kube-api-access-wt996\") pod \"8e4537ec-37c7-4e84-8769-a22d4f2451d4\" (UID: \"8e4537ec-37c7-4e84-8769-a22d4f2451d4\") "
Jul 28 11:36:52.124441 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:36:52.124407    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e4537ec-37c7-4e84-8769-a22d4f2451d4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8e4537ec-37c7-4e84-8769-a22d4f2451d4" (UID: "8e4537ec-37c7-4e84-8769-a22d4f2451d4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 11:36:52.124574 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:36:52.124554    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e4537ec-37c7-4e84-8769-a22d4f2451d4-kube-api-access-wt996" (OuterVolumeSpecName: "kube-api-access-wt996") pod "8e4537ec-37c7-4e84-8769-a22d4f2451d4" (UID: "8e4537ec-37c7-4e84-8769-a22d4f2451d4"). InnerVolumeSpecName "kube-api-access-wt996". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 11:36:52.223131 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:36:52.223085    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e4537ec-37c7-4e84-8769-a22d4f2451d4-tls-certs\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 11:36:52.223131 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:36:52.223123    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wt996\" (UniqueName: \"kubernetes.io/projected/8e4537ec-37c7-4e84-8769-a22d4f2451d4-kube-api-access-wt996\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 11:36:52.244296 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:36:52.244262    2579 generic.go:356] "Generic (PLEG): container finished" podID="8e4537ec-37c7-4e84-8769-a22d4f2451d4" containerID="2dd7fd324381edb9375d89f28332895b8c0a6304d833dbae06bff3e177d00841" exitCode=0
Jul 28 11:36:52.244463 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:36:52.244307    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-t5ptm"
Jul 28 11:36:52.244463 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:36:52.244346    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-t5ptm" event={"ID":"8e4537ec-37c7-4e84-8769-a22d4f2451d4","Type":"ContainerDied","Data":"2dd7fd324381edb9375d89f28332895b8c0a6304d833dbae06bff3e177d00841"}
Jul 28 11:36:52.244463 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:36:52.244405    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-t5ptm" event={"ID":"8e4537ec-37c7-4e84-8769-a22d4f2451d4","Type":"ContainerDied","Data":"744b29f821a2754e0bfff6c3f5f86ba0c8b39707c899690d5729a69d7cb42669"}
Jul 28 11:36:52.244463 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:36:52.244435    2579 scope.go:117] "RemoveContainer" containerID="2dd7fd324381edb9375d89f28332895b8c0a6304d833dbae06bff3e177d00841"
Jul 28 11:36:52.253397 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:36:52.253375    2579 scope.go:117] "RemoveContainer" containerID="2dd7fd324381edb9375d89f28332895b8c0a6304d833dbae06bff3e177d00841"
Jul 28 11:36:52.253704 ip-10-0-141-50 kubenswrapper[2579]: E0728 11:36:52.253680    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2dd7fd324381edb9375d89f28332895b8c0a6304d833dbae06bff3e177d00841\": container with ID starting with 2dd7fd324381edb9375d89f28332895b8c0a6304d833dbae06bff3e177d00841 not found: ID does not exist" containerID="2dd7fd324381edb9375d89f28332895b8c0a6304d833dbae06bff3e177d00841"
Jul 28 11:36:52.253786 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:36:52.253715    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2dd7fd324381edb9375d89f28332895b8c0a6304d833dbae06bff3e177d00841"} err="failed to get container status \"2dd7fd324381edb9375d89f28332895b8c0a6304d833dbae06bff3e177d00841\": rpc error: code = NotFound desc = could not find container \"2dd7fd324381edb9375d89f28332895b8c0a6304d833dbae06bff3e177d00841\": container with ID starting with 2dd7fd324381edb9375d89f28332895b8c0a6304d833dbae06bff3e177d00841 not found: ID does not exist"
Jul 28 11:36:52.267312 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:36:52.267277    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-t5ptm"]
Jul 28 11:36:52.275672 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:36:52.275649    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-t5ptm"]
Jul 28 11:36:53.939982 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:36:53.939945    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:36:54.041113 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:36:54.041080    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:36:54.276994 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:36:54.276903    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e4537ec-37c7-4e84-8769-a22d4f2451d4" path="/var/lib/kubelet/pods/8e4537ec-37c7-4e84-8769-a22d4f2451d4/volumes"
Jul 28 11:36:57.418830 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:36:57.418800    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5d95897cf6-dlh6m"]
Jul 28 11:36:57.419212 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:36:57.419107    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ddf1a404-943c-4200-8e26-bdcead4f81be" containerName="main"
Jul 28 11:36:57.419212 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:36:57.419118    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ddf1a404-943c-4200-8e26-bdcead4f81be" containerName="main"
Jul 28 11:36:57.419212 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:36:57.419132    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e4537ec-37c7-4e84-8769-a22d4f2451d4" containerName="main"
Jul 28 11:36:57.419212 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:36:57.419137    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e4537ec-37c7-4e84-8769-a22d4f2451d4" containerName="main"
Jul 28 11:36:57.419212 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:36:57.419145    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18ad311c-36fc-4d60-a0c3-a9282849c227" containerName="main"
Jul 28 11:36:57.419212 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:36:57.419151    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="18ad311c-36fc-4d60-a0c3-a9282849c227" containerName="main"
Jul 28 11:36:57.419212 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:36:57.419163    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ddf1a404-943c-4200-8e26-bdcead4f81be" containerName="storage-initializer"
Jul 28 11:36:57.419212 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:36:57.419169    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ddf1a404-943c-4200-8e26-bdcead4f81be" containerName="storage-initializer"
Jul 28 11:36:57.419511 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:36:57.419218    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="18ad311c-36fc-4d60-a0c3-a9282849c227" containerName="main"
Jul 28 11:36:57.419511 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:36:57.419228    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e4537ec-37c7-4e84-8769-a22d4f2451d4" containerName="main"
Jul 28 11:36:57.419511 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:36:57.419234    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ddf1a404-943c-4200-8e26-bdcead4f81be" containerName="main"
Jul 28 11:36:57.423560 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:36:57.423523    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5d95897cf6-dlh6m"
Jul 28 11:36:57.425833 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:36:57.425808    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"default-dockercfg-2c29q\""
Jul 28 11:36:57.425965 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:36:57.425926    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"openshift-service-ca.crt\""
Jul 28 11:36:57.425965 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:36:57.425940    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"lora-multiple-adapters-test-kserve-self-signed-certs\""
Jul 28 11:36:57.425965 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:36:57.425926    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"seaweedfs-s3-creds\""
Jul 28 11:36:57.426115 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:36:57.426042    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 11:36:57.426275 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:36:57.426263    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"kube-root-ca.crt\""
Jul 28 11:36:57.439843 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:36:57.439820    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5d95897cf6-dlh6m"]
Jul 28 11:36:57.444155 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:36:57.444127    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:36:57.465861 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:36:57.465822    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/63310b78-d9ee-4761-89cd-6abc7f42b296-home\") pod \"lora-multiple-adapters-test-kserve-5d95897cf6-dlh6m\" (UID: \"63310b78-d9ee-4761-89cd-6abc7f42b296\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5d95897cf6-dlh6m"
Jul 28 11:36:57.466034 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:36:57.465868    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63310b78-d9ee-4761-89cd-6abc7f42b296-tls-certs\") pod \"lora-multiple-adapters-test-kserve-5d95897cf6-dlh6m\" (UID: \"63310b78-d9ee-4761-89cd-6abc7f42b296\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5d95897cf6-dlh6m"
Jul 28 11:36:57.466034 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:36:57.465906    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/63310b78-d9ee-4761-89cd-6abc7f42b296-dshm\") pod \"lora-multiple-adapters-test-kserve-5d95897cf6-dlh6m\" (UID: \"63310b78-d9ee-4761-89cd-6abc7f42b296\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5d95897cf6-dlh6m"
Jul 28 11:36:57.466034 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:36:57.465931    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/63310b78-d9ee-4761-89cd-6abc7f42b296-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-5d95897cf6-dlh6m\" (UID: \"63310b78-d9ee-4761-89cd-6abc7f42b296\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5d95897cf6-dlh6m"
Jul 28 11:36:57.466034 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:36:57.466005    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/63310b78-d9ee-4761-89cd-6abc7f42b296-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-5d95897cf6-dlh6m\" (UID: \"63310b78-d9ee-4761-89cd-6abc7f42b296\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5d95897cf6-dlh6m"
Jul 28 11:36:57.466257 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:36:57.466040    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63310b78-d9ee-4761-89cd-6abc7f42b296-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-5d95897cf6-dlh6m\" (UID: \"63310b78-d9ee-4761-89cd-6abc7f42b296\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5d95897cf6-dlh6m"
Jul 28 11:36:57.466257 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:36:57.466086    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82hzk\" (UniqueName: \"kubernetes.io/projected/63310b78-d9ee-4761-89cd-6abc7f42b296-kube-api-access-82hzk\") pod \"lora-multiple-adapters-test-kserve-5d95897cf6-dlh6m\" (UID: \"63310b78-d9ee-4761-89cd-6abc7f42b296\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5d95897cf6-dlh6m"
Jul 28 11:36:57.466257 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:36:57.466156    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63310b78-d9ee-4761-89cd-6abc7f42b296-model-cache\") pod \"lora-multiple-adapters-test-kserve-5d95897cf6-dlh6m\" (UID: \"63310b78-d9ee-4761-89cd-6abc7f42b296\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5d95897cf6-dlh6m"
Jul 28 11:36:57.567178 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:36:57.567141    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/63310b78-d9ee-4761-89cd-6abc7f42b296-home\") pod \"lora-multiple-adapters-test-kserve-5d95897cf6-dlh6m\" (UID: \"63310b78-d9ee-4761-89cd-6abc7f42b296\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5d95897cf6-dlh6m"
Jul 28 11:36:57.567178 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:36:57.567181    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63310b78-d9ee-4761-89cd-6abc7f42b296-tls-certs\") pod \"lora-multiple-adapters-test-kserve-5d95897cf6-dlh6m\" (UID: \"63310b78-d9ee-4761-89cd-6abc7f42b296\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5d95897cf6-dlh6m"
Jul 28 11:36:57.567454 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:36:57.567205    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/63310b78-d9ee-4761-89cd-6abc7f42b296-dshm\") pod \"lora-multiple-adapters-test-kserve-5d95897cf6-dlh6m\" (UID: \"63310b78-d9ee-4761-89cd-6abc7f42b296\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5d95897cf6-dlh6m"
Jul 28 11:36:57.567454 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:36:57.567316    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/63310b78-d9ee-4761-89cd-6abc7f42b296-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-5d95897cf6-dlh6m\" (UID: \"63310b78-d9ee-4761-89cd-6abc7f42b296\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5d95897cf6-dlh6m"
Jul 28 11:36:57.567454 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:36:57.567357    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/63310b78-d9ee-4761-89cd-6abc7f42b296-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-5d95897cf6-dlh6m\" (UID: \"63310b78-d9ee-4761-89cd-6abc7f42b296\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5d95897cf6-dlh6m"
Jul 28 11:36:57.567454 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:36:57.567381    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63310b78-d9ee-4761-89cd-6abc7f42b296-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-5d95897cf6-dlh6m\" (UID: \"63310b78-d9ee-4761-89cd-6abc7f42b296\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5d95897cf6-dlh6m"
Jul 28 11:36:57.567454 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:36:57.567422    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82hzk\" (UniqueName: \"kubernetes.io/projected/63310b78-d9ee-4761-89cd-6abc7f42b296-kube-api-access-82hzk\") pod \"lora-multiple-adapters-test-kserve-5d95897cf6-dlh6m\" (UID: \"63310b78-d9ee-4761-89cd-6abc7f42b296\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5d95897cf6-dlh6m"
Jul 28 11:36:57.567750 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:36:57.567469    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63310b78-d9ee-4761-89cd-6abc7f42b296-model-cache\") pod \"lora-multiple-adapters-test-kserve-5d95897cf6-dlh6m\" (UID: \"63310b78-d9ee-4761-89cd-6abc7f42b296\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5d95897cf6-dlh6m"
Jul 28 11:36:57.567750 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:36:57.567670    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/63310b78-d9ee-4761-89cd-6abc7f42b296-home\") pod \"lora-multiple-adapters-test-kserve-5d95897cf6-dlh6m\" (UID: \"63310b78-d9ee-4761-89cd-6abc7f42b296\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5d95897cf6-dlh6m"
Jul 28 11:36:57.567832 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:36:57.567759    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63310b78-d9ee-4761-89cd-6abc7f42b296-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-5d95897cf6-dlh6m\" (UID: \"63310b78-d9ee-4761-89cd-6abc7f42b296\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5d95897cf6-dlh6m"
Jul 28 11:36:57.567873 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:36:57.567832    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63310b78-d9ee-4761-89cd-6abc7f42b296-model-cache\") pod \"lora-multiple-adapters-test-kserve-5d95897cf6-dlh6m\" (UID: \"63310b78-d9ee-4761-89cd-6abc7f42b296\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5d95897cf6-dlh6m"
Jul 28 11:36:57.567997 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:36:57.567979    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/63310b78-d9ee-4761-89cd-6abc7f42b296-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-5d95897cf6-dlh6m\" (UID: \"63310b78-d9ee-4761-89cd-6abc7f42b296\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5d95897cf6-dlh6m"
Jul 28 11:36:57.568080 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:36:57.568065    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/63310b78-d9ee-4761-89cd-6abc7f42b296-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-5d95897cf6-dlh6m\" (UID: \"63310b78-d9ee-4761-89cd-6abc7f42b296\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5d95897cf6-dlh6m"
Jul 28 11:36:57.569649 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:36:57.569624    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/63310b78-d9ee-4761-89cd-6abc7f42b296-dshm\") pod \"lora-multiple-adapters-test-kserve-5d95897cf6-dlh6m\" (UID: \"63310b78-d9ee-4761-89cd-6abc7f42b296\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5d95897cf6-dlh6m"
Jul 28 11:36:57.569904 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:36:57.569888    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63310b78-d9ee-4761-89cd-6abc7f42b296-tls-certs\") pod \"lora-multiple-adapters-test-kserve-5d95897cf6-dlh6m\" (UID: \"63310b78-d9ee-4761-89cd-6abc7f42b296\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5d95897cf6-dlh6m"
Jul 28 11:36:57.576807 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:36:57.576787    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-82hzk\" (UniqueName: \"kubernetes.io/projected/63310b78-d9ee-4761-89cd-6abc7f42b296-kube-api-access-82hzk\") pod \"lora-multiple-adapters-test-kserve-5d95897cf6-dlh6m\" (UID: \"63310b78-d9ee-4761-89cd-6abc7f42b296\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5d95897cf6-dlh6m"
Jul 28 11:36:57.734123 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:36:57.734032    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5d95897cf6-dlh6m"
Jul 28 11:36:57.876076 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:36:57.876050    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5d95897cf6-dlh6m"]
Jul 28 11:36:57.877863 ip-10-0-141-50 kubenswrapper[2579]: W0728 11:36:57.877822    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63310b78_d9ee_4761_89cd_6abc7f42b296.slice/crio-82f2e870e35c6c7d06947c7631aaaae9af65f2cf1cd368843fc0295e507ffb29 WatchSource:0}: Error finding container 82f2e870e35c6c7d06947c7631aaaae9af65f2cf1cd368843fc0295e507ffb29: Status 404 returned error can't find the container with id 82f2e870e35c6c7d06947c7631aaaae9af65f2cf1cd368843fc0295e507ffb29
Jul 28 11:36:58.283791 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:36:58.283757    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5d95897cf6-dlh6m" event={"ID":"63310b78-d9ee-4761-89cd-6abc7f42b296","Type":"ContainerStarted","Data":"1312d606ec784286f8fa92d0696fd4220ad29323f1c1d4739bc9006fdfc76828"}
Jul 28 11:36:58.283960 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:36:58.283807    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5d95897cf6-dlh6m" event={"ID":"63310b78-d9ee-4761-89cd-6abc7f42b296","Type":"ContainerStarted","Data":"82f2e870e35c6c7d06947c7631aaaae9af65f2cf1cd368843fc0295e507ffb29"}
Jul 28 11:37:03.309857 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:37:03.309822    2579 generic.go:356] "Generic (PLEG): container finished" podID="63310b78-d9ee-4761-89cd-6abc7f42b296" containerID="1312d606ec784286f8fa92d0696fd4220ad29323f1c1d4739bc9006fdfc76828" exitCode=0
Jul 28 11:37:03.310326 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:37:03.309891    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5d95897cf6-dlh6m" event={"ID":"63310b78-d9ee-4761-89cd-6abc7f42b296","Type":"ContainerDied","Data":"1312d606ec784286f8fa92d0696fd4220ad29323f1c1d4739bc9006fdfc76828"}
Jul 28 11:37:04.321927 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:37:04.321887    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5d95897cf6-dlh6m" event={"ID":"63310b78-d9ee-4761-89cd-6abc7f42b296","Type":"ContainerStarted","Data":"c3b7b7ab897332f5372eba2dbcc24cb77ac336bea970d46bb1c3e6db323c4e6e"}
Jul 28 11:37:04.347641 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:37:04.347567    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5d95897cf6-dlh6m" podStartSLOduration=2.877793559 podStartE2EDuration="7.347526736s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.469733177s" isStatefulPod=false podCreationTimestamp="2026-07-28 11:36:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 11:37:04.343613994 +0000 UTC m=+5358.607010324" watchObservedRunningTime="2026-07-28 11:37:04.347526736 +0000 UTC m=+5358.610923067"
Jul 28 11:37:04.734297 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:37:04.734200    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5d95897cf6-dlh6m"
Jul 28 11:37:07.734456 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:37:07.734417    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5d95897cf6-dlh6m"
Jul 28 11:37:07.735849 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:37:07.735813    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5d95897cf6-dlh6m" podUID="63310b78-d9ee-4761-89cd-6abc7f42b296" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/health\": dial tcp 10.134.0.73:8000: connect: connection refused"
Jul 28 11:37:17.735095 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:37:17.735054    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5d95897cf6-dlh6m" podUID="63310b78-d9ee-4761-89cd-6abc7f42b296" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/health\": dial tcp 10.134.0.73:8000: connect: connection refused"
Jul 28 11:37:20.161572 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:37:20.161514    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:37:27.735005 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:37:27.734954    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5d95897cf6-dlh6m" podUID="63310b78-d9ee-4761-89cd-6abc7f42b296" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/health\": dial tcp 10.134.0.73:8000: connect: connection refused"
Jul 28 11:37:28.327258 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:37:28.327223    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:37:28.503063 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:37:28.503021    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-router-scheduler-6694c4458b-vmx75"]
Jul 28 11:37:28.506109 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:37:28.506085    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-router-scheduler-6694c4458b-vmx75"
Jul 28 11:37:28.508751 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:37:28.508723    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40\"/\"openshift-service-ca.crt\""
Jul 28 11:37:28.508908 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:37:28.508824    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40\"/\"autoscale-cleanup-hpa-epp-sa-dockercfg-fmgzs\""
Jul 28 11:37:28.509911 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:37:28.509887    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40\"/\"kube-root-ca.crt\""
Jul 28 11:37:28.509911 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:37:28.509904    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40\"/\"default-dockercfg-n45mf\""
Jul 28 11:37:28.510087 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:37:28.509935    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40\"/\"autoscale-cleanup-hpa-kserve-self-signed-certs\""
Jul 28 11:37:28.517851 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:37:28.517818    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-router-scheduler-6694c4458b-vmx75"]
Jul 28 11:37:28.670245 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:37:28.670141    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8107ec9e-840d-40d3-8355-aede968b720c-tls-certs\") pod \"autoscale-cleanup-hpa-kserve-router-scheduler-6694c4458b-vmx75\" (UID: \"8107ec9e-840d-40d3-8355-aede968b720c\") " pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-router-scheduler-6694c4458b-vmx75"
Jul 28 11:37:28.670245 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:37:28.670234    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xd8h5\" (UniqueName: \"kubernetes.io/projected/8107ec9e-840d-40d3-8355-aede968b720c-kube-api-access-xd8h5\") pod \"autoscale-cleanup-hpa-kserve-router-scheduler-6694c4458b-vmx75\" (UID: \"8107ec9e-840d-40d3-8355-aede968b720c\") " pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-router-scheduler-6694c4458b-vmx75"
Jul 28 11:37:28.771090 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:37:28.771037    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8107ec9e-840d-40d3-8355-aede968b720c-tls-certs\") pod \"autoscale-cleanup-hpa-kserve-router-scheduler-6694c4458b-vmx75\" (UID: \"8107ec9e-840d-40d3-8355-aede968b720c\") " pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-router-scheduler-6694c4458b-vmx75"
Jul 28 11:37:28.771624 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:37:28.771157    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xd8h5\" (UniqueName: \"kubernetes.io/projected/8107ec9e-840d-40d3-8355-aede968b720c-kube-api-access-xd8h5\") pod \"autoscale-cleanup-hpa-kserve-router-scheduler-6694c4458b-vmx75\" (UID: \"8107ec9e-840d-40d3-8355-aede968b720c\") " pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-router-scheduler-6694c4458b-vmx75"
Jul 28 11:37:28.774494 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:37:28.774461    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8107ec9e-840d-40d3-8355-aede968b720c-tls-certs\") pod \"autoscale-cleanup-hpa-kserve-router-scheduler-6694c4458b-vmx75\" (UID: \"8107ec9e-840d-40d3-8355-aede968b720c\") " pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-router-scheduler-6694c4458b-vmx75"
Jul 28 11:37:28.780374 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:37:28.780344    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xd8h5\" (UniqueName: \"kubernetes.io/projected/8107ec9e-840d-40d3-8355-aede968b720c-kube-api-access-xd8h5\") pod \"autoscale-cleanup-hpa-kserve-router-scheduler-6694c4458b-vmx75\" (UID: \"8107ec9e-840d-40d3-8355-aede968b720c\") " pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-router-scheduler-6694c4458b-vmx75"
Jul 28 11:37:28.821061 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:37:28.821022    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-router-scheduler-6694c4458b-vmx75"
Jul 28 11:37:28.968400 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:37:28.968369    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-router-scheduler-6694c4458b-vmx75"]
Jul 28 11:37:28.970002 ip-10-0-141-50 kubenswrapper[2579]: W0728 11:37:28.969972    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8107ec9e_840d_40d3_8355_aede968b720c.slice/crio-ae35a44aee806629316a92e69eeaf0b36b66e45e0aea23435894f6ef250ca839 WatchSource:0}: Error finding container ae35a44aee806629316a92e69eeaf0b36b66e45e0aea23435894f6ef250ca839: Status 404 returned error can't find the container with id ae35a44aee806629316a92e69eeaf0b36b66e45e0aea23435894f6ef250ca839
Jul 28 11:37:29.464181 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:37:29.464140    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-router-scheduler-6694c4458b-vmx75" event={"ID":"8107ec9e-840d-40d3-8355-aede968b720c","Type":"ContainerStarted","Data":"e2b1fec452f4626e9c18310c1fec5ef9d80d40e3f37d38e293be881d1b91eed9"}
Jul 28 11:37:29.464368 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:37:29.464201    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-router-scheduler-6694c4458b-vmx75" event={"ID":"8107ec9e-840d-40d3-8355-aede968b720c","Type":"ContainerStarted","Data":"ae35a44aee806629316a92e69eeaf0b36b66e45e0aea23435894f6ef250ca839"}
Jul 28 11:37:29.464368 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:37:29.464277    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-router-scheduler-6694c4458b-vmx75"
Jul 28 11:37:29.485837 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:37:29.485528    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-router-scheduler-6694c4458b-vmx75" podStartSLOduration=1.485506478 podStartE2EDuration="1.485506478s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 11:37:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 11:37:29.48502343 +0000 UTC m=+5383.748419760" watchObservedRunningTime="2026-07-28 11:37:29.485506478 +0000 UTC m=+5383.748902811"
Jul 28 11:37:29.673244 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:37:29.673203    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:37:37.734697 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:37:37.734598    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5d95897cf6-dlh6m" podUID="63310b78-d9ee-4761-89cd-6abc7f42b296" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/health\": dial tcp 10.134.0.73:8000: connect: connection refused"
Jul 28 11:37:47.734895 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:37:47.734835    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5d95897cf6-dlh6m" podUID="63310b78-d9ee-4761-89cd-6abc7f42b296" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/health\": dial tcp 10.134.0.73:8000: connect: connection refused"
Jul 28 11:37:57.734953 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:37:57.734900    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5d95897cf6-dlh6m" podUID="63310b78-d9ee-4761-89cd-6abc7f42b296" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/health\": dial tcp 10.134.0.73:8000: connect: connection refused"
Jul 28 11:38:00.471659 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:38:00.471626    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-router-scheduler-6694c4458b-vmx75"
Jul 28 11:38:03.010906 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:38:03.010861    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:38:07.315409 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:38:07.315367    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:38:07.415717 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:38:07.415678    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:38:07.735247 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:38:07.735131    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5d95897cf6-dlh6m" podUID="63310b78-d9ee-4761-89cd-6abc7f42b296" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/health\": dial tcp 10.134.0.73:8000: connect: connection refused"
Jul 28 11:38:17.734734 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:38:17.734688    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5d95897cf6-dlh6m" podUID="63310b78-d9ee-4761-89cd-6abc7f42b296" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/health\": dial tcp 10.134.0.73:8000: connect: connection refused"
Jul 28 11:38:27.747473 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:38:27.747436    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5d95897cf6-dlh6m"
Jul 28 11:38:27.755653 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:38:27.755620    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5d95897cf6-dlh6m"
Jul 28 11:38:34.136601 ip-10-0-141-50 kubenswrapper[2579]: E0728 11:38:34.136550    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bgw9ln" podUID="9f2f8d3c-6c27-4e8b-92c4-d22da1bb1cbe"
Jul 28 11:38:34.825775 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:38:34.825741    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bgw9ln"
Jul 28 11:39:22.215163 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:39:22.215129    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-bbeb2681_lora-multiple-adapters-test-kserve-5d95897cf6-dlh6m_63310b78-d9ee-4761-89cd-6abc7f42b296/storage-initializer/0.log"
Jul 28 11:39:22.226858 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:39:22.226806    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-bbeb2681_lora-multiple-adapters-test-kserve-5d95897cf6-dlh6m_63310b78-d9ee-4761-89cd-6abc7f42b296/main/0.log"
Jul 28 11:40:03.852879 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:40:03.852842    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5d95897cf6-dlh6m"]
Jul 28 11:40:03.889335 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:40:03.889298    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:40:18.875601 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:40:18.875554    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5d95897cf6-dlh6m" podUID="63310b78-d9ee-4761-89cd-6abc7f42b296" containerName="main" containerID="cri-o://c3b7b7ab897332f5372eba2dbcc24cb77ac336bea970d46bb1c3e6db323c4e6e" gracePeriod=45
Jul 28 11:40:19.416651 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:40:19.416610    2579 generic.go:356] "Generic (PLEG): container finished" podID="63310b78-d9ee-4761-89cd-6abc7f42b296" containerID="c3b7b7ab897332f5372eba2dbcc24cb77ac336bea970d46bb1c3e6db323c4e6e" exitCode=0
Jul 28 11:40:19.416845 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:40:19.416671    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5d95897cf6-dlh6m" event={"ID":"63310b78-d9ee-4761-89cd-6abc7f42b296","Type":"ContainerDied","Data":"c3b7b7ab897332f5372eba2dbcc24cb77ac336bea970d46bb1c3e6db323c4e6e"}
Jul 28 11:40:19.540533 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:40:19.540501    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5d95897cf6-dlh6m"
Jul 28 11:40:19.645651 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:40:19.645514    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63310b78-d9ee-4761-89cd-6abc7f42b296-tls-certs\") pod \"63310b78-d9ee-4761-89cd-6abc7f42b296\" (UID: \"63310b78-d9ee-4761-89cd-6abc7f42b296\") "
Jul 28 11:40:19.645651 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:40:19.645623    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-82hzk\" (UniqueName: \"kubernetes.io/projected/63310b78-d9ee-4761-89cd-6abc7f42b296-kube-api-access-82hzk\") pod \"63310b78-d9ee-4761-89cd-6abc7f42b296\" (UID: \"63310b78-d9ee-4761-89cd-6abc7f42b296\") "
Jul 28 11:40:19.645651 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:40:19.645649    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/63310b78-d9ee-4761-89cd-6abc7f42b296-kserve-provision-location\") pod \"63310b78-d9ee-4761-89cd-6abc7f42b296\" (UID: \"63310b78-d9ee-4761-89cd-6abc7f42b296\") "
Jul 28 11:40:19.645964 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:40:19.645679    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63310b78-d9ee-4761-89cd-6abc7f42b296-tmp-dir\") pod \"63310b78-d9ee-4761-89cd-6abc7f42b296\" (UID: \"63310b78-d9ee-4761-89cd-6abc7f42b296\") "
Jul 28 11:40:19.645964 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:40:19.645711    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/63310b78-d9ee-4761-89cd-6abc7f42b296-home\") pod \"63310b78-d9ee-4761-89cd-6abc7f42b296\" (UID: \"63310b78-d9ee-4761-89cd-6abc7f42b296\") "
Jul 28 11:40:19.645964 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:40:19.645748    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/63310b78-d9ee-4761-89cd-6abc7f42b296-dshm\") pod \"63310b78-d9ee-4761-89cd-6abc7f42b296\" (UID: \"63310b78-d9ee-4761-89cd-6abc7f42b296\") "
Jul 28 11:40:19.645964 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:40:19.645770    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63310b78-d9ee-4761-89cd-6abc7f42b296-model-cache\") pod \"63310b78-d9ee-4761-89cd-6abc7f42b296\" (UID: \"63310b78-d9ee-4761-89cd-6abc7f42b296\") "
Jul 28 11:40:19.645964 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:40:19.645826    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/63310b78-d9ee-4761-89cd-6abc7f42b296-cabundle-cert\") pod \"63310b78-d9ee-4761-89cd-6abc7f42b296\" (UID: \"63310b78-d9ee-4761-89cd-6abc7f42b296\") "
Jul 28 11:40:19.646508 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:40:19.646478    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63310b78-d9ee-4761-89cd-6abc7f42b296-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "63310b78-d9ee-4761-89cd-6abc7f42b296" (UID: "63310b78-d9ee-4761-89cd-6abc7f42b296"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 11:40:19.646729 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:40:19.646698    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63310b78-d9ee-4761-89cd-6abc7f42b296-home" (OuterVolumeSpecName: "home") pod "63310b78-d9ee-4761-89cd-6abc7f42b296" (UID: "63310b78-d9ee-4761-89cd-6abc7f42b296"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 11:40:19.647177 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:40:19.647079    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63310b78-d9ee-4761-89cd-6abc7f42b296-model-cache" (OuterVolumeSpecName: "model-cache") pod "63310b78-d9ee-4761-89cd-6abc7f42b296" (UID: "63310b78-d9ee-4761-89cd-6abc7f42b296"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 11:40:19.648369 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:40:19.648319    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63310b78-d9ee-4761-89cd-6abc7f42b296-kube-api-access-82hzk" (OuterVolumeSpecName: "kube-api-access-82hzk") pod "63310b78-d9ee-4761-89cd-6abc7f42b296" (UID: "63310b78-d9ee-4761-89cd-6abc7f42b296"). InnerVolumeSpecName "kube-api-access-82hzk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 11:40:19.648475 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:40:19.648406    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63310b78-d9ee-4761-89cd-6abc7f42b296-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "63310b78-d9ee-4761-89cd-6abc7f42b296" (UID: "63310b78-d9ee-4761-89cd-6abc7f42b296"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 11:40:19.648809 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:40:19.648782    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63310b78-d9ee-4761-89cd-6abc7f42b296-dshm" (OuterVolumeSpecName: "dshm") pod "63310b78-d9ee-4761-89cd-6abc7f42b296" (UID: "63310b78-d9ee-4761-89cd-6abc7f42b296"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 11:40:19.660530 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:40:19.660498    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63310b78-d9ee-4761-89cd-6abc7f42b296-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "63310b78-d9ee-4761-89cd-6abc7f42b296" (UID: "63310b78-d9ee-4761-89cd-6abc7f42b296"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 11:40:19.716076 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:40:19.716025    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63310b78-d9ee-4761-89cd-6abc7f42b296-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "63310b78-d9ee-4761-89cd-6abc7f42b296" (UID: "63310b78-d9ee-4761-89cd-6abc7f42b296"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 11:40:19.747421 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:40:19.747388    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63310b78-d9ee-4761-89cd-6abc7f42b296-tls-certs\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 11:40:19.747421 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:40:19.747415    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82hzk\" (UniqueName: \"kubernetes.io/projected/63310b78-d9ee-4761-89cd-6abc7f42b296-kube-api-access-82hzk\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 11:40:19.747421 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:40:19.747427    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/63310b78-d9ee-4761-89cd-6abc7f42b296-kserve-provision-location\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 11:40:19.747699 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:40:19.747437    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63310b78-d9ee-4761-89cd-6abc7f42b296-tmp-dir\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 11:40:19.747699 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:40:19.747447    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/63310b78-d9ee-4761-89cd-6abc7f42b296-home\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 11:40:19.747699 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:40:19.747455    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/63310b78-d9ee-4761-89cd-6abc7f42b296-dshm\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 11:40:19.747699 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:40:19.747462    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63310b78-d9ee-4761-89cd-6abc7f42b296-model-cache\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 11:40:19.747699 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:40:19.747469    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/63310b78-d9ee-4761-89cd-6abc7f42b296-cabundle-cert\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 11:40:20.423070 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:40:20.423028    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5d95897cf6-dlh6m" event={"ID":"63310b78-d9ee-4761-89cd-6abc7f42b296","Type":"ContainerDied","Data":"82f2e870e35c6c7d06947c7631aaaae9af65f2cf1cd368843fc0295e507ffb29"}
Jul 28 11:40:20.423529 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:40:20.423100    2579 scope.go:117] "RemoveContainer" containerID="c3b7b7ab897332f5372eba2dbcc24cb77ac336bea970d46bb1c3e6db323c4e6e"
Jul 28 11:40:20.423529 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:40:20.423043    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5d95897cf6-dlh6m"
Jul 28 11:40:20.434122 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:40:20.434098    2579 scope.go:117] "RemoveContainer" containerID="1312d606ec784286f8fa92d0696fd4220ad29323f1c1d4739bc9006fdfc76828"
Jul 28 11:40:20.444512 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:40:20.444486    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5d95897cf6-dlh6m"]
Jul 28 11:40:20.447822 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:40:20.447795    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5d95897cf6-dlh6m"]
Jul 28 11:40:22.276263 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:40:22.276227    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="63310b78-d9ee-4761-89cd-6abc7f42b296" path="/var/lib/kubelet/pods/63310b78-d9ee-4761-89cd-6abc7f42b296/volumes"
Jul 28 11:40:24.422647 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:40:24.422607    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:40:24.520806 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:40:24.520768    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:40:24.623050 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:40:24.623016    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:40:30.731251 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:40:30.731211    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:40:30.831602 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:40:30.831564    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:40:37.827001 ip-10-0-141-50 kubenswrapper[2579]: E0728 11:40:37.826904    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bgw9ln" podUID="9f2f8d3c-6c27-4e8b-92c4-d22da1bb1cbe"
Jul 28 11:40:38.513104 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:40:38.513067    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bgw9ln"
Jul 28 11:40:49.081129 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:40:49.081090    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:40:56.707784 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:40:56.707754    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5499b44b6c-pd2jt"]
Jul 28 11:40:56.708235 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:40:56.708069    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63310b78-d9ee-4761-89cd-6abc7f42b296" containerName="main"
Jul 28 11:40:56.708235 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:40:56.708080    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="63310b78-d9ee-4761-89cd-6abc7f42b296" containerName="main"
Jul 28 11:40:56.708235 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:40:56.708097    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63310b78-d9ee-4761-89cd-6abc7f42b296" containerName="storage-initializer"
Jul 28 11:40:56.708235 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:40:56.708103    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="63310b78-d9ee-4761-89cd-6abc7f42b296" containerName="storage-initializer"
Jul 28 11:40:56.708235 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:40:56.708163    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="63310b78-d9ee-4761-89cd-6abc7f42b296" containerName="main"
Jul 28 11:40:56.712421 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:40:56.712406    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5499b44b6c-pd2jt"
Jul 28 11:40:56.714787 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:40:56.714759    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"default-dockercfg-9h6sv\""
Jul 28 11:40:56.714787 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:40:56.714761    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"kube-root-ca.crt\""
Jul 28 11:40:56.714993 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:40:56.714813    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"tls-verification-test-kserve-self-signed-certs\""
Jul 28 11:40:56.714993 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:40:56.714926    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"seaweedfs-s3-creds\""
Jul 28 11:40:56.716381 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:40:56.716360    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"openshift-service-ca.crt\""
Jul 28 11:40:56.723275 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:40:56.723253    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5499b44b6c-pd2jt"]
Jul 28 11:40:56.733878 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:40:56.733852    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:40:56.773392 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:40:56.773361    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/210b298c-9d4d-4480-8e7f-97ed2f4d59de-tmp-dir\") pod \"tls-verification-test-kserve-5499b44b6c-pd2jt\" (UID: \"210b298c-9d4d-4480-8e7f-97ed2f4d59de\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5499b44b6c-pd2jt"
Jul 28 11:40:56.773584 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:40:56.773414    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/210b298c-9d4d-4480-8e7f-97ed2f4d59de-model-cache\") pod \"tls-verification-test-kserve-5499b44b6c-pd2jt\" (UID: \"210b298c-9d4d-4480-8e7f-97ed2f4d59de\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5499b44b6c-pd2jt"
Jul 28 11:40:56.773584 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:40:56.773495    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/210b298c-9d4d-4480-8e7f-97ed2f4d59de-dshm\") pod \"tls-verification-test-kserve-5499b44b6c-pd2jt\" (UID: \"210b298c-9d4d-4480-8e7f-97ed2f4d59de\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5499b44b6c-pd2jt"
Jul 28 11:40:56.773584 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:40:56.773570    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/210b298c-9d4d-4480-8e7f-97ed2f4d59de-home\") pod \"tls-verification-test-kserve-5499b44b6c-pd2jt\" (UID: \"210b298c-9d4d-4480-8e7f-97ed2f4d59de\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5499b44b6c-pd2jt"
Jul 28 11:40:56.773709 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:40:56.773603    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njjlq\" (UniqueName: \"kubernetes.io/projected/210b298c-9d4d-4480-8e7f-97ed2f4d59de-kube-api-access-njjlq\") pod \"tls-verification-test-kserve-5499b44b6c-pd2jt\" (UID: \"210b298c-9d4d-4480-8e7f-97ed2f4d59de\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5499b44b6c-pd2jt"
Jul 28 11:40:56.773709 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:40:56.773630    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/210b298c-9d4d-4480-8e7f-97ed2f4d59de-tls-certs\") pod \"tls-verification-test-kserve-5499b44b6c-pd2jt\" (UID: \"210b298c-9d4d-4480-8e7f-97ed2f4d59de\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5499b44b6c-pd2jt"
Jul 28 11:40:56.773709 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:40:56.773647    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/210b298c-9d4d-4480-8e7f-97ed2f4d59de-kserve-provision-location\") pod \"tls-verification-test-kserve-5499b44b6c-pd2jt\" (UID: \"210b298c-9d4d-4480-8e7f-97ed2f4d59de\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5499b44b6c-pd2jt"
Jul 28 11:40:56.874911 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:40:56.874873    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njjlq\" (UniqueName: \"kubernetes.io/projected/210b298c-9d4d-4480-8e7f-97ed2f4d59de-kube-api-access-njjlq\") pod \"tls-verification-test-kserve-5499b44b6c-pd2jt\" (UID: \"210b298c-9d4d-4480-8e7f-97ed2f4d59de\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5499b44b6c-pd2jt"
Jul 28 11:40:56.875111 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:40:56.874917    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/210b298c-9d4d-4480-8e7f-97ed2f4d59de-tls-certs\") pod \"tls-verification-test-kserve-5499b44b6c-pd2jt\" (UID: \"210b298c-9d4d-4480-8e7f-97ed2f4d59de\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5499b44b6c-pd2jt"
Jul 28 11:40:56.875111 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:40:56.874935    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/210b298c-9d4d-4480-8e7f-97ed2f4d59de-kserve-provision-location\") pod \"tls-verification-test-kserve-5499b44b6c-pd2jt\" (UID: \"210b298c-9d4d-4480-8e7f-97ed2f4d59de\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5499b44b6c-pd2jt"
Jul 28 11:40:56.875111 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:40:56.874990    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/210b298c-9d4d-4480-8e7f-97ed2f4d59de-tmp-dir\") pod \"tls-verification-test-kserve-5499b44b6c-pd2jt\" (UID: \"210b298c-9d4d-4480-8e7f-97ed2f4d59de\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5499b44b6c-pd2jt"
Jul 28 11:40:56.875111 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:40:56.875023    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/210b298c-9d4d-4480-8e7f-97ed2f4d59de-model-cache\") pod \"tls-verification-test-kserve-5499b44b6c-pd2jt\" (UID: \"210b298c-9d4d-4480-8e7f-97ed2f4d59de\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5499b44b6c-pd2jt"
Jul 28 11:40:56.875111 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:40:56.875065    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/210b298c-9d4d-4480-8e7f-97ed2f4d59de-dshm\") pod \"tls-verification-test-kserve-5499b44b6c-pd2jt\" (UID: \"210b298c-9d4d-4480-8e7f-97ed2f4d59de\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5499b44b6c-pd2jt"
Jul 28 11:40:56.875111 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:40:56.875082    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/210b298c-9d4d-4480-8e7f-97ed2f4d59de-home\") pod \"tls-verification-test-kserve-5499b44b6c-pd2jt\" (UID: \"210b298c-9d4d-4480-8e7f-97ed2f4d59de\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5499b44b6c-pd2jt"
Jul 28 11:40:56.875433 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:40:56.875360    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/210b298c-9d4d-4480-8e7f-97ed2f4d59de-kserve-provision-location\") pod \"tls-verification-test-kserve-5499b44b6c-pd2jt\" (UID: \"210b298c-9d4d-4480-8e7f-97ed2f4d59de\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5499b44b6c-pd2jt"
Jul 28 11:40:56.875433 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:40:56.875382    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/210b298c-9d4d-4480-8e7f-97ed2f4d59de-tmp-dir\") pod \"tls-verification-test-kserve-5499b44b6c-pd2jt\" (UID: \"210b298c-9d4d-4480-8e7f-97ed2f4d59de\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5499b44b6c-pd2jt"
Jul 28 11:40:56.875563 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:40:56.875489    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/210b298c-9d4d-4480-8e7f-97ed2f4d59de-model-cache\") pod \"tls-verification-test-kserve-5499b44b6c-pd2jt\" (UID: \"210b298c-9d4d-4480-8e7f-97ed2f4d59de\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5499b44b6c-pd2jt"
Jul 28 11:40:56.875563 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:40:56.875504    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/210b298c-9d4d-4480-8e7f-97ed2f4d59de-home\") pod \"tls-verification-test-kserve-5499b44b6c-pd2jt\" (UID: \"210b298c-9d4d-4480-8e7f-97ed2f4d59de\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5499b44b6c-pd2jt"
Jul 28 11:40:56.877384 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:40:56.877362    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/210b298c-9d4d-4480-8e7f-97ed2f4d59de-dshm\") pod \"tls-verification-test-kserve-5499b44b6c-pd2jt\" (UID: \"210b298c-9d4d-4480-8e7f-97ed2f4d59de\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5499b44b6c-pd2jt"
Jul 28 11:40:56.877548 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:40:56.877518    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/210b298c-9d4d-4480-8e7f-97ed2f4d59de-tls-certs\") pod \"tls-verification-test-kserve-5499b44b6c-pd2jt\" (UID: \"210b298c-9d4d-4480-8e7f-97ed2f4d59de\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5499b44b6c-pd2jt"
Jul 28 11:40:56.884068 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:40:56.884047    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-njjlq\" (UniqueName: \"kubernetes.io/projected/210b298c-9d4d-4480-8e7f-97ed2f4d59de-kube-api-access-njjlq\") pod \"tls-verification-test-kserve-5499b44b6c-pd2jt\" (UID: \"210b298c-9d4d-4480-8e7f-97ed2f4d59de\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5499b44b6c-pd2jt"
Jul 28 11:40:56.989060 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:40:56.988947    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:40:57.023890 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:40:57.023853    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5499b44b6c-pd2jt"
Jul 28 11:40:57.170448 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:40:57.170420    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5499b44b6c-pd2jt"]
Jul 28 11:40:57.172388 ip-10-0-141-50 kubenswrapper[2579]: W0728 11:40:57.172359    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod210b298c_9d4d_4480_8e7f_97ed2f4d59de.slice/crio-2680748f59d942d65ef35d8961d0b4f78c6dea40c761c3f6c610c1d1e0239f00 WatchSource:0}: Error finding container 2680748f59d942d65ef35d8961d0b4f78c6dea40c761c3f6c610c1d1e0239f00: Status 404 returned error can't find the container with id 2680748f59d942d65ef35d8961d0b4f78c6dea40c761c3f6c610c1d1e0239f00
Jul 28 11:40:57.611511 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:40:57.611468    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5499b44b6c-pd2jt" event={"ID":"210b298c-9d4d-4480-8e7f-97ed2f4d59de","Type":"ContainerStarted","Data":"085fafc96277bc738f0ab66e9e0b9d7e82a9fa33e18baa1595fe4d364a275804"}
Jul 28 11:40:57.611740 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:40:57.611575    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5499b44b6c-pd2jt" event={"ID":"210b298c-9d4d-4480-8e7f-97ed2f4d59de","Type":"ContainerStarted","Data":"2680748f59d942d65ef35d8961d0b4f78c6dea40c761c3f6c610c1d1e0239f00"}
Jul 28 11:40:58.522416 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:40:58.522379    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:41:02.640836 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:41:02.640793    2579 generic.go:356] "Generic (PLEG): container finished" podID="210b298c-9d4d-4480-8e7f-97ed2f4d59de" containerID="085fafc96277bc738f0ab66e9e0b9d7e82a9fa33e18baa1595fe4d364a275804" exitCode=0
Jul 28 11:41:02.641354 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:41:02.640876    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5499b44b6c-pd2jt" event={"ID":"210b298c-9d4d-4480-8e7f-97ed2f4d59de","Type":"ContainerDied","Data":"085fafc96277bc738f0ab66e9e0b9d7e82a9fa33e18baa1595fe4d364a275804"}
Jul 28 11:41:03.648085 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:41:03.648040    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5499b44b6c-pd2jt" event={"ID":"210b298c-9d4d-4480-8e7f-97ed2f4d59de","Type":"ContainerStarted","Data":"9ba28ba0d68f20bcb359ad2fe4307d07a02d0d4f52252cba410672db2a0f6ae6"}
Jul 28 11:41:03.674939 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:41:03.674880    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5499b44b6c-pd2jt" podStartSLOduration=2.848104316 podStartE2EDuration="7.674861091s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.826756775s" isStatefulPod=false podCreationTimestamp="2026-07-28 11:40:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 11:41:03.67250011 +0000 UTC m=+5597.935896439" watchObservedRunningTime="2026-07-28 11:41:03.674861091 +0000 UTC m=+5597.938257467"
Jul 28 11:41:04.024246 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:41:04.024156    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5499b44b6c-pd2jt"
Jul 28 11:41:07.024045 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:41:07.023983    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5499b44b6c-pd2jt"
Jul 28 11:41:07.025650 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:41:07.025616    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5499b44b6c-pd2jt" podUID="210b298c-9d4d-4480-8e7f-97ed2f4d59de" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": dial tcp 10.134.0.75:8000: connect: connection refused"
Jul 28 11:41:17.024322 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:41:17.024271    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5499b44b6c-pd2jt" podUID="210b298c-9d4d-4480-8e7f-97ed2f4d59de" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": dial tcp 10.134.0.75:8000: connect: connection refused"
Jul 28 11:41:27.025334 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:41:27.025289    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5499b44b6c-pd2jt" podUID="210b298c-9d4d-4480-8e7f-97ed2f4d59de" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": dial tcp 10.134.0.75:8000: connect: connection refused"
Jul 28 11:41:28.297020 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:41:28.296975    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:41:37.024646 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:41:37.024599    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5499b44b6c-pd2jt" podUID="210b298c-9d4d-4480-8e7f-97ed2f4d59de" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": dial tcp 10.134.0.75:8000: connect: connection refused"
Jul 28 11:41:47.025365 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:41:47.025313    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5499b44b6c-pd2jt" podUID="210b298c-9d4d-4480-8e7f-97ed2f4d59de" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": dial tcp 10.134.0.75:8000: connect: connection refused"
Jul 28 11:41:57.024303 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:41:57.024255    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5499b44b6c-pd2jt" podUID="210b298c-9d4d-4480-8e7f-97ed2f4d59de" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": dial tcp 10.134.0.75:8000: connect: connection refused"
Jul 28 11:42:07.024896 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:42:07.024847    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5499b44b6c-pd2jt" podUID="210b298c-9d4d-4480-8e7f-97ed2f4d59de" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": dial tcp 10.134.0.75:8000: connect: connection refused"
Jul 28 11:42:17.025032 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:42:17.024972    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5499b44b6c-pd2jt" podUID="210b298c-9d4d-4480-8e7f-97ed2f4d59de" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": dial tcp 10.134.0.75:8000: connect: connection refused"
Jul 28 11:42:27.024316 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:42:27.024271    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5499b44b6c-pd2jt" podUID="210b298c-9d4d-4480-8e7f-97ed2f4d59de" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": dial tcp 10.134.0.75:8000: connect: connection refused"
Jul 28 11:42:31.495937 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:42:31.495899    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:42:37.025174 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:42:37.025124    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5499b44b6c-pd2jt" podUID="210b298c-9d4d-4480-8e7f-97ed2f4d59de" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": dial tcp 10.134.0.75:8000: connect: connection refused"
Jul 28 11:42:41.513870 ip-10-0-141-50 kubenswrapper[2579]: E0728 11:42:41.513821    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bgw9ln" podUID="9f2f8d3c-6c27-4e8b-92c4-d22da1bb1cbe"
Jul 28 11:42:42.176049 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:42:42.176017    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bgw9ln"
Jul 28 11:42:47.025144 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:42:47.025098    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5499b44b6c-pd2jt" podUID="210b298c-9d4d-4480-8e7f-97ed2f4d59de" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": dial tcp 10.134.0.75:8000: connect: connection refused"
Jul 28 11:42:57.024223 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:42:57.024178    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5499b44b6c-pd2jt" podUID="210b298c-9d4d-4480-8e7f-97ed2f4d59de" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": dial tcp 10.134.0.75:8000: connect: connection refused"
Jul 28 11:43:07.034340 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:43:07.034300    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5499b44b6c-pd2jt"
Jul 28 11:43:07.041945 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:43:07.041919    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5499b44b6c-pd2jt"
Jul 28 11:43:28.160996 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:43:28.160961    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:44:07.501901 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:44:07.501818    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5499b44b6c-pd2jt"]
Jul 28 11:44:07.514439 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:44:07.514402    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:44:07.571332 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:44:07.571288    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:44:22.524945 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:44:22.524901    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5499b44b6c-pd2jt" podUID="210b298c-9d4d-4480-8e7f-97ed2f4d59de" containerName="main" containerID="cri-o://9ba28ba0d68f20bcb359ad2fe4307d07a02d0d4f52252cba410672db2a0f6ae6" gracePeriod=45
Jul 28 11:44:23.035898 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:44:23.035857    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5499b44b6c-pd2jt" podUID="210b298c-9d4d-4480-8e7f-97ed2f4d59de" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": dial tcp 10.134.0.75:8000: connect: connection refused"
Jul 28 11:44:23.293727 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:44:23.293661    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5499b44b6c-pd2jt"
Jul 28 11:44:23.323087 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:44:23.323048    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/210b298c-9d4d-4480-8e7f-97ed2f4d59de-tmp-dir\") pod \"210b298c-9d4d-4480-8e7f-97ed2f4d59de\" (UID: \"210b298c-9d4d-4480-8e7f-97ed2f4d59de\") "
Jul 28 11:44:23.323361 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:44:23.323335    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-njjlq\" (UniqueName: \"kubernetes.io/projected/210b298c-9d4d-4480-8e7f-97ed2f4d59de-kube-api-access-njjlq\") pod \"210b298c-9d4d-4480-8e7f-97ed2f4d59de\" (UID: \"210b298c-9d4d-4480-8e7f-97ed2f4d59de\") "
Jul 28 11:44:23.323527 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:44:23.323510    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/210b298c-9d4d-4480-8e7f-97ed2f4d59de-tls-certs\") pod \"210b298c-9d4d-4480-8e7f-97ed2f4d59de\" (UID: \"210b298c-9d4d-4480-8e7f-97ed2f4d59de\") "
Jul 28 11:44:23.323687 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:44:23.323671    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/210b298c-9d4d-4480-8e7f-97ed2f4d59de-model-cache\") pod \"210b298c-9d4d-4480-8e7f-97ed2f4d59de\" (UID: \"210b298c-9d4d-4480-8e7f-97ed2f4d59de\") "
Jul 28 11:44:23.323881 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:44:23.323864    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/210b298c-9d4d-4480-8e7f-97ed2f4d59de-kserve-provision-location\") pod \"210b298c-9d4d-4480-8e7f-97ed2f4d59de\" (UID: \"210b298c-9d4d-4480-8e7f-97ed2f4d59de\") "
Jul 28 11:44:23.324017 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:44:23.324003    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/210b298c-9d4d-4480-8e7f-97ed2f4d59de-home\") pod \"210b298c-9d4d-4480-8e7f-97ed2f4d59de\" (UID: \"210b298c-9d4d-4480-8e7f-97ed2f4d59de\") "
Jul 28 11:44:23.324135 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:44:23.324121    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/210b298c-9d4d-4480-8e7f-97ed2f4d59de-dshm\") pod \"210b298c-9d4d-4480-8e7f-97ed2f4d59de\" (UID: \"210b298c-9d4d-4480-8e7f-97ed2f4d59de\") "
Jul 28 11:44:23.326808 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:44:23.326365    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/210b298c-9d4d-4480-8e7f-97ed2f4d59de-model-cache" (OuterVolumeSpecName: "model-cache") pod "210b298c-9d4d-4480-8e7f-97ed2f4d59de" (UID: "210b298c-9d4d-4480-8e7f-97ed2f4d59de"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 11:44:23.327450 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:44:23.327411    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/210b298c-9d4d-4480-8e7f-97ed2f4d59de-dshm" (OuterVolumeSpecName: "dshm") pod "210b298c-9d4d-4480-8e7f-97ed2f4d59de" (UID: "210b298c-9d4d-4480-8e7f-97ed2f4d59de"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 11:44:23.327945 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:44:23.327918    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/210b298c-9d4d-4480-8e7f-97ed2f4d59de-home" (OuterVolumeSpecName: "home") pod "210b298c-9d4d-4480-8e7f-97ed2f4d59de" (UID: "210b298c-9d4d-4480-8e7f-97ed2f4d59de"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 11:44:23.328055 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:44:23.327998    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/210b298c-9d4d-4480-8e7f-97ed2f4d59de-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "210b298c-9d4d-4480-8e7f-97ed2f4d59de" (UID: "210b298c-9d4d-4480-8e7f-97ed2f4d59de"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 11:44:23.329102 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:44:23.329074    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/210b298c-9d4d-4480-8e7f-97ed2f4d59de-kube-api-access-njjlq" (OuterVolumeSpecName: "kube-api-access-njjlq") pod "210b298c-9d4d-4480-8e7f-97ed2f4d59de" (UID: "210b298c-9d4d-4480-8e7f-97ed2f4d59de"). InnerVolumeSpecName "kube-api-access-njjlq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 11:44:23.340305 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:44:23.340271    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/210b298c-9d4d-4480-8e7f-97ed2f4d59de-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "210b298c-9d4d-4480-8e7f-97ed2f4d59de" (UID: "210b298c-9d4d-4480-8e7f-97ed2f4d59de"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 11:44:23.398602 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:44:23.398556    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/210b298c-9d4d-4480-8e7f-97ed2f4d59de-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "210b298c-9d4d-4480-8e7f-97ed2f4d59de" (UID: "210b298c-9d4d-4480-8e7f-97ed2f4d59de"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 11:44:23.425879 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:44:23.425851    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/210b298c-9d4d-4480-8e7f-97ed2f4d59de-tls-certs\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 11:44:23.425879 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:44:23.425878    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/210b298c-9d4d-4480-8e7f-97ed2f4d59de-model-cache\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 11:44:23.425879 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:44:23.425887    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/210b298c-9d4d-4480-8e7f-97ed2f4d59de-kserve-provision-location\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 11:44:23.426115 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:44:23.425895    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/210b298c-9d4d-4480-8e7f-97ed2f4d59de-home\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 11:44:23.426115 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:44:23.425904    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/210b298c-9d4d-4480-8e7f-97ed2f4d59de-dshm\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 11:44:23.426115 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:44:23.425911    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/210b298c-9d4d-4480-8e7f-97ed2f4d59de-tmp-dir\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 11:44:23.426115 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:44:23.425919    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njjlq\" (UniqueName: \"kubernetes.io/projected/210b298c-9d4d-4480-8e7f-97ed2f4d59de-kube-api-access-njjlq\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 11:44:23.700859 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:44:23.700759    2579 generic.go:356] "Generic (PLEG): container finished" podID="210b298c-9d4d-4480-8e7f-97ed2f4d59de" containerID="9ba28ba0d68f20bcb359ad2fe4307d07a02d0d4f52252cba410672db2a0f6ae6" exitCode=0
Jul 28 11:44:23.700859 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:44:23.700820    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5499b44b6c-pd2jt"
Jul 28 11:44:23.700859 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:44:23.700840    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5499b44b6c-pd2jt" event={"ID":"210b298c-9d4d-4480-8e7f-97ed2f4d59de","Type":"ContainerDied","Data":"9ba28ba0d68f20bcb359ad2fe4307d07a02d0d4f52252cba410672db2a0f6ae6"}
Jul 28 11:44:23.701394 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:44:23.700921    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5499b44b6c-pd2jt" event={"ID":"210b298c-9d4d-4480-8e7f-97ed2f4d59de","Type":"ContainerDied","Data":"2680748f59d942d65ef35d8961d0b4f78c6dea40c761c3f6c610c1d1e0239f00"}
Jul 28 11:44:23.701394 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:44:23.700947    2579 scope.go:117] "RemoveContainer" containerID="9ba28ba0d68f20bcb359ad2fe4307d07a02d0d4f52252cba410672db2a0f6ae6"
Jul 28 11:44:23.709992 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:44:23.709974    2579 scope.go:117] "RemoveContainer" containerID="085fafc96277bc738f0ab66e9e0b9d7e82a9fa33e18baa1595fe4d364a275804"
Jul 28 11:44:23.718282 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:44:23.718263    2579 scope.go:117] "RemoveContainer" containerID="9ba28ba0d68f20bcb359ad2fe4307d07a02d0d4f52252cba410672db2a0f6ae6"
Jul 28 11:44:23.718551 ip-10-0-141-50 kubenswrapper[2579]: E0728 11:44:23.718515    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ba28ba0d68f20bcb359ad2fe4307d07a02d0d4f52252cba410672db2a0f6ae6\": container with ID starting with 9ba28ba0d68f20bcb359ad2fe4307d07a02d0d4f52252cba410672db2a0f6ae6 not found: ID does not exist" containerID="9ba28ba0d68f20bcb359ad2fe4307d07a02d0d4f52252cba410672db2a0f6ae6"
Jul 28 11:44:23.718616 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:44:23.718567    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ba28ba0d68f20bcb359ad2fe4307d07a02d0d4f52252cba410672db2a0f6ae6"} err="failed to get container status \"9ba28ba0d68f20bcb359ad2fe4307d07a02d0d4f52252cba410672db2a0f6ae6\": rpc error: code = NotFound desc = could not find container \"9ba28ba0d68f20bcb359ad2fe4307d07a02d0d4f52252cba410672db2a0f6ae6\": container with ID starting with 9ba28ba0d68f20bcb359ad2fe4307d07a02d0d4f52252cba410672db2a0f6ae6 not found: ID does not exist"
Jul 28 11:44:23.718616 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:44:23.718586    2579 scope.go:117] "RemoveContainer" containerID="085fafc96277bc738f0ab66e9e0b9d7e82a9fa33e18baa1595fe4d364a275804"
Jul 28 11:44:23.718812 ip-10-0-141-50 kubenswrapper[2579]: E0728 11:44:23.718797    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"085fafc96277bc738f0ab66e9e0b9d7e82a9fa33e18baa1595fe4d364a275804\": container with ID starting with 085fafc96277bc738f0ab66e9e0b9d7e82a9fa33e18baa1595fe4d364a275804 not found: ID does not exist" containerID="085fafc96277bc738f0ab66e9e0b9d7e82a9fa33e18baa1595fe4d364a275804"
Jul 28 11:44:23.718849 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:44:23.718816    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"085fafc96277bc738f0ab66e9e0b9d7e82a9fa33e18baa1595fe4d364a275804"} err="failed to get container status \"085fafc96277bc738f0ab66e9e0b9d7e82a9fa33e18baa1595fe4d364a275804\": rpc error: code = NotFound desc = could not find container \"085fafc96277bc738f0ab66e9e0b9d7e82a9fa33e18baa1595fe4d364a275804\": container with ID starting with 085fafc96277bc738f0ab66e9e0b9d7e82a9fa33e18baa1595fe4d364a275804 not found: ID does not exist"
Jul 28 11:44:23.723308 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:44:23.723284    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5499b44b6c-pd2jt"]
Jul 28 11:44:23.726875 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:44:23.726853    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5499b44b6c-pd2jt"]
Jul 28 11:44:24.275624 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:44:24.275593    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="210b298c-9d4d-4480-8e7f-97ed2f4d59de" path="/var/lib/kubelet/pods/210b298c-9d4d-4480-8e7f-97ed2f4d59de/volumes"
Jul 28 11:44:25.459419 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:44:25.459385    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:44:25.553403 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:44:25.553362    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:44:31.895030 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:44:31.894996    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:44:37.478731 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:44:37.478688    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:44:37.542126 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:44:37.542084    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:44:40.301199 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:44:40.301164    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6b9d46dd5-xfjdj"]
Jul 28 11:44:40.301717 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:44:40.301693    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="210b298c-9d4d-4480-8e7f-97ed2f4d59de" containerName="main"
Jul 28 11:44:40.301795 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:44:40.301721    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="210b298c-9d4d-4480-8e7f-97ed2f4d59de" containerName="main"
Jul 28 11:44:40.301795 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:44:40.301744    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="210b298c-9d4d-4480-8e7f-97ed2f4d59de" containerName="storage-initializer"
Jul 28 11:44:40.301795 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:44:40.301753    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="210b298c-9d4d-4480-8e7f-97ed2f4d59de" containerName="storage-initializer"
Jul 28 11:44:40.301962 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:44:40.301841    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="210b298c-9d4d-4480-8e7f-97ed2f4d59de" containerName="main"
Jul 28 11:44:40.306962 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:44:40.306938    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6b9d46dd5-xfjdj"
Jul 28 11:44:40.309104 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:44:40.309076    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"kube-root-ca.crt\""
Jul 28 11:44:40.309253 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:44:40.309136    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"default-dockercfg-7jch4\""
Jul 28 11:44:40.309253 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:44:40.309137    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"openshift-service-ca.crt\""
Jul 28 11:44:40.309389 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:44:40.309255    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"prestop-hook-test-kserve-self-signed-certs\""
Jul 28 11:44:40.309389 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:44:40.309301    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"seaweedfs-s3-creds\""
Jul 28 11:44:40.315739 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:44:40.315715    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6b9d46dd5-xfjdj"]
Jul 28 11:44:40.331858 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:44:40.331828    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:44:40.375950 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:44:40.375902    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0c7fc8d-5fda-4241-9795-5d6a8967fec4-tls-certs\") pod \"prestop-hook-test-kserve-6b9d46dd5-xfjdj\" (UID: \"b0c7fc8d-5fda-4241-9795-5d6a8967fec4\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6b9d46dd5-xfjdj"
Jul 28 11:44:40.376142 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:44:40.375965    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0c7fc8d-5fda-4241-9795-5d6a8967fec4-model-cache\") pod \"prestop-hook-test-kserve-6b9d46dd5-xfjdj\" (UID: \"b0c7fc8d-5fda-4241-9795-5d6a8967fec4\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6b9d46dd5-xfjdj"
Jul 28 11:44:40.376142 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:44:40.376010    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0c7fc8d-5fda-4241-9795-5d6a8967fec4-dshm\") pod \"prestop-hook-test-kserve-6b9d46dd5-xfjdj\" (UID: \"b0c7fc8d-5fda-4241-9795-5d6a8967fec4\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6b9d46dd5-xfjdj"
Jul 28 11:44:40.376142 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:44:40.376075    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b0c7fc8d-5fda-4241-9795-5d6a8967fec4-kserve-provision-location\") pod \"prestop-hook-test-kserve-6b9d46dd5-xfjdj\" (UID: \"b0c7fc8d-5fda-4241-9795-5d6a8967fec4\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6b9d46dd5-xfjdj"
Jul 28 11:44:40.376142 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:44:40.376107    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxw9f\" (UniqueName: \"kubernetes.io/projected/b0c7fc8d-5fda-4241-9795-5d6a8967fec4-kube-api-access-jxw9f\") pod \"prestop-hook-test-kserve-6b9d46dd5-xfjdj\" (UID: \"b0c7fc8d-5fda-4241-9795-5d6a8967fec4\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6b9d46dd5-xfjdj"
Jul 28 11:44:40.376142 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:44:40.376138    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0c7fc8d-5fda-4241-9795-5d6a8967fec4-tmp-dir\") pod \"prestop-hook-test-kserve-6b9d46dd5-xfjdj\" (UID: \"b0c7fc8d-5fda-4241-9795-5d6a8967fec4\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6b9d46dd5-xfjdj"
Jul 28 11:44:40.376314 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:44:40.376180    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0c7fc8d-5fda-4241-9795-5d6a8967fec4-home\") pod \"prestop-hook-test-kserve-6b9d46dd5-xfjdj\" (UID: \"b0c7fc8d-5fda-4241-9795-5d6a8967fec4\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6b9d46dd5-xfjdj"
Jul 28 11:44:40.476877 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:44:40.476840    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0c7fc8d-5fda-4241-9795-5d6a8967fec4-home\") pod \"prestop-hook-test-kserve-6b9d46dd5-xfjdj\" (UID: \"b0c7fc8d-5fda-4241-9795-5d6a8967fec4\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6b9d46dd5-xfjdj"
Jul 28 11:44:40.477095 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:44:40.476916    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0c7fc8d-5fda-4241-9795-5d6a8967fec4-tls-certs\") pod \"prestop-hook-test-kserve-6b9d46dd5-xfjdj\" (UID: \"b0c7fc8d-5fda-4241-9795-5d6a8967fec4\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6b9d46dd5-xfjdj"
Jul 28 11:44:40.477095 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:44:40.476949    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0c7fc8d-5fda-4241-9795-5d6a8967fec4-model-cache\") pod \"prestop-hook-test-kserve-6b9d46dd5-xfjdj\" (UID: \"b0c7fc8d-5fda-4241-9795-5d6a8967fec4\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6b9d46dd5-xfjdj"
Jul 28 11:44:40.477095 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:44:40.477007    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0c7fc8d-5fda-4241-9795-5d6a8967fec4-dshm\") pod \"prestop-hook-test-kserve-6b9d46dd5-xfjdj\" (UID: \"b0c7fc8d-5fda-4241-9795-5d6a8967fec4\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6b9d46dd5-xfjdj"
Jul 28 11:44:40.477095 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:44:40.477037    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b0c7fc8d-5fda-4241-9795-5d6a8967fec4-kserve-provision-location\") pod \"prestop-hook-test-kserve-6b9d46dd5-xfjdj\" (UID: \"b0c7fc8d-5fda-4241-9795-5d6a8967fec4\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6b9d46dd5-xfjdj"
Jul 28 11:44:40.477095 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:44:40.477058    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxw9f\" (UniqueName: \"kubernetes.io/projected/b0c7fc8d-5fda-4241-9795-5d6a8967fec4-kube-api-access-jxw9f\") pod \"prestop-hook-test-kserve-6b9d46dd5-xfjdj\" (UID: \"b0c7fc8d-5fda-4241-9795-5d6a8967fec4\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6b9d46dd5-xfjdj"
Jul 28 11:44:40.477095 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:44:40.477088    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0c7fc8d-5fda-4241-9795-5d6a8967fec4-tmp-dir\") pod \"prestop-hook-test-kserve-6b9d46dd5-xfjdj\" (UID: \"b0c7fc8d-5fda-4241-9795-5d6a8967fec4\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6b9d46dd5-xfjdj"
Jul 28 11:44:40.477481 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:44:40.477324    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0c7fc8d-5fda-4241-9795-5d6a8967fec4-home\") pod \"prestop-hook-test-kserve-6b9d46dd5-xfjdj\" (UID: \"b0c7fc8d-5fda-4241-9795-5d6a8967fec4\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6b9d46dd5-xfjdj"
Jul 28 11:44:40.477481 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:44:40.477379    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0c7fc8d-5fda-4241-9795-5d6a8967fec4-tmp-dir\") pod \"prestop-hook-test-kserve-6b9d46dd5-xfjdj\" (UID: \"b0c7fc8d-5fda-4241-9795-5d6a8967fec4\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6b9d46dd5-xfjdj"
Jul 28 11:44:40.477982 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:44:40.477956    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0c7fc8d-5fda-4241-9795-5d6a8967fec4-model-cache\") pod \"prestop-hook-test-kserve-6b9d46dd5-xfjdj\" (UID: \"b0c7fc8d-5fda-4241-9795-5d6a8967fec4\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6b9d46dd5-xfjdj"
Jul 28 11:44:40.478106 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:44:40.478025    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b0c7fc8d-5fda-4241-9795-5d6a8967fec4-kserve-provision-location\") pod \"prestop-hook-test-kserve-6b9d46dd5-xfjdj\" (UID: \"b0c7fc8d-5fda-4241-9795-5d6a8967fec4\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6b9d46dd5-xfjdj"
Jul 28 11:44:40.479692 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:44:40.479667    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0c7fc8d-5fda-4241-9795-5d6a8967fec4-dshm\") pod \"prestop-hook-test-kserve-6b9d46dd5-xfjdj\" (UID: \"b0c7fc8d-5fda-4241-9795-5d6a8967fec4\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6b9d46dd5-xfjdj"
Jul 28 11:44:40.480078 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:44:40.480057    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0c7fc8d-5fda-4241-9795-5d6a8967fec4-tls-certs\") pod \"prestop-hook-test-kserve-6b9d46dd5-xfjdj\" (UID: \"b0c7fc8d-5fda-4241-9795-5d6a8967fec4\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6b9d46dd5-xfjdj"
Jul 28 11:44:40.485102 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:44:40.485069    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxw9f\" (UniqueName: \"kubernetes.io/projected/b0c7fc8d-5fda-4241-9795-5d6a8967fec4-kube-api-access-jxw9f\") pod \"prestop-hook-test-kserve-6b9d46dd5-xfjdj\" (UID: \"b0c7fc8d-5fda-4241-9795-5d6a8967fec4\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6b9d46dd5-xfjdj"
Jul 28 11:44:40.618670 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:44:40.618578    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6b9d46dd5-xfjdj"
Jul 28 11:44:40.625090 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:44:40.625058    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:44:40.764904 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:44:40.764869    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6b9d46dd5-xfjdj"]
Jul 28 11:44:40.767966 ip-10-0-141-50 kubenswrapper[2579]: W0728 11:44:40.767939    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0c7fc8d_5fda_4241_9795_5d6a8967fec4.slice/crio-94d41ca7896db2777f034fa8310119f8b2811cd3413b18f98673cabaa3d515b6 WatchSource:0}: Error finding container 94d41ca7896db2777f034fa8310119f8b2811cd3413b18f98673cabaa3d515b6: Status 404 returned error can't find the container with id 94d41ca7896db2777f034fa8310119f8b2811cd3413b18f98673cabaa3d515b6
Jul 28 11:44:40.784519 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:44:40.784492    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6b9d46dd5-xfjdj" event={"ID":"b0c7fc8d-5fda-4241-9795-5d6a8967fec4","Type":"ContainerStarted","Data":"94d41ca7896db2777f034fa8310119f8b2811cd3413b18f98673cabaa3d515b6"}
Jul 28 11:44:41.794755 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:44:41.794711    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6b9d46dd5-xfjdj" event={"ID":"b0c7fc8d-5fda-4241-9795-5d6a8967fec4","Type":"ContainerStarted","Data":"e9efc4f2170e739ac010664066f5fa651f0932ddacca42bcc667c3c12e42992e"}
Jul 28 11:44:45.176930 ip-10-0-141-50 kubenswrapper[2579]: E0728 11:44:45.176884    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bgw9ln" podUID="9f2f8d3c-6c27-4e8b-92c4-d22da1bb1cbe"
Jul 28 11:44:45.817147 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:44:45.817110    2579 generic.go:356] "Generic (PLEG): container finished" podID="b0c7fc8d-5fda-4241-9795-5d6a8967fec4" containerID="e9efc4f2170e739ac010664066f5fa651f0932ddacca42bcc667c3c12e42992e" exitCode=0
Jul 28 11:44:45.817378 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:44:45.817186    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6b9d46dd5-xfjdj" event={"ID":"b0c7fc8d-5fda-4241-9795-5d6a8967fec4","Type":"ContainerDied","Data":"e9efc4f2170e739ac010664066f5fa651f0932ddacca42bcc667c3c12e42992e"}
Jul 28 11:44:45.817378 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:44:45.817214    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bgw9ln"
Jul 28 11:44:46.824286 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:44:46.824250    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6b9d46dd5-xfjdj" event={"ID":"b0c7fc8d-5fda-4241-9795-5d6a8967fec4","Type":"ContainerStarted","Data":"eda6b3f893c891ddd6cbb6c8a694e1d6ada706d259c7f12d3e1ec8256f003659"}
Jul 28 11:44:46.849826 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:44:46.849768    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6b9d46dd5-xfjdj" podStartSLOduration=2.599293732 podStartE2EDuration="6.849749788s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.250456056s" isStatefulPod=false podCreationTimestamp="2026-07-28 11:44:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 11:44:46.84715541 +0000 UTC m=+5821.110551742" watchObservedRunningTime="2026-07-28 11:44:46.849749788 +0000 UTC m=+5821.113146121"
Jul 28 11:44:47.618897 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:44:47.618858    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6b9d46dd5-xfjdj"
Jul 28 11:44:50.618940 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:44:50.618902    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6b9d46dd5-xfjdj"
Jul 28 11:44:50.620591 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:44:50.620557    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6b9d46dd5-xfjdj" podUID="b0c7fc8d-5fda-4241-9795-5d6a8967fec4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/health\": dial tcp 10.134.0.76:8000: connect: connection refused"
Jul 28 11:45:00.619809 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:45:00.619757    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6b9d46dd5-xfjdj" podUID="b0c7fc8d-5fda-4241-9795-5d6a8967fec4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/health\": dial tcp 10.134.0.76:8000: connect: connection refused"
Jul 28 11:45:08.094890 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:45:08.094805    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:45:10.619447 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:45:10.619402    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6b9d46dd5-xfjdj" podUID="b0c7fc8d-5fda-4241-9795-5d6a8967fec4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/health\": dial tcp 10.134.0.76:8000: connect: connection refused"
Jul 28 11:45:20.619787 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:45:20.619737    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6b9d46dd5-xfjdj" podUID="b0c7fc8d-5fda-4241-9795-5d6a8967fec4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/health\": dial tcp 10.134.0.76:8000: connect: connection refused"
Jul 28 11:45:30.619378 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:45:30.619328    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6b9d46dd5-xfjdj" podUID="b0c7fc8d-5fda-4241-9795-5d6a8967fec4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/health\": dial tcp 10.134.0.76:8000: connect: connection refused"
Jul 28 11:45:40.619765 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:45:40.619717    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6b9d46dd5-xfjdj" podUID="b0c7fc8d-5fda-4241-9795-5d6a8967fec4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/health\": dial tcp 10.134.0.76:8000: connect: connection refused"
Jul 28 11:45:50.619241 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:45:50.619194    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6b9d46dd5-xfjdj" podUID="b0c7fc8d-5fda-4241-9795-5d6a8967fec4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/health\": dial tcp 10.134.0.76:8000: connect: connection refused"
Jul 28 11:46:00.619694 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:46:00.619632    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6b9d46dd5-xfjdj" podUID="b0c7fc8d-5fda-4241-9795-5d6a8967fec4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/health\": dial tcp 10.134.0.76:8000: connect: connection refused"
Jul 28 11:46:10.619326 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:46:10.619264    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6b9d46dd5-xfjdj" podUID="b0c7fc8d-5fda-4241-9795-5d6a8967fec4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/health\": dial tcp 10.134.0.76:8000: connect: connection refused"
Jul 28 11:46:20.619432 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:46:20.619386    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6b9d46dd5-xfjdj" podUID="b0c7fc8d-5fda-4241-9795-5d6a8967fec4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/health\": dial tcp 10.134.0.76:8000: connect: connection refused"
Jul 28 11:46:30.619947 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:46:30.619900    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6b9d46dd5-xfjdj" podUID="b0c7fc8d-5fda-4241-9795-5d6a8967fec4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/health\": dial tcp 10.134.0.76:8000: connect: connection refused"
Jul 28 11:46:40.619448 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:46:40.619357    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6b9d46dd5-xfjdj" podUID="b0c7fc8d-5fda-4241-9795-5d6a8967fec4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/health\": dial tcp 10.134.0.76:8000: connect: connection refused"
Jul 28 11:46:48.817946 ip-10-0-141-50 kubenswrapper[2579]: E0728 11:46:48.817892    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bgw9ln" podUID="9f2f8d3c-6c27-4e8b-92c4-d22da1bb1cbe"
Jul 28 11:46:49.486079 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:46:49.486041    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bgw9ln"
Jul 28 11:46:50.630052 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:46:50.630021    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6b9d46dd5-xfjdj"
Jul 28 11:46:50.638122 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:46:50.638095    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6b9d46dd5-xfjdj"
Jul 28 11:46:52.108270 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:46:52.108232    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6b9d46dd5-xfjdj"]
Jul 28 11:46:52.135593 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:46:52.135552    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6b9d46dd5-kzblc"]
Jul 28 11:46:52.138633 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:46:52.138608    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6b9d46dd5-kzblc"
Jul 28 11:46:52.154768 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:46:52.154740    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6b9d46dd5-kzblc"]
Jul 28 11:46:52.261797 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:46:52.261747    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/132c670c-a118-496d-869f-e987c77fcae2-kserve-provision-location\") pod \"prestop-hook-test-kserve-6b9d46dd5-kzblc\" (UID: \"132c670c-a118-496d-869f-e987c77fcae2\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6b9d46dd5-kzblc"
Jul 28 11:46:52.262040 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:46:52.261806    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/132c670c-a118-496d-869f-e987c77fcae2-dshm\") pod \"prestop-hook-test-kserve-6b9d46dd5-kzblc\" (UID: \"132c670c-a118-496d-869f-e987c77fcae2\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6b9d46dd5-kzblc"
Jul 28 11:46:52.262040 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:46:52.261901    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5d8kl\" (UniqueName: \"kubernetes.io/projected/132c670c-a118-496d-869f-e987c77fcae2-kube-api-access-5d8kl\") pod \"prestop-hook-test-kserve-6b9d46dd5-kzblc\" (UID: \"132c670c-a118-496d-869f-e987c77fcae2\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6b9d46dd5-kzblc"
Jul 28 11:46:52.262040 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:46:52.261934    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/132c670c-a118-496d-869f-e987c77fcae2-model-cache\") pod \"prestop-hook-test-kserve-6b9d46dd5-kzblc\" (UID: \"132c670c-a118-496d-869f-e987c77fcae2\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6b9d46dd5-kzblc"
Jul 28 11:46:52.262040 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:46:52.261951    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/132c670c-a118-496d-869f-e987c77fcae2-tmp-dir\") pod \"prestop-hook-test-kserve-6b9d46dd5-kzblc\" (UID: \"132c670c-a118-496d-869f-e987c77fcae2\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6b9d46dd5-kzblc"
Jul 28 11:46:52.262040 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:46:52.262028    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/132c670c-a118-496d-869f-e987c77fcae2-tls-certs\") pod \"prestop-hook-test-kserve-6b9d46dd5-kzblc\" (UID: \"132c670c-a118-496d-869f-e987c77fcae2\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6b9d46dd5-kzblc"
Jul 28 11:46:52.262247 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:46:52.262063    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/132c670c-a118-496d-869f-e987c77fcae2-home\") pod \"prestop-hook-test-kserve-6b9d46dd5-kzblc\" (UID: \"132c670c-a118-496d-869f-e987c77fcae2\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6b9d46dd5-kzblc"
Jul 28 11:46:52.363100 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:46:52.363003    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/132c670c-a118-496d-869f-e987c77fcae2-kserve-provision-location\") pod \"prestop-hook-test-kserve-6b9d46dd5-kzblc\" (UID: \"132c670c-a118-496d-869f-e987c77fcae2\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6b9d46dd5-kzblc"
Jul 28 11:46:52.363100 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:46:52.363042    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/132c670c-a118-496d-869f-e987c77fcae2-dshm\") pod \"prestop-hook-test-kserve-6b9d46dd5-kzblc\" (UID: \"132c670c-a118-496d-869f-e987c77fcae2\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6b9d46dd5-kzblc"
Jul 28 11:46:52.363100 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:46:52.363075    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5d8kl\" (UniqueName: \"kubernetes.io/projected/132c670c-a118-496d-869f-e987c77fcae2-kube-api-access-5d8kl\") pod \"prestop-hook-test-kserve-6b9d46dd5-kzblc\" (UID: \"132c670c-a118-496d-869f-e987c77fcae2\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6b9d46dd5-kzblc"
Jul 28 11:46:52.363416 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:46:52.363191    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/132c670c-a118-496d-869f-e987c77fcae2-model-cache\") pod \"prestop-hook-test-kserve-6b9d46dd5-kzblc\" (UID: \"132c670c-a118-496d-869f-e987c77fcae2\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6b9d46dd5-kzblc"
Jul 28 11:46:52.363416 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:46:52.363230    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/132c670c-a118-496d-869f-e987c77fcae2-tmp-dir\") pod \"prestop-hook-test-kserve-6b9d46dd5-kzblc\" (UID: \"132c670c-a118-496d-869f-e987c77fcae2\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6b9d46dd5-kzblc"
Jul 28 11:46:52.363416 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:46:52.363277    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/132c670c-a118-496d-869f-e987c77fcae2-tls-certs\") pod \"prestop-hook-test-kserve-6b9d46dd5-kzblc\" (UID: \"132c670c-a118-496d-869f-e987c77fcae2\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6b9d46dd5-kzblc"
Jul 28 11:46:52.363416 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:46:52.363314    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/132c670c-a118-496d-869f-e987c77fcae2-home\") pod \"prestop-hook-test-kserve-6b9d46dd5-kzblc\" (UID: \"132c670c-a118-496d-869f-e987c77fcae2\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6b9d46dd5-kzblc"
Jul 28 11:46:52.363639 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:46:52.363555    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/132c670c-a118-496d-869f-e987c77fcae2-kserve-provision-location\") pod \"prestop-hook-test-kserve-6b9d46dd5-kzblc\" (UID: \"132c670c-a118-496d-869f-e987c77fcae2\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6b9d46dd5-kzblc"
Jul 28 11:46:52.363639 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:46:52.363626    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/132c670c-a118-496d-869f-e987c77fcae2-model-cache\") pod \"prestop-hook-test-kserve-6b9d46dd5-kzblc\" (UID: \"132c670c-a118-496d-869f-e987c77fcae2\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6b9d46dd5-kzblc"
Jul 28 11:46:52.363746 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:46:52.363730    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/132c670c-a118-496d-869f-e987c77fcae2-home\") pod \"prestop-hook-test-kserve-6b9d46dd5-kzblc\" (UID: \"132c670c-a118-496d-869f-e987c77fcae2\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6b9d46dd5-kzblc"
Jul 28 11:46:52.363784 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:46:52.363726    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/132c670c-a118-496d-869f-e987c77fcae2-tmp-dir\") pod \"prestop-hook-test-kserve-6b9d46dd5-kzblc\" (UID: \"132c670c-a118-496d-869f-e987c77fcae2\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6b9d46dd5-kzblc"
Jul 28 11:46:52.365361 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:46:52.365340    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/132c670c-a118-496d-869f-e987c77fcae2-dshm\") pod \"prestop-hook-test-kserve-6b9d46dd5-kzblc\" (UID: \"132c670c-a118-496d-869f-e987c77fcae2\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6b9d46dd5-kzblc"
Jul 28 11:46:52.365792 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:46:52.365773    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/132c670c-a118-496d-869f-e987c77fcae2-tls-certs\") pod \"prestop-hook-test-kserve-6b9d46dd5-kzblc\" (UID: \"132c670c-a118-496d-869f-e987c77fcae2\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6b9d46dd5-kzblc"
Jul 28 11:46:52.371440 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:46:52.371414    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5d8kl\" (UniqueName: \"kubernetes.io/projected/132c670c-a118-496d-869f-e987c77fcae2-kube-api-access-5d8kl\") pod \"prestop-hook-test-kserve-6b9d46dd5-kzblc\" (UID: \"132c670c-a118-496d-869f-e987c77fcae2\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6b9d46dd5-kzblc"
Jul 28 11:46:52.449260 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:46:52.449215    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6b9d46dd5-kzblc"
Jul 28 11:46:52.599070 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:46:52.599040    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6b9d46dd5-kzblc"]
Jul 28 11:46:52.601344 ip-10-0-141-50 kubenswrapper[2579]: W0728 11:46:52.601314    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod132c670c_a118_496d_869f_e987c77fcae2.slice/crio-90d223dc2432868c27c6809ccf33635e2b787f87f3d73c1406f03aaaf4eb5bc4 WatchSource:0}: Error finding container 90d223dc2432868c27c6809ccf33635e2b787f87f3d73c1406f03aaaf4eb5bc4: Status 404 returned error can't find the container with id 90d223dc2432868c27c6809ccf33635e2b787f87f3d73c1406f03aaaf4eb5bc4
Jul 28 11:46:53.510716 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:46:53.510655    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6b9d46dd5-kzblc" event={"ID":"132c670c-a118-496d-869f-e987c77fcae2","Type":"ContainerStarted","Data":"8a78bdb98774a93db9f66ae565d2cafac834b2b8494b8873bb5dff8471ed834e"}
Jul 28 11:46:53.511203 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:46:53.510751    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6b9d46dd5-kzblc" event={"ID":"132c670c-a118-496d-869f-e987c77fcae2","Type":"ContainerStarted","Data":"90d223dc2432868c27c6809ccf33635e2b787f87f3d73c1406f03aaaf4eb5bc4"}
Jul 28 11:46:57.532798 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:46:57.532761    2579 generic.go:356] "Generic (PLEG): container finished" podID="132c670c-a118-496d-869f-e987c77fcae2" containerID="8a78bdb98774a93db9f66ae565d2cafac834b2b8494b8873bb5dff8471ed834e" exitCode=0
Jul 28 11:46:57.533197 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:46:57.532833    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6b9d46dd5-kzblc" event={"ID":"132c670c-a118-496d-869f-e987c77fcae2","Type":"ContainerDied","Data":"8a78bdb98774a93db9f66ae565d2cafac834b2b8494b8873bb5dff8471ed834e"}
Jul 28 11:46:58.539948 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:46:58.539915    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6b9d46dd5-kzblc" event={"ID":"132c670c-a118-496d-869f-e987c77fcae2","Type":"ContainerStarted","Data":"7043fb148cf1aca0d777896b1f75df14e4a0dc227feba640afbd6856a5d52c1e"}
Jul 28 11:46:58.564431 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:46:58.564373    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6b9d46dd5-kzblc" podStartSLOduration=1.888180732 podStartE2EDuration="6.564357729s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.676176997s" isStatefulPod=false podCreationTimestamp="2026-07-28 11:46:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 11:46:58.562702583 +0000 UTC m=+5952.826098913" watchObservedRunningTime="2026-07-28 11:46:58.564357729 +0000 UTC m=+5952.827754059"
Jul 28 11:46:59.449811 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:46:59.449764    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6b9d46dd5-kzblc"
Jul 28 11:47:02.450450 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:47:02.450409    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6b9d46dd5-kzblc"
Jul 28 11:47:02.452005 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:47:02.451971    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6b9d46dd5-kzblc" podUID="132c670c-a118-496d-869f-e987c77fcae2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": dial tcp 10.134.0.77:8000: connect: connection refused"
Jul 28 11:47:07.532751 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:47:07.532692    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6b9d46dd5-xfjdj" podUID="b0c7fc8d-5fda-4241-9795-5d6a8967fec4" containerName="main" containerID="cri-o://eda6b3f893c891ddd6cbb6c8a694e1d6ada706d259c7f12d3e1ec8256f003659" gracePeriod=45
Jul 28 11:47:07.631047 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:47:07.630997    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6b9d46dd5-xfjdj" podUID="b0c7fc8d-5fda-4241-9795-5d6a8967fec4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/health\": dial tcp 10.134.0.76:8000: connect: connection refused"
Jul 28 11:47:08.313577 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:47:08.313532    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6b9d46dd5-xfjdj"
Jul 28 11:47:08.396284 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:47:08.396185    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0c7fc8d-5fda-4241-9795-5d6a8967fec4-model-cache\") pod \"b0c7fc8d-5fda-4241-9795-5d6a8967fec4\" (UID: \"b0c7fc8d-5fda-4241-9795-5d6a8967fec4\") "
Jul 28 11:47:08.396284 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:47:08.396239    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b0c7fc8d-5fda-4241-9795-5d6a8967fec4-kserve-provision-location\") pod \"b0c7fc8d-5fda-4241-9795-5d6a8967fec4\" (UID: \"b0c7fc8d-5fda-4241-9795-5d6a8967fec4\") "
Jul 28 11:47:08.396284 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:47:08.396261    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0c7fc8d-5fda-4241-9795-5d6a8967fec4-dshm\") pod \"b0c7fc8d-5fda-4241-9795-5d6a8967fec4\" (UID: \"b0c7fc8d-5fda-4241-9795-5d6a8967fec4\") "
Jul 28 11:47:08.396596 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:47:08.396303    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0c7fc8d-5fda-4241-9795-5d6a8967fec4-tmp-dir\") pod \"b0c7fc8d-5fda-4241-9795-5d6a8967fec4\" (UID: \"b0c7fc8d-5fda-4241-9795-5d6a8967fec4\") "
Jul 28 11:47:08.396596 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:47:08.396328    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jxw9f\" (UniqueName: \"kubernetes.io/projected/b0c7fc8d-5fda-4241-9795-5d6a8967fec4-kube-api-access-jxw9f\") pod \"b0c7fc8d-5fda-4241-9795-5d6a8967fec4\" (UID: \"b0c7fc8d-5fda-4241-9795-5d6a8967fec4\") "
Jul 28 11:47:08.396596 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:47:08.396357    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0c7fc8d-5fda-4241-9795-5d6a8967fec4-tls-certs\") pod \"b0c7fc8d-5fda-4241-9795-5d6a8967fec4\" (UID: \"b0c7fc8d-5fda-4241-9795-5d6a8967fec4\") "
Jul 28 11:47:08.396596 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:47:08.396387    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0c7fc8d-5fda-4241-9795-5d6a8967fec4-home\") pod \"b0c7fc8d-5fda-4241-9795-5d6a8967fec4\" (UID: \"b0c7fc8d-5fda-4241-9795-5d6a8967fec4\") "
Jul 28 11:47:08.396596 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:47:08.396458    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0c7fc8d-5fda-4241-9795-5d6a8967fec4-model-cache" (OuterVolumeSpecName: "model-cache") pod "b0c7fc8d-5fda-4241-9795-5d6a8967fec4" (UID: "b0c7fc8d-5fda-4241-9795-5d6a8967fec4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 11:47:08.396874 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:47:08.396709    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0c7fc8d-5fda-4241-9795-5d6a8967fec4-model-cache\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 11:47:08.397355 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:47:08.397334    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0c7fc8d-5fda-4241-9795-5d6a8967fec4-home" (OuterVolumeSpecName: "home") pod "b0c7fc8d-5fda-4241-9795-5d6a8967fec4" (UID: "b0c7fc8d-5fda-4241-9795-5d6a8967fec4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 11:47:08.399311 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:47:08.399270    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0c7fc8d-5fda-4241-9795-5d6a8967fec4-dshm" (OuterVolumeSpecName: "dshm") pod "b0c7fc8d-5fda-4241-9795-5d6a8967fec4" (UID: "b0c7fc8d-5fda-4241-9795-5d6a8967fec4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 11:47:08.399419 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:47:08.399318    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0c7fc8d-5fda-4241-9795-5d6a8967fec4-kube-api-access-jxw9f" (OuterVolumeSpecName: "kube-api-access-jxw9f") pod "b0c7fc8d-5fda-4241-9795-5d6a8967fec4" (UID: "b0c7fc8d-5fda-4241-9795-5d6a8967fec4"). InnerVolumeSpecName "kube-api-access-jxw9f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 11:47:08.399419 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:47:08.399396    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0c7fc8d-5fda-4241-9795-5d6a8967fec4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b0c7fc8d-5fda-4241-9795-5d6a8967fec4" (UID: "b0c7fc8d-5fda-4241-9795-5d6a8967fec4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 11:47:08.414634 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:47:08.414592    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0c7fc8d-5fda-4241-9795-5d6a8967fec4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b0c7fc8d-5fda-4241-9795-5d6a8967fec4" (UID: "b0c7fc8d-5fda-4241-9795-5d6a8967fec4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 11:47:08.472498 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:47:08.472453    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0c7fc8d-5fda-4241-9795-5d6a8967fec4-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b0c7fc8d-5fda-4241-9795-5d6a8967fec4" (UID: "b0c7fc8d-5fda-4241-9795-5d6a8967fec4"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 11:47:08.497663 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:47:08.497628    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0c7fc8d-5fda-4241-9795-5d6a8967fec4-tmp-dir\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 11:47:08.497663 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:47:08.497657    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jxw9f\" (UniqueName: \"kubernetes.io/projected/b0c7fc8d-5fda-4241-9795-5d6a8967fec4-kube-api-access-jxw9f\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 11:47:08.497663 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:47:08.497667    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0c7fc8d-5fda-4241-9795-5d6a8967fec4-tls-certs\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 11:47:08.497882 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:47:08.497675    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0c7fc8d-5fda-4241-9795-5d6a8967fec4-home\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 11:47:08.497882 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:47:08.497683    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b0c7fc8d-5fda-4241-9795-5d6a8967fec4-kserve-provision-location\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 11:47:08.497882 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:47:08.497691    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0c7fc8d-5fda-4241-9795-5d6a8967fec4-dshm\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 11:47:08.601196 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:47:08.601163    2579 generic.go:356] "Generic (PLEG): container finished" podID="b0c7fc8d-5fda-4241-9795-5d6a8967fec4" containerID="eda6b3f893c891ddd6cbb6c8a694e1d6ada706d259c7f12d3e1ec8256f003659" exitCode=0
Jul 28 11:47:08.601634 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:47:08.601217    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6b9d46dd5-xfjdj"
Jul 28 11:47:08.601634 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:47:08.601239    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6b9d46dd5-xfjdj" event={"ID":"b0c7fc8d-5fda-4241-9795-5d6a8967fec4","Type":"ContainerDied","Data":"eda6b3f893c891ddd6cbb6c8a694e1d6ada706d259c7f12d3e1ec8256f003659"}
Jul 28 11:47:08.601634 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:47:08.601296    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6b9d46dd5-xfjdj" event={"ID":"b0c7fc8d-5fda-4241-9795-5d6a8967fec4","Type":"ContainerDied","Data":"94d41ca7896db2777f034fa8310119f8b2811cd3413b18f98673cabaa3d515b6"}
Jul 28 11:47:08.601634 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:47:08.601318    2579 scope.go:117] "RemoveContainer" containerID="eda6b3f893c891ddd6cbb6c8a694e1d6ada706d259c7f12d3e1ec8256f003659"
Jul 28 11:47:08.612071 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:47:08.612047    2579 scope.go:117] "RemoveContainer" containerID="e9efc4f2170e739ac010664066f5fa651f0932ddacca42bcc667c3c12e42992e"
Jul 28 11:47:08.622682 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:47:08.622654    2579 scope.go:117] "RemoveContainer" containerID="eda6b3f893c891ddd6cbb6c8a694e1d6ada706d259c7f12d3e1ec8256f003659"
Jul 28 11:47:08.623046 ip-10-0-141-50 kubenswrapper[2579]: E0728 11:47:08.623024    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eda6b3f893c891ddd6cbb6c8a694e1d6ada706d259c7f12d3e1ec8256f003659\": container with ID starting with eda6b3f893c891ddd6cbb6c8a694e1d6ada706d259c7f12d3e1ec8256f003659 not found: ID does not exist" containerID="eda6b3f893c891ddd6cbb6c8a694e1d6ada706d259c7f12d3e1ec8256f003659"
Jul 28 11:47:08.623111 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:47:08.623078    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eda6b3f893c891ddd6cbb6c8a694e1d6ada706d259c7f12d3e1ec8256f003659"} err="failed to get container status \"eda6b3f893c891ddd6cbb6c8a694e1d6ada706d259c7f12d3e1ec8256f003659\": rpc error: code = NotFound desc = could not find container \"eda6b3f893c891ddd6cbb6c8a694e1d6ada706d259c7f12d3e1ec8256f003659\": container with ID starting with eda6b3f893c891ddd6cbb6c8a694e1d6ada706d259c7f12d3e1ec8256f003659 not found: ID does not exist"
Jul 28 11:47:08.623111 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:47:08.623100    2579 scope.go:117] "RemoveContainer" containerID="e9efc4f2170e739ac010664066f5fa651f0932ddacca42bcc667c3c12e42992e"
Jul 28 11:47:08.623374 ip-10-0-141-50 kubenswrapper[2579]: E0728 11:47:08.623354    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9efc4f2170e739ac010664066f5fa651f0932ddacca42bcc667c3c12e42992e\": container with ID starting with e9efc4f2170e739ac010664066f5fa651f0932ddacca42bcc667c3c12e42992e not found: ID does not exist" containerID="e9efc4f2170e739ac010664066f5fa651f0932ddacca42bcc667c3c12e42992e"
Jul 28 11:47:08.623438 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:47:08.623377    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9efc4f2170e739ac010664066f5fa651f0932ddacca42bcc667c3c12e42992e"} err="failed to get container status \"e9efc4f2170e739ac010664066f5fa651f0932ddacca42bcc667c3c12e42992e\": rpc error: code = NotFound desc = could not find container \"e9efc4f2170e739ac010664066f5fa651f0932ddacca42bcc667c3c12e42992e\": container with ID starting with e9efc4f2170e739ac010664066f5fa651f0932ddacca42bcc667c3c12e42992e not found: ID does not exist"
Jul 28 11:47:08.623899 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:47:08.623870    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6b9d46dd5-xfjdj"]
Jul 28 11:47:08.628950 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:47:08.628920    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6b9d46dd5-xfjdj"]
Jul 28 11:47:10.276498 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:47:10.276455    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b0c7fc8d-5fda-4241-9795-5d6a8967fec4" path="/var/lib/kubelet/pods/b0c7fc8d-5fda-4241-9795-5d6a8967fec4/volumes"
Jul 28 11:47:12.450049 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:47:12.449990    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6b9d46dd5-kzblc" podUID="132c670c-a118-496d-869f-e987c77fcae2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": dial tcp 10.134.0.77:8000: connect: connection refused"
Jul 28 11:47:17.289863 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:47:17.289830    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6b9d46dd5-kzblc"]
Jul 28 11:47:17.315525 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:47:17.315495    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:47:17.400755 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:47:17.400722    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:47:32.311895 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:47:32.311848    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6b9d46dd5-kzblc" podUID="132c670c-a118-496d-869f-e987c77fcae2" containerName="main" containerID="cri-o://7043fb148cf1aca0d777896b1f75df14e4a0dc227feba640afbd6856a5d52c1e" gracePeriod=45
Jul 28 11:48:17.970939 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:48:17.970910    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-prestop-hook-22727e61_prestop-hook-test-kserve-6b9d46dd5-kzblc_132c670c-a118-496d-869f-e987c77fcae2/main/0.log"
Jul 28 11:48:17.972464 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:48:17.972447    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6b9d46dd5-kzblc"
Jul 28 11:48:17.989250 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:48:17.989226    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-prestop-hook-22727e61_prestop-hook-test-kserve-6b9d46dd5-kzblc_132c670c-a118-496d-869f-e987c77fcae2/main/0.log"
Jul 28 11:48:17.989615 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:48:17.989593    2579 generic.go:356] "Generic (PLEG): container finished" podID="132c670c-a118-496d-869f-e987c77fcae2" containerID="7043fb148cf1aca0d777896b1f75df14e4a0dc227feba640afbd6856a5d52c1e" exitCode=137
Jul 28 11:48:17.989698 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:48:17.989645    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6b9d46dd5-kzblc"
Jul 28 11:48:17.989698 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:48:17.989664    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6b9d46dd5-kzblc" event={"ID":"132c670c-a118-496d-869f-e987c77fcae2","Type":"ContainerDied","Data":"7043fb148cf1aca0d777896b1f75df14e4a0dc227feba640afbd6856a5d52c1e"}
Jul 28 11:48:17.989789 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:48:17.989706    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6b9d46dd5-kzblc" event={"ID":"132c670c-a118-496d-869f-e987c77fcae2","Type":"ContainerDied","Data":"90d223dc2432868c27c6809ccf33635e2b787f87f3d73c1406f03aaaf4eb5bc4"}
Jul 28 11:48:17.989789 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:48:17.989728    2579 scope.go:117] "RemoveContainer" containerID="7043fb148cf1aca0d777896b1f75df14e4a0dc227feba640afbd6856a5d52c1e"
Jul 28 11:48:17.999934 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:48:17.999913    2579 scope.go:117] "RemoveContainer" containerID="8a78bdb98774a93db9f66ae565d2cafac834b2b8494b8873bb5dff8471ed834e"
Jul 28 11:48:18.010416 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:48:18.010392    2579 scope.go:117] "RemoveContainer" containerID="7043fb148cf1aca0d777896b1f75df14e4a0dc227feba640afbd6856a5d52c1e"
Jul 28 11:48:18.010754 ip-10-0-141-50 kubenswrapper[2579]: E0728 11:48:18.010734    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7043fb148cf1aca0d777896b1f75df14e4a0dc227feba640afbd6856a5d52c1e\": container with ID starting with 7043fb148cf1aca0d777896b1f75df14e4a0dc227feba640afbd6856a5d52c1e not found: ID does not exist" containerID="7043fb148cf1aca0d777896b1f75df14e4a0dc227feba640afbd6856a5d52c1e"
Jul 28 11:48:18.010818 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:48:18.010765    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7043fb148cf1aca0d777896b1f75df14e4a0dc227feba640afbd6856a5d52c1e"} err="failed to get container status \"7043fb148cf1aca0d777896b1f75df14e4a0dc227feba640afbd6856a5d52c1e\": rpc error: code = NotFound desc = could not find container \"7043fb148cf1aca0d777896b1f75df14e4a0dc227feba640afbd6856a5d52c1e\": container with ID starting with 7043fb148cf1aca0d777896b1f75df14e4a0dc227feba640afbd6856a5d52c1e not found: ID does not exist"
Jul 28 11:48:18.010818 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:48:18.010784    2579 scope.go:117] "RemoveContainer" containerID="8a78bdb98774a93db9f66ae565d2cafac834b2b8494b8873bb5dff8471ed834e"
Jul 28 11:48:18.011050 ip-10-0-141-50 kubenswrapper[2579]: E0728 11:48:18.011031    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a78bdb98774a93db9f66ae565d2cafac834b2b8494b8873bb5dff8471ed834e\": container with ID starting with 8a78bdb98774a93db9f66ae565d2cafac834b2b8494b8873bb5dff8471ed834e not found: ID does not exist" containerID="8a78bdb98774a93db9f66ae565d2cafac834b2b8494b8873bb5dff8471ed834e"
Jul 28 11:48:18.011103 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:48:18.011055    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a78bdb98774a93db9f66ae565d2cafac834b2b8494b8873bb5dff8471ed834e"} err="failed to get container status \"8a78bdb98774a93db9f66ae565d2cafac834b2b8494b8873bb5dff8471ed834e\": rpc error: code = NotFound desc = could not find container \"8a78bdb98774a93db9f66ae565d2cafac834b2b8494b8873bb5dff8471ed834e\": container with ID starting with 8a78bdb98774a93db9f66ae565d2cafac834b2b8494b8873bb5dff8471ed834e not found: ID does not exist"
Jul 28 11:48:18.126690 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:48:18.126596    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5d8kl\" (UniqueName: \"kubernetes.io/projected/132c670c-a118-496d-869f-e987c77fcae2-kube-api-access-5d8kl\") pod \"132c670c-a118-496d-869f-e987c77fcae2\" (UID: \"132c670c-a118-496d-869f-e987c77fcae2\") "
Jul 28 11:48:18.126690 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:48:18.126666    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/132c670c-a118-496d-869f-e987c77fcae2-kserve-provision-location\") pod \"132c670c-a118-496d-869f-e987c77fcae2\" (UID: \"132c670c-a118-496d-869f-e987c77fcae2\") "
Jul 28 11:48:18.126690 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:48:18.126690    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/132c670c-a118-496d-869f-e987c77fcae2-dshm\") pod \"132c670c-a118-496d-869f-e987c77fcae2\" (UID: \"132c670c-a118-496d-869f-e987c77fcae2\") "
Jul 28 11:48:18.126985 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:48:18.126726    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/132c670c-a118-496d-869f-e987c77fcae2-tls-certs\") pod \"132c670c-a118-496d-869f-e987c77fcae2\" (UID: \"132c670c-a118-496d-869f-e987c77fcae2\") "
Jul 28 11:48:18.126985 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:48:18.126763    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/132c670c-a118-496d-869f-e987c77fcae2-home\") pod \"132c670c-a118-496d-869f-e987c77fcae2\" (UID: \"132c670c-a118-496d-869f-e987c77fcae2\") "
Jul 28 11:48:18.126985 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:48:18.126836    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/132c670c-a118-496d-869f-e987c77fcae2-model-cache\") pod \"132c670c-a118-496d-869f-e987c77fcae2\" (UID: \"132c670c-a118-496d-869f-e987c77fcae2\") "
Jul 28 11:48:18.126985 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:48:18.126861    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/132c670c-a118-496d-869f-e987c77fcae2-tmp-dir\") pod \"132c670c-a118-496d-869f-e987c77fcae2\" (UID: \"132c670c-a118-496d-869f-e987c77fcae2\") "
Jul 28 11:48:18.127278 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:48:18.127224    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/132c670c-a118-496d-869f-e987c77fcae2-model-cache" (OuterVolumeSpecName: "model-cache") pod "132c670c-a118-496d-869f-e987c77fcae2" (UID: "132c670c-a118-496d-869f-e987c77fcae2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 11:48:18.127409 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:48:18.127342    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/132c670c-a118-496d-869f-e987c77fcae2-home" (OuterVolumeSpecName: "home") pod "132c670c-a118-496d-869f-e987c77fcae2" (UID: "132c670c-a118-496d-869f-e987c77fcae2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 11:48:18.127475 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:48:18.127426    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/132c670c-a118-496d-869f-e987c77fcae2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "132c670c-a118-496d-869f-e987c77fcae2" (UID: "132c670c-a118-496d-869f-e987c77fcae2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 11:48:18.128916 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:48:18.128883    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/132c670c-a118-496d-869f-e987c77fcae2-kube-api-access-5d8kl" (OuterVolumeSpecName: "kube-api-access-5d8kl") pod "132c670c-a118-496d-869f-e987c77fcae2" (UID: "132c670c-a118-496d-869f-e987c77fcae2"). InnerVolumeSpecName "kube-api-access-5d8kl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 11:48:18.129020 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:48:18.128950    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/132c670c-a118-496d-869f-e987c77fcae2-dshm" (OuterVolumeSpecName: "dshm") pod "132c670c-a118-496d-869f-e987c77fcae2" (UID: "132c670c-a118-496d-869f-e987c77fcae2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 11:48:18.129020 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:48:18.128968    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/132c670c-a118-496d-869f-e987c77fcae2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "132c670c-a118-496d-869f-e987c77fcae2" (UID: "132c670c-a118-496d-869f-e987c77fcae2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 11:48:18.183944 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:48:18.183905    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/132c670c-a118-496d-869f-e987c77fcae2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "132c670c-a118-496d-869f-e987c77fcae2" (UID: "132c670c-a118-496d-869f-e987c77fcae2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 11:48:18.228282 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:48:18.228250    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/132c670c-a118-496d-869f-e987c77fcae2-tmp-dir\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 11:48:18.228282 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:48:18.228281    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5d8kl\" (UniqueName: \"kubernetes.io/projected/132c670c-a118-496d-869f-e987c77fcae2-kube-api-access-5d8kl\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 11:48:18.228478 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:48:18.228292    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/132c670c-a118-496d-869f-e987c77fcae2-kserve-provision-location\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 11:48:18.228478 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:48:18.228301    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/132c670c-a118-496d-869f-e987c77fcae2-dshm\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 11:48:18.228478 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:48:18.228310    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/132c670c-a118-496d-869f-e987c77fcae2-tls-certs\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 11:48:18.228478 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:48:18.228318    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/132c670c-a118-496d-869f-e987c77fcae2-home\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 11:48:18.228478 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:48:18.228325    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/132c670c-a118-496d-869f-e987c77fcae2-model-cache\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 11:48:18.306443 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:48:18.306412    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6b9d46dd5-kzblc"]
Jul 28 11:48:18.311004 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:48:18.310982    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6b9d46dd5-kzblc"]
Jul 28 11:48:20.275805 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:48:20.275765    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="132c670c-a118-496d-869f-e987c77fcae2" path="/var/lib/kubelet/pods/132c670c-a118-496d-869f-e987c77fcae2/volumes"
Jul 28 11:48:20.826395 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:48:20.826359    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:48:22.077203 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:48:22.077168    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:48:27.076395 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:48:27.076368    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:48:32.657271 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:48:32.657240    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:48:32.762619 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:48:32.762585    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:48:50.422103 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:48:50.422071    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:48:50.592680 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:48:50.592645    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-mqg9m"]
Jul 28 11:48:50.593054 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:48:50.593036    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0c7fc8d-5fda-4241-9795-5d6a8967fec4" containerName="storage-initializer"
Jul 28 11:48:50.593128 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:48:50.593057    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0c7fc8d-5fda-4241-9795-5d6a8967fec4" containerName="storage-initializer"
Jul 28 11:48:50.593128 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:48:50.593070    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="132c670c-a118-496d-869f-e987c77fcae2" containerName="storage-initializer"
Jul 28 11:48:50.593128 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:48:50.593080    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="132c670c-a118-496d-869f-e987c77fcae2" containerName="storage-initializer"
Jul 28 11:48:50.593128 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:48:50.593096    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0c7fc8d-5fda-4241-9795-5d6a8967fec4" containerName="main"
Jul 28 11:48:50.593128 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:48:50.593104    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0c7fc8d-5fda-4241-9795-5d6a8967fec4" containerName="main"
Jul 28 11:48:50.593128 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:48:50.593113    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="132c670c-a118-496d-869f-e987c77fcae2" containerName="main"
Jul 28 11:48:50.593128 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:48:50.593121    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="132c670c-a118-496d-869f-e987c77fcae2" containerName="main"
Jul 28 11:48:50.593382 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:48:50.593199    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="b0c7fc8d-5fda-4241-9795-5d6a8967fec4" containerName="main"
Jul 28 11:48:50.593382 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:48:50.593213    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="132c670c-a118-496d-869f-e987c77fcae2" containerName="main"
Jul 28 11:48:50.597353 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:48:50.597332    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-mqg9m"
Jul 28 11:48:50.600400 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:48:50.600368    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"openshift-service-ca.crt\""
Jul 28 11:48:50.600527 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:48:50.600419    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"kube-root-ca.crt\""
Jul 28 11:48:50.600527 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:48:50.600424    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"rolling-upgrade-test-epp-sa-dockercfg-h854g\""
Jul 28 11:48:50.600527 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:48:50.600431    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"rolling-upgrade-test-kserve-self-signed-certs\""
Jul 28 11:48:50.600527 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:48:50.600480    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"default-dockercfg-2tsmf\""
Jul 28 11:48:50.614064 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:48:50.614039    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-mqg9m"]
Jul 28 11:48:50.631914 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:48:50.631870    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:48:50.708106 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:48:50.707995    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gm74v\" (UniqueName: \"kubernetes.io/projected/7c49ed6e-6906-418c-a3c2-5980804aa29e-kube-api-access-gm74v\") pod \"rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-mqg9m\" (UID: \"7c49ed6e-6906-418c-a3c2-5980804aa29e\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-mqg9m"
Jul 28 11:48:50.708106 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:48:50.708086    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c49ed6e-6906-418c-a3c2-5980804aa29e-tls-certs\") pod \"rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-mqg9m\" (UID: \"7c49ed6e-6906-418c-a3c2-5980804aa29e\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-mqg9m"
Jul 28 11:48:50.809118 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:48:50.809076    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gm74v\" (UniqueName: \"kubernetes.io/projected/7c49ed6e-6906-418c-a3c2-5980804aa29e-kube-api-access-gm74v\") pod \"rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-mqg9m\" (UID: \"7c49ed6e-6906-418c-a3c2-5980804aa29e\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-mqg9m"
Jul 28 11:48:50.809315 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:48:50.809171    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c49ed6e-6906-418c-a3c2-5980804aa29e-tls-certs\") pod \"rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-mqg9m\" (UID: \"7c49ed6e-6906-418c-a3c2-5980804aa29e\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-mqg9m"
Jul 28 11:48:50.811759 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:48:50.811739    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c49ed6e-6906-418c-a3c2-5980804aa29e-tls-certs\") pod \"rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-mqg9m\" (UID: \"7c49ed6e-6906-418c-a3c2-5980804aa29e\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-mqg9m"
Jul 28 11:48:50.818419 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:48:50.818388    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gm74v\" (UniqueName: \"kubernetes.io/projected/7c49ed6e-6906-418c-a3c2-5980804aa29e-kube-api-access-gm74v\") pod \"rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-mqg9m\" (UID: \"7c49ed6e-6906-418c-a3c2-5980804aa29e\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-mqg9m"
Jul 28 11:48:50.912728 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:48:50.912692    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-mqg9m"
Jul 28 11:48:51.048084 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:48:51.048060    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-mqg9m"]
Jul 28 11:48:51.050122 ip-10-0-141-50 kubenswrapper[2579]: W0728 11:48:51.050090    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c49ed6e_6906_418c_a3c2_5980804aa29e.slice/crio-aa08002819d036b820e348357fc621ef8c8b7fe2ad5918d6cb05e2e6f9d27600 WatchSource:0}: Error finding container aa08002819d036b820e348357fc621ef8c8b7fe2ad5918d6cb05e2e6f9d27600: Status 404 returned error can't find the container with id aa08002819d036b820e348357fc621ef8c8b7fe2ad5918d6cb05e2e6f9d27600
Jul 28 11:48:51.146357 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:48:51.146320    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-mqg9m" event={"ID":"7c49ed6e-6906-418c-a3c2-5980804aa29e","Type":"ContainerStarted","Data":"733926411f55c648d451d6a9f6bf924ce37cf5482f1458cbb8bc9139cdf0b64c"}
Jul 28 11:48:51.146606 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:48:51.146384    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-mqg9m" event={"ID":"7c49ed6e-6906-418c-a3c2-5980804aa29e","Type":"ContainerStarted","Data":"aa08002819d036b820e348357fc621ef8c8b7fe2ad5918d6cb05e2e6f9d27600"}
Jul 28 11:48:51.146606 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:48:51.146427    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-mqg9m"
Jul 28 11:48:51.165268 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:48:51.165209    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-mqg9m" podStartSLOduration=1.165193302 podStartE2EDuration="1.165193302s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 11:48:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 11:48:51.163374387 +0000 UTC m=+6065.426770716" watchObservedRunningTime="2026-07-28 11:48:51.165193302 +0000 UTC m=+6065.428589630"
Jul 28 11:48:52.487338 ip-10-0-141-50 kubenswrapper[2579]: E0728 11:48:52.487289    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bgw9ln" podUID="9f2f8d3c-6c27-4e8b-92c4-d22da1bb1cbe"
Jul 28 11:48:53.156796 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:48:53.156761    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bgw9ln"
Jul 28 11:49:14.086713 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:49:14.086678    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:49:14.188153 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:49:14.188116    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:49:22.154662 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:49:22.154631    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-mqg9m"
Jul 28 11:49:27.908295 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:49:27.908252    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:50:04.090290 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:50:04.090197    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-mpp4c"]
Jul 28 11:50:04.093566 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:50:04.093531    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-mpp4c"
Jul 28 11:50:04.104417 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:50:04.104386    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-mpp4c"]
Jul 28 11:50:04.146120 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:50:04.146079    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3584de97-6154-451e-805c-39e65477b94a-dshm\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-mpp4c\" (UID: \"3584de97-6154-451e-805c-39e65477b94a\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-mpp4c"
Jul 28 11:50:04.146342 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:50:04.146156    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3584de97-6154-451e-805c-39e65477b94a-tmp-dir\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-mpp4c\" (UID: \"3584de97-6154-451e-805c-39e65477b94a\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-mpp4c"
Jul 28 11:50:04.146342 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:50:04.146217    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3584de97-6154-451e-805c-39e65477b94a-model-cache\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-mpp4c\" (UID: \"3584de97-6154-451e-805c-39e65477b94a\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-mpp4c"
Jul 28 11:50:04.146342 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:50:04.146251    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nf25x\" (UniqueName: \"kubernetes.io/projected/3584de97-6154-451e-805c-39e65477b94a-kube-api-access-nf25x\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-mpp4c\" (UID: \"3584de97-6154-451e-805c-39e65477b94a\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-mpp4c"
Jul 28 11:50:04.146342 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:50:04.146282    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3584de97-6154-451e-805c-39e65477b94a-tls-certs\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-mpp4c\" (UID: \"3584de97-6154-451e-805c-39e65477b94a\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-mpp4c"
Jul 28 11:50:04.146342 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:50:04.146303    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3584de97-6154-451e-805c-39e65477b94a-home\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-mpp4c\" (UID: \"3584de97-6154-451e-805c-39e65477b94a\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-mpp4c"
Jul 28 11:50:04.247559 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:50:04.247499    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3584de97-6154-451e-805c-39e65477b94a-model-cache\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-mpp4c\" (UID: \"3584de97-6154-451e-805c-39e65477b94a\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-mpp4c"
Jul 28 11:50:04.247781 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:50:04.247568    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nf25x\" (UniqueName: \"kubernetes.io/projected/3584de97-6154-451e-805c-39e65477b94a-kube-api-access-nf25x\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-mpp4c\" (UID: \"3584de97-6154-451e-805c-39e65477b94a\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-mpp4c"
Jul 28 11:50:04.247781 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:50:04.247601    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3584de97-6154-451e-805c-39e65477b94a-tls-certs\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-mpp4c\" (UID: \"3584de97-6154-451e-805c-39e65477b94a\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-mpp4c"
Jul 28 11:50:04.247781 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:50:04.247627    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3584de97-6154-451e-805c-39e65477b94a-home\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-mpp4c\" (UID: \"3584de97-6154-451e-805c-39e65477b94a\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-mpp4c"
Jul 28 11:50:04.247781 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:50:04.247678    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3584de97-6154-451e-805c-39e65477b94a-dshm\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-mpp4c\" (UID: \"3584de97-6154-451e-805c-39e65477b94a\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-mpp4c"
Jul 28 11:50:04.247781 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:50:04.247749    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3584de97-6154-451e-805c-39e65477b94a-tmp-dir\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-mpp4c\" (UID: \"3584de97-6154-451e-805c-39e65477b94a\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-mpp4c"
Jul 28 11:50:04.248263 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:50:04.248231    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3584de97-6154-451e-805c-39e65477b94a-tmp-dir\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-mpp4c\" (UID: \"3584de97-6154-451e-805c-39e65477b94a\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-mpp4c"
Jul 28 11:50:04.248370 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:50:04.248276    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3584de97-6154-451e-805c-39e65477b94a-home\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-mpp4c\" (UID: \"3584de97-6154-451e-805c-39e65477b94a\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-mpp4c"
Jul 28 11:50:04.248752 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:50:04.248729    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3584de97-6154-451e-805c-39e65477b94a-model-cache\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-mpp4c\" (UID: \"3584de97-6154-451e-805c-39e65477b94a\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-mpp4c"
Jul 28 11:50:04.251201 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:50:04.251175    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3584de97-6154-451e-805c-39e65477b94a-dshm\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-mpp4c\" (UID: \"3584de97-6154-451e-805c-39e65477b94a\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-mpp4c"
Jul 28 11:50:04.251395 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:50:04.251362    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3584de97-6154-451e-805c-39e65477b94a-tls-certs\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-mpp4c\" (UID: \"3584de97-6154-451e-805c-39e65477b94a\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-mpp4c"
Jul 28 11:50:04.256473 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:50:04.256443    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nf25x\" (UniqueName: \"kubernetes.io/projected/3584de97-6154-451e-805c-39e65477b94a-kube-api-access-nf25x\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-mpp4c\" (UID: \"3584de97-6154-451e-805c-39e65477b94a\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-mpp4c"
Jul 28 11:50:04.405656 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:50:04.405520    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-mpp4c"
Jul 28 11:50:04.551356 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:50:04.551317    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-mpp4c"]
Jul 28 11:50:05.513970 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:50:05.513929    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-mpp4c" event={"ID":"3584de97-6154-451e-805c-39e65477b94a","Type":"ContainerStarted","Data":"1e815a61c46e5579f0133e9edf813e4518c97ac724eda6abba0c10470f5458e8"}
Jul 28 11:50:05.514492 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:50:05.513987    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-mpp4c" event={"ID":"3584de97-6154-451e-805c-39e65477b94a","Type":"ContainerStarted","Data":"c28abe2c99b97ddf70d038a308f1698a5ebb4b351861d6aecc5f49651662d614"}
Jul 28 11:50:05.534639 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:50:05.534579    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-mpp4c" podStartSLOduration=1.5345622840000002 podStartE2EDuration="1.534562284s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 11:50:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 11:50:05.53235693 +0000 UTC m=+6139.795753262" watchObservedRunningTime="2026-07-28 11:50:05.534562284 +0000 UTC m=+6139.797958612"
Jul 28 11:50:06.406495 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:50:06.406446    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-mpp4c"
Jul 28 11:50:14.405990 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:50:14.405950    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-mpp4c"
Jul 28 11:50:14.418828 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:50:14.418797    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-mpp4c"
Jul 28 11:50:14.570123 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:50:14.570087    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-mpp4c"
Jul 28 11:50:40.692799 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:50:40.692748    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-mqg9m"]
Jul 28 11:50:40.696039 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:50:40.696008    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-mpp4c"]
Jul 28 11:50:40.696740 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:50:40.696690    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:50:40.707305 ip-10-0-141-50 kubenswrapper[2579]: E0728 11:50:40.707255    2579 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 28 11:50:40.707305 ip-10-0-141-50 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 11:50:40.707305 ip-10-0-141-50 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Jul 28 11:50:40.707305 ip-10-0-141-50 kubenswrapper[2579]:  > containerID="1e815a61c46e5579f0133e9edf813e4518c97ac724eda6abba0c10470f5458e8" cmd=["/bin/sleep","15"]
Jul 28 11:50:40.707614 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:50:40.707333    2579 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-mpp4c" message="" err=<
Jul 28 11:50:40.707614 ip-10-0-141-50 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 11:50:40.707614 ip-10-0-141-50 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Jul 28 11:50:40.707614 ip-10-0-141-50 kubenswrapper[2579]:  >
Jul 28 11:50:40.707614 ip-10-0-141-50 kubenswrapper[2579]: E0728 11:50:40.707351    2579 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 28 11:50:40.707614 ip-10-0-141-50 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 11:50:40.707614 ip-10-0-141-50 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Jul 28 11:50:40.707614 ip-10-0-141-50 kubenswrapper[2579]:  > pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-mpp4c" podUID="3584de97-6154-451e-805c-39e65477b94a" containerName="main" containerID="cri-o://1e815a61c46e5579f0133e9edf813e4518c97ac724eda6abba0c10470f5458e8"
Jul 28 11:50:40.707614 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:50:40.707413    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-mpp4c" podUID="3584de97-6154-451e-805c-39e65477b94a" containerName="main" containerID="cri-o://1e815a61c46e5579f0133e9edf813e4518c97ac724eda6abba0c10470f5458e8" gracePeriod=60
Jul 28 11:50:40.796869 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:50:40.796832    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:50:41.461407 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:50:41.461381    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-mpp4c"
Jul 28 11:50:41.578560 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:50:41.578450    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3584de97-6154-451e-805c-39e65477b94a-model-cache\") pod \"3584de97-6154-451e-805c-39e65477b94a\" (UID: \"3584de97-6154-451e-805c-39e65477b94a\") "
Jul 28 11:50:41.578560 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:50:41.578515    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3584de97-6154-451e-805c-39e65477b94a-home\") pod \"3584de97-6154-451e-805c-39e65477b94a\" (UID: \"3584de97-6154-451e-805c-39e65477b94a\") "
Jul 28 11:50:41.578795 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:50:41.578594    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3584de97-6154-451e-805c-39e65477b94a-dshm\") pod \"3584de97-6154-451e-805c-39e65477b94a\" (UID: \"3584de97-6154-451e-805c-39e65477b94a\") "
Jul 28 11:50:41.578795 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:50:41.578643    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nf25x\" (UniqueName: \"kubernetes.io/projected/3584de97-6154-451e-805c-39e65477b94a-kube-api-access-nf25x\") pod \"3584de97-6154-451e-805c-39e65477b94a\" (UID: \"3584de97-6154-451e-805c-39e65477b94a\") "
Jul 28 11:50:41.578795 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:50:41.578729    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3584de97-6154-451e-805c-39e65477b94a-tls-certs\") pod \"3584de97-6154-451e-805c-39e65477b94a\" (UID: \"3584de97-6154-451e-805c-39e65477b94a\") "
Jul 28 11:50:41.578795 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:50:41.578758    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3584de97-6154-451e-805c-39e65477b94a-tmp-dir\") pod \"3584de97-6154-451e-805c-39e65477b94a\" (UID: \"3584de97-6154-451e-805c-39e65477b94a\") "
Jul 28 11:50:41.579019 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:50:41.578924    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3584de97-6154-451e-805c-39e65477b94a-model-cache" (OuterVolumeSpecName: "model-cache") pod "3584de97-6154-451e-805c-39e65477b94a" (UID: "3584de97-6154-451e-805c-39e65477b94a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 11:50:41.579085 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:50:41.579035    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3584de97-6154-451e-805c-39e65477b94a-model-cache\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 11:50:41.579187 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:50:41.579162    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3584de97-6154-451e-805c-39e65477b94a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3584de97-6154-451e-805c-39e65477b94a" (UID: "3584de97-6154-451e-805c-39e65477b94a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 11:50:41.579473 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:50:41.579448    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3584de97-6154-451e-805c-39e65477b94a-home" (OuterVolumeSpecName: "home") pod "3584de97-6154-451e-805c-39e65477b94a" (UID: "3584de97-6154-451e-805c-39e65477b94a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 11:50:41.581121 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:50:41.581094    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3584de97-6154-451e-805c-39e65477b94a-dshm" (OuterVolumeSpecName: "dshm") pod "3584de97-6154-451e-805c-39e65477b94a" (UID: "3584de97-6154-451e-805c-39e65477b94a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 11:50:41.581415 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:50:41.581388    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3584de97-6154-451e-805c-39e65477b94a-kube-api-access-nf25x" (OuterVolumeSpecName: "kube-api-access-nf25x") pod "3584de97-6154-451e-805c-39e65477b94a" (UID: "3584de97-6154-451e-805c-39e65477b94a"). InnerVolumeSpecName "kube-api-access-nf25x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 11:50:41.581672 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:50:41.581650    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3584de97-6154-451e-805c-39e65477b94a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3584de97-6154-451e-805c-39e65477b94a" (UID: "3584de97-6154-451e-805c-39e65477b94a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 11:50:41.679780 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:50:41.679746    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3584de97-6154-451e-805c-39e65477b94a-dshm\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 11:50:41.679780 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:50:41.679776    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nf25x\" (UniqueName: \"kubernetes.io/projected/3584de97-6154-451e-805c-39e65477b94a-kube-api-access-nf25x\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 11:50:41.679780 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:50:41.679787    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3584de97-6154-451e-805c-39e65477b94a-tls-certs\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 11:50:41.680035 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:50:41.679795    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3584de97-6154-451e-805c-39e65477b94a-tmp-dir\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 11:50:41.680035 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:50:41.679803    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3584de97-6154-451e-805c-39e65477b94a-home\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 11:50:41.707240 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:50:41.707209    2579 generic.go:356] "Generic (PLEG): container finished" podID="3584de97-6154-451e-805c-39e65477b94a" containerID="1e815a61c46e5579f0133e9edf813e4518c97ac724eda6abba0c10470f5458e8" exitCode=0
Jul 28 11:50:41.707692 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:50:41.707252    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-mpp4c"
Jul 28 11:50:41.707692 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:50:41.707300    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-mpp4c" event={"ID":"3584de97-6154-451e-805c-39e65477b94a","Type":"ContainerDied","Data":"1e815a61c46e5579f0133e9edf813e4518c97ac724eda6abba0c10470f5458e8"}
Jul 28 11:50:41.707692 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:50:41.707349    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-mpp4c" event={"ID":"3584de97-6154-451e-805c-39e65477b94a","Type":"ContainerDied","Data":"c28abe2c99b97ddf70d038a308f1698a5ebb4b351861d6aecc5f49651662d614"}
Jul 28 11:50:41.707692 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:50:41.707372    2579 scope.go:117] "RemoveContainer" containerID="1e815a61c46e5579f0133e9edf813e4518c97ac724eda6abba0c10470f5458e8"
Jul 28 11:50:41.716818 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:50:41.716799    2579 scope.go:117] "RemoveContainer" containerID="1e815a61c46e5579f0133e9edf813e4518c97ac724eda6abba0c10470f5458e8"
Jul 28 11:50:41.717108 ip-10-0-141-50 kubenswrapper[2579]: E0728 11:50:41.717084    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e815a61c46e5579f0133e9edf813e4518c97ac724eda6abba0c10470f5458e8\": container with ID starting with 1e815a61c46e5579f0133e9edf813e4518c97ac724eda6abba0c10470f5458e8 not found: ID does not exist" containerID="1e815a61c46e5579f0133e9edf813e4518c97ac724eda6abba0c10470f5458e8"
Jul 28 11:50:41.717172 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:50:41.717115    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e815a61c46e5579f0133e9edf813e4518c97ac724eda6abba0c10470f5458e8"} err="failed to get container status \"1e815a61c46e5579f0133e9edf813e4518c97ac724eda6abba0c10470f5458e8\": rpc error: code = NotFound desc = could not find container \"1e815a61c46e5579f0133e9edf813e4518c97ac724eda6abba0c10470f5458e8\": container with ID starting with 1e815a61c46e5579f0133e9edf813e4518c97ac724eda6abba0c10470f5458e8 not found: ID does not exist"
Jul 28 11:50:41.729313 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:50:41.729288    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-mpp4c"]
Jul 28 11:50:41.731258 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:50:41.731238    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-mpp4c"]
Jul 28 11:50:42.154111 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:50:42.154071    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-mqg9m" podUID="7c49ed6e-6906-418c-a3c2-5980804aa29e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 11:50:42.277482 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:50:42.277445    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3584de97-6154-451e-805c-39e65477b94a" path="/var/lib/kubelet/pods/3584de97-6154-451e-805c-39e65477b94a/volumes"
Jul 28 11:50:52.154834 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:50:52.154793    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-mqg9m" podUID="7c49ed6e-6906-418c-a3c2-5980804aa29e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 11:50:55.715377 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:50:55.715332    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-mqg9m" podUID="7c49ed6e-6906-418c-a3c2-5980804aa29e" containerName="main" containerID="cri-o://733926411f55c648d451d6a9f6bf924ce37cf5482f1458cbb8bc9139cdf0b64c" gracePeriod=45
Jul 28 11:50:55.969052 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:50:55.968972    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-mqg9m"
Jul 28 11:50:56.109676 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:50:56.109638    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c49ed6e-6906-418c-a3c2-5980804aa29e-tls-certs\") pod \"7c49ed6e-6906-418c-a3c2-5980804aa29e\" (UID: \"7c49ed6e-6906-418c-a3c2-5980804aa29e\") "
Jul 28 11:50:56.109825 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:50:56.109736    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gm74v\" (UniqueName: \"kubernetes.io/projected/7c49ed6e-6906-418c-a3c2-5980804aa29e-kube-api-access-gm74v\") pod \"7c49ed6e-6906-418c-a3c2-5980804aa29e\" (UID: \"7c49ed6e-6906-418c-a3c2-5980804aa29e\") "
Jul 28 11:50:56.111900 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:50:56.111866    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c49ed6e-6906-418c-a3c2-5980804aa29e-kube-api-access-gm74v" (OuterVolumeSpecName: "kube-api-access-gm74v") pod "7c49ed6e-6906-418c-a3c2-5980804aa29e" (UID: "7c49ed6e-6906-418c-a3c2-5980804aa29e"). InnerVolumeSpecName "kube-api-access-gm74v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 11:50:56.112022 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:50:56.111895    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c49ed6e-6906-418c-a3c2-5980804aa29e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7c49ed6e-6906-418c-a3c2-5980804aa29e" (UID: "7c49ed6e-6906-418c-a3c2-5980804aa29e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 11:50:56.158288 ip-10-0-141-50 kubenswrapper[2579]: E0728 11:50:56.158243    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bgw9ln" podUID="9f2f8d3c-6c27-4e8b-92c4-d22da1bb1cbe"
Jul 28 11:50:56.210616 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:50:56.210579    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c49ed6e-6906-418c-a3c2-5980804aa29e-tls-certs\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 11:50:56.210616 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:50:56.210609    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gm74v\" (UniqueName: \"kubernetes.io/projected/7c49ed6e-6906-418c-a3c2-5980804aa29e-kube-api-access-gm74v\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 11:50:56.784092 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:50:56.784058    2579 generic.go:356] "Generic (PLEG): container finished" podID="7c49ed6e-6906-418c-a3c2-5980804aa29e" containerID="733926411f55c648d451d6a9f6bf924ce37cf5482f1458cbb8bc9139cdf0b64c" exitCode=0
Jul 28 11:50:56.784599 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:50:56.784104    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-mqg9m"
Jul 28 11:50:56.784599 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:50:56.784144    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-mqg9m" event={"ID":"7c49ed6e-6906-418c-a3c2-5980804aa29e","Type":"ContainerDied","Data":"733926411f55c648d451d6a9f6bf924ce37cf5482f1458cbb8bc9139cdf0b64c"}
Jul 28 11:50:56.784599 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:50:56.784195    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-mqg9m" event={"ID":"7c49ed6e-6906-418c-a3c2-5980804aa29e","Type":"ContainerDied","Data":"aa08002819d036b820e348357fc621ef8c8b7fe2ad5918d6cb05e2e6f9d27600"}
Jul 28 11:50:56.784599 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:50:56.784221    2579 scope.go:117] "RemoveContainer" containerID="733926411f55c648d451d6a9f6bf924ce37cf5482f1458cbb8bc9139cdf0b64c"
Jul 28 11:50:56.784599 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:50:56.784335    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bgw9ln"
Jul 28 11:50:56.793496 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:50:56.793474    2579 scope.go:117] "RemoveContainer" containerID="733926411f55c648d451d6a9f6bf924ce37cf5482f1458cbb8bc9139cdf0b64c"
Jul 28 11:50:56.793759 ip-10-0-141-50 kubenswrapper[2579]: E0728 11:50:56.793739    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"733926411f55c648d451d6a9f6bf924ce37cf5482f1458cbb8bc9139cdf0b64c\": container with ID starting with 733926411f55c648d451d6a9f6bf924ce37cf5482f1458cbb8bc9139cdf0b64c not found: ID does not exist" containerID="733926411f55c648d451d6a9f6bf924ce37cf5482f1458cbb8bc9139cdf0b64c"
Jul 28 11:50:56.793830 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:50:56.793768    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"733926411f55c648d451d6a9f6bf924ce37cf5482f1458cbb8bc9139cdf0b64c"} err="failed to get container status \"733926411f55c648d451d6a9f6bf924ce37cf5482f1458cbb8bc9139cdf0b64c\": rpc error: code = NotFound desc = could not find container \"733926411f55c648d451d6a9f6bf924ce37cf5482f1458cbb8bc9139cdf0b64c\": container with ID starting with 733926411f55c648d451d6a9f6bf924ce37cf5482f1458cbb8bc9139cdf0b64c not found: ID does not exist"
Jul 28 11:50:56.803932 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:50:56.803907    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-mqg9m"]
Jul 28 11:50:56.805911 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:50:56.805890    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-mqg9m"]
Jul 28 11:50:58.276335 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:50:58.276302    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c49ed6e-6906-418c-a3c2-5980804aa29e" path="/var/lib/kubelet/pods/7c49ed6e-6906-418c-a3c2-5980804aa29e/volumes"
Jul 28 11:51:00.952289 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:51:00.952250    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:51:01.088141 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:51:01.088110    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:51:01.183445 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:51:01.183412    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:51:04.220486 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:51:04.220443    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-6fdfcc6654-ltdmm"]
Jul 28 11:51:04.220918 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:51:04.220772    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c49ed6e-6906-418c-a3c2-5980804aa29e" containerName="main"
Jul 28 11:51:04.220918 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:51:04.220784    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c49ed6e-6906-418c-a3c2-5980804aa29e" containerName="main"
Jul 28 11:51:04.220918 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:51:04.220802    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3584de97-6154-451e-805c-39e65477b94a" containerName="main"
Jul 28 11:51:04.220918 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:51:04.220808    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="3584de97-6154-451e-805c-39e65477b94a" containerName="main"
Jul 28 11:51:04.220918 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:51:04.220864    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c49ed6e-6906-418c-a3c2-5980804aa29e" containerName="main"
Jul 28 11:51:04.220918 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:51:04.220872    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="3584de97-6154-451e-805c-39e65477b94a" containerName="main"
Jul 28 11:51:04.225097 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:51:04.225074    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-6fdfcc6654-ltdmm"
Jul 28 11:51:04.227757 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:51:04.227728    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-hp4jb\""
Jul 28 11:51:04.227757 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:51:04.227747    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
Jul 28 11:51:04.233788 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:51:04.233757    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-6fdfcc6654-ltdmm"]
Jul 28 11:51:04.385358 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:51:04.385314    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/306f4879-2b49-4fe6-a4ee-f86e354c7017-cert\") pod \"llmisvc-controller-manager-6fdfcc6654-ltdmm\" (UID: \"306f4879-2b49-4fe6-a4ee-f86e354c7017\") " pod="kserve/llmisvc-controller-manager-6fdfcc6654-ltdmm"
Jul 28 11:51:04.385358 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:51:04.385359    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9szx6\" (UniqueName: \"kubernetes.io/projected/306f4879-2b49-4fe6-a4ee-f86e354c7017-kube-api-access-9szx6\") pod \"llmisvc-controller-manager-6fdfcc6654-ltdmm\" (UID: \"306f4879-2b49-4fe6-a4ee-f86e354c7017\") " pod="kserve/llmisvc-controller-manager-6fdfcc6654-ltdmm"
Jul 28 11:51:04.486788 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:51:04.486685    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/306f4879-2b49-4fe6-a4ee-f86e354c7017-cert\") pod \"llmisvc-controller-manager-6fdfcc6654-ltdmm\" (UID: \"306f4879-2b49-4fe6-a4ee-f86e354c7017\") " pod="kserve/llmisvc-controller-manager-6fdfcc6654-ltdmm"
Jul 28 11:51:04.486788 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:51:04.486732    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9szx6\" (UniqueName: \"kubernetes.io/projected/306f4879-2b49-4fe6-a4ee-f86e354c7017-kube-api-access-9szx6\") pod \"llmisvc-controller-manager-6fdfcc6654-ltdmm\" (UID: \"306f4879-2b49-4fe6-a4ee-f86e354c7017\") " pod="kserve/llmisvc-controller-manager-6fdfcc6654-ltdmm"
Jul 28 11:51:04.489368 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:51:04.489342    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/306f4879-2b49-4fe6-a4ee-f86e354c7017-cert\") pod \"llmisvc-controller-manager-6fdfcc6654-ltdmm\" (UID: \"306f4879-2b49-4fe6-a4ee-f86e354c7017\") " pod="kserve/llmisvc-controller-manager-6fdfcc6654-ltdmm"
Jul 28 11:51:04.495086 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:51:04.495050    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9szx6\" (UniqueName: \"kubernetes.io/projected/306f4879-2b49-4fe6-a4ee-f86e354c7017-kube-api-access-9szx6\") pod \"llmisvc-controller-manager-6fdfcc6654-ltdmm\" (UID: \"306f4879-2b49-4fe6-a4ee-f86e354c7017\") " pod="kserve/llmisvc-controller-manager-6fdfcc6654-ltdmm"
Jul 28 11:51:04.535751 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:51:04.535713    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-6fdfcc6654-ltdmm"
Jul 28 11:51:04.670896 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:51:04.670868    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-6fdfcc6654-ltdmm"]
Jul 28 11:51:04.673163 ip-10-0-141-50 kubenswrapper[2579]: W0728 11:51:04.673132    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod306f4879_2b49_4fe6_a4ee_f86e354c7017.slice/crio-958653088a7bd2263845e3ea6c530144dafb7d0f4b40ee56869dac45c52c55af WatchSource:0}: Error finding container 958653088a7bd2263845e3ea6c530144dafb7d0f4b40ee56869dac45c52c55af: Status 404 returned error can't find the container with id 958653088a7bd2263845e3ea6c530144dafb7d0f4b40ee56869dac45c52c55af
Jul 28 11:51:04.674475 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:51:04.674459    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 11:51:04.825293 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:51:04.825255    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-6fdfcc6654-ltdmm" event={"ID":"306f4879-2b49-4fe6-a4ee-f86e354c7017","Type":"ContainerStarted","Data":"958653088a7bd2263845e3ea6c530144dafb7d0f4b40ee56869dac45c52c55af"}
Jul 28 11:51:07.100298 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:51:07.100262    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:51:07.841113 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:51:07.841077    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-6fdfcc6654-ltdmm" event={"ID":"306f4879-2b49-4fe6-a4ee-f86e354c7017","Type":"ContainerStarted","Data":"60cd8268bc137ae9573726c8f1052462d6423161a6e9dfcd645330e733a64709"}
Jul 28 11:51:07.841316 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:51:07.841195    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-6fdfcc6654-ltdmm"
Jul 28 11:51:07.858839 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:51:07.858788    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-6fdfcc6654-ltdmm" podStartSLOduration=1.038122607 podStartE2EDuration="3.858772963s" totalImagesPullingTime="2.820650356s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 11:51:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 11:51:07.8558401 +0000 UTC m=+6202.119236428" watchObservedRunningTime="2026-07-28 11:51:07.858772963 +0000 UTC m=+6202.122169292"
Jul 28 11:51:12.629639 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:51:12.629602    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:51:12.743403 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:51:12.743375    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:51:15.298705 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:51:15.298670    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:51:15.400096 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:51:15.400061    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:51:29.887086 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:51:29.887056    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:51:38.848520 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:51:38.848487    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-6fdfcc6654-ltdmm"
Jul 28 11:51:54.654298 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:51:54.654259    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:52:03.527150 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:52:03.527102    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-router-scheduler-6694c4458b-vmx75"]
Jul 28 11:52:03.530782 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:52:03.530747    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:52:03.593185 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:52:03.593143    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:52:10.471689 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:52:10.471652    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-router-scheduler-6694c4458b-vmx75" podUID="8107ec9e-840d-40d3-8355-aede968b720c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 11:52:18.549925 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:52:18.549887    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-router-scheduler-6694c4458b-vmx75" podUID="8107ec9e-840d-40d3-8355-aede968b720c" containerName="main" containerID="cri-o://e2b1fec452f4626e9c18310c1fec5ef9d80d40e3f37d38e293be881d1b91eed9" gracePeriod=45
Jul 28 11:52:18.806364 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:52:18.806295    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-router-scheduler-6694c4458b-vmx75"
Jul 28 11:52:18.870915 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:52:18.870870    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xd8h5\" (UniqueName: \"kubernetes.io/projected/8107ec9e-840d-40d3-8355-aede968b720c-kube-api-access-xd8h5\") pod \"8107ec9e-840d-40d3-8355-aede968b720c\" (UID: \"8107ec9e-840d-40d3-8355-aede968b720c\") "
Jul 28 11:52:18.871073 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:52:18.870931    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8107ec9e-840d-40d3-8355-aede968b720c-tls-certs\") pod \"8107ec9e-840d-40d3-8355-aede968b720c\" (UID: \"8107ec9e-840d-40d3-8355-aede968b720c\") "
Jul 28 11:52:18.873145 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:52:18.873105    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8107ec9e-840d-40d3-8355-aede968b720c-kube-api-access-xd8h5" (OuterVolumeSpecName: "kube-api-access-xd8h5") pod "8107ec9e-840d-40d3-8355-aede968b720c" (UID: "8107ec9e-840d-40d3-8355-aede968b720c"). InnerVolumeSpecName "kube-api-access-xd8h5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 11:52:18.873145 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:52:18.873132    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8107ec9e-840d-40d3-8355-aede968b720c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8107ec9e-840d-40d3-8355-aede968b720c" (UID: "8107ec9e-840d-40d3-8355-aede968b720c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 11:52:18.972140 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:52:18.972096    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xd8h5\" (UniqueName: \"kubernetes.io/projected/8107ec9e-840d-40d3-8355-aede968b720c-kube-api-access-xd8h5\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 11:52:18.972140 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:52:18.972131    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8107ec9e-840d-40d3-8355-aede968b720c-tls-certs\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Jul 28 11:52:19.203082 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:52:19.202980    2579 generic.go:356] "Generic (PLEG): container finished" podID="8107ec9e-840d-40d3-8355-aede968b720c" containerID="e2b1fec452f4626e9c18310c1fec5ef9d80d40e3f37d38e293be881d1b91eed9" exitCode=0
Jul 28 11:52:19.203082 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:52:19.203022    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-router-scheduler-6694c4458b-vmx75"
Jul 28 11:52:19.203082 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:52:19.203062    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-router-scheduler-6694c4458b-vmx75" event={"ID":"8107ec9e-840d-40d3-8355-aede968b720c","Type":"ContainerDied","Data":"e2b1fec452f4626e9c18310c1fec5ef9d80d40e3f37d38e293be881d1b91eed9"}
Jul 28 11:52:19.203379 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:52:19.203116    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-router-scheduler-6694c4458b-vmx75" event={"ID":"8107ec9e-840d-40d3-8355-aede968b720c","Type":"ContainerDied","Data":"ae35a44aee806629316a92e69eeaf0b36b66e45e0aea23435894f6ef250ca839"}
Jul 28 11:52:19.203379 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:52:19.203139    2579 scope.go:117] "RemoveContainer" containerID="e2b1fec452f4626e9c18310c1fec5ef9d80d40e3f37d38e293be881d1b91eed9"
Jul 28 11:52:19.212284 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:52:19.212261    2579 scope.go:117] "RemoveContainer" containerID="e2b1fec452f4626e9c18310c1fec5ef9d80d40e3f37d38e293be881d1b91eed9"
Jul 28 11:52:19.212576 ip-10-0-141-50 kubenswrapper[2579]: E0728 11:52:19.212556    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2b1fec452f4626e9c18310c1fec5ef9d80d40e3f37d38e293be881d1b91eed9\": container with ID starting with e2b1fec452f4626e9c18310c1fec5ef9d80d40e3f37d38e293be881d1b91eed9 not found: ID does not exist" containerID="e2b1fec452f4626e9c18310c1fec5ef9d80d40e3f37d38e293be881d1b91eed9"
Jul 28 11:52:19.212639 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:52:19.212584    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2b1fec452f4626e9c18310c1fec5ef9d80d40e3f37d38e293be881d1b91eed9"} err="failed to get container status \"e2b1fec452f4626e9c18310c1fec5ef9d80d40e3f37d38e293be881d1b91eed9\": rpc error: code = NotFound desc = could not find container \"e2b1fec452f4626e9c18310c1fec5ef9d80d40e3f37d38e293be881d1b91eed9\": container with ID starting with e2b1fec452f4626e9c18310c1fec5ef9d80d40e3f37d38e293be881d1b91eed9 not found: ID does not exist"
Jul 28 11:52:19.225037 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:52:19.225002    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-router-scheduler-6694c4458b-vmx75"]
Jul 28 11:52:19.231894 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:52:19.231869    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-router-scheduler-6694c4458b-vmx75"]
Jul 28 11:52:20.276118 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:52:20.276074    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8107ec9e-840d-40d3-8355-aede968b720c" path="/var/lib/kubelet/pods/8107ec9e-840d-40d3-8355-aede968b720c/volumes"
Jul 28 11:52:20.476875 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:52:20.476823    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:52:24.830343 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:52:24.830309    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:52:24.909258 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:52:24.909225    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:52:26.383732 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:52:26.383690    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:52:26.484071 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:52:26.484035    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:52:49.053386 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:52:49.053354    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-w6tmc_348441b2-cf7f-43e6-9f8a-1d45d48301d2/discovery/0.log"
Jul 28 11:52:49.070306 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:52:49.070276    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-zf8mf_490ca628-de57-478e-b4b8-af249144d0d2/istio-proxy/0.log"
Jul 28 11:52:49.098507 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:52:49.098477    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-zf8mf_490ca628-de57-478e-b4b8-af249144d0d2/istio-proxy/1.log"
Jul 28 11:52:49.996380 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:52:49.996337    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-w6tmc_348441b2-cf7f-43e6-9f8a-1d45d48301d2/discovery/0.log"
Jul 28 11:52:50.015968 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:52:50.015938    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-zf8mf_490ca628-de57-478e-b4b8-af249144d0d2/istio-proxy/0.log"
Jul 28 11:52:50.042254 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:52:50.042220    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-zf8mf_490ca628-de57-478e-b4b8-af249144d0d2/istio-proxy/1.log"
Jul 28 11:52:50.977876 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:52:50.977842    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-w6tmc_348441b2-cf7f-43e6-9f8a-1d45d48301d2/discovery/0.log"
Jul 28 11:52:51.000160 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:52:51.000124    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-zf8mf_490ca628-de57-478e-b4b8-af249144d0d2/istio-proxy/0.log"
Jul 28 11:52:51.028327 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:52:51.028294    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-zf8mf_490ca628-de57-478e-b4b8-af249144d0d2/istio-proxy/1.log"
Jul 28 11:52:51.980335 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:52:51.980300    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-w6tmc_348441b2-cf7f-43e6-9f8a-1d45d48301d2/discovery/0.log"
Jul 28 11:52:52.026152 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:52:52.026118    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-zf8mf_490ca628-de57-478e-b4b8-af249144d0d2/istio-proxy/0.log"
Jul 28 11:52:52.054647 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:52:52.054614    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-zf8mf_490ca628-de57-478e-b4b8-af249144d0d2/istio-proxy/1.log"
Jul 28 11:52:53.112715 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:52:53.112685    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-tlmfp_f1643f06-8d61-4ace-b0b1-f3082ae86210/manager/0.log"
Jul 28 11:52:53.129999 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:52:53.129971    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-8bdq9_6b5d8a26-4656-441f-8bfa-7579a7956ed0/manager/0.log"
Jul 28 11:52:53.303518 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:52:53.303481    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-2z85v_523cd2de-0f16-403c-be29-876f77b2b2b1/manager/0.log"
Jul 28 11:52:54.527485 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:52:54.527452    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-tlmfp_f1643f06-8d61-4ace-b0b1-f3082ae86210/manager/0.log"
Jul 28 11:52:54.555763 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:52:54.555731    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-8bdq9_6b5d8a26-4656-441f-8bfa-7579a7956ed0/manager/0.log"
Jul 28 11:52:54.729039 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:52:54.729005    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-2z85v_523cd2de-0f16-403c-be29-876f77b2b2b1/manager/0.log"
Jul 28 11:52:55.848826 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:52:55.848794    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-tlmfp_f1643f06-8d61-4ace-b0b1-f3082ae86210/manager/0.log"
Jul 28 11:52:55.884283 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:52:55.884258    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-8bdq9_6b5d8a26-4656-441f-8bfa-7579a7956ed0/manager/0.log"
Jul 28 11:52:56.072964 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:52:56.072928    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-2z85v_523cd2de-0f16-403c-be29-876f77b2b2b1/manager/0.log"
Jul 28 11:52:57.155621 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:52:57.155590    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-tlmfp_f1643f06-8d61-4ace-b0b1-f3082ae86210/manager/0.log"
Jul 28 11:52:57.171172 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:52:57.171141    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-8bdq9_6b5d8a26-4656-441f-8bfa-7579a7956ed0/manager/0.log"
Jul 28 11:52:57.343159 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:52:57.343112    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-2z85v_523cd2de-0f16-403c-be29-876f77b2b2b1/manager/0.log"
Jul 28 11:52:58.421890 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:52:58.421860    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-tlmfp_f1643f06-8d61-4ace-b0b1-f3082ae86210/manager/0.log"
Jul 28 11:52:58.437458 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:52:58.437434    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-8bdq9_6b5d8a26-4656-441f-8bfa-7579a7956ed0/manager/0.log"
Jul 28 11:52:58.617854 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:52:58.617823    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-2z85v_523cd2de-0f16-403c-be29-876f77b2b2b1/manager/0.log"
Jul 28 11:52:59.697467 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:52:59.697435    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-tlmfp_f1643f06-8d61-4ace-b0b1-f3082ae86210/manager/0.log"
Jul 28 11:52:59.714206 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:52:59.714179    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-8bdq9_6b5d8a26-4656-441f-8bfa-7579a7956ed0/manager/0.log"
Jul 28 11:52:59.784836 ip-10-0-141-50 kubenswrapper[2579]: E0728 11:52:59.784790    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bgw9ln" podUID="9f2f8d3c-6c27-4e8b-92c4-d22da1bb1cbe"
Jul 28 11:52:59.884872 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:52:59.884827    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-2z85v_523cd2de-0f16-403c-be29-876f77b2b2b1/manager/0.log"
Jul 28 11:53:00.400311 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:53:00.400282    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bgw9ln"
Jul 28 11:53:00.950890 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:53:00.950850    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-tlmfp_f1643f06-8d61-4ace-b0b1-f3082ae86210/manager/0.log"
Jul 28 11:53:00.965603 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:53:00.965572    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-8bdq9_6b5d8a26-4656-441f-8bfa-7579a7956ed0/manager/0.log"
Jul 28 11:53:01.133372 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:53:01.133336    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-2z85v_523cd2de-0f16-403c-be29-876f77b2b2b1/manager/0.log"
Jul 28 11:53:02.202986 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:53:02.202936    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-tlmfp_f1643f06-8d61-4ace-b0b1-f3082ae86210/manager/0.log"
Jul 28 11:53:02.218478 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:53:02.218450    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-8bdq9_6b5d8a26-4656-441f-8bfa-7579a7956ed0/manager/0.log"
Jul 28 11:53:02.394597 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:53:02.394558    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-2z85v_523cd2de-0f16-403c-be29-876f77b2b2b1/manager/0.log"
Jul 28 11:53:03.468941 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:53:03.468915    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-tlmfp_f1643f06-8d61-4ace-b0b1-f3082ae86210/manager/0.log"
Jul 28 11:53:03.488887 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:53:03.488860    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-8bdq9_6b5d8a26-4656-441f-8bfa-7579a7956ed0/manager/0.log"
Jul 28 11:53:03.670022 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:53:03.669989    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-2z85v_523cd2de-0f16-403c-be29-876f77b2b2b1/manager/0.log"
Jul 28 11:53:04.766679 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:53:04.766643    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-tlmfp_f1643f06-8d61-4ace-b0b1-f3082ae86210/manager/0.log"
Jul 28 11:53:04.784206 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:53:04.784180    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-8bdq9_6b5d8a26-4656-441f-8bfa-7579a7956ed0/manager/0.log"
Jul 28 11:53:04.960420 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:53:04.960381    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-2z85v_523cd2de-0f16-403c-be29-876f77b2b2b1/manager/0.log"
Jul 28 11:53:06.051480 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:53:06.051444    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-tlmfp_f1643f06-8d61-4ace-b0b1-f3082ae86210/manager/0.log"
Jul 28 11:53:06.069919 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:53:06.069894    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-8bdq9_6b5d8a26-4656-441f-8bfa-7579a7956ed0/manager/0.log"
Jul 28 11:53:06.235034 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:53:06.234997    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-2z85v_523cd2de-0f16-403c-be29-876f77b2b2b1/manager/0.log"
Jul 28 11:53:07.300972 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:53:07.300939    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-tlmfp_f1643f06-8d61-4ace-b0b1-f3082ae86210/manager/0.log"
Jul 28 11:53:07.315961 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:53:07.315932    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-8bdq9_6b5d8a26-4656-441f-8bfa-7579a7956ed0/manager/0.log"
Jul 28 11:53:07.489071 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:53:07.489033    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-2z85v_523cd2de-0f16-403c-be29-876f77b2b2b1/manager/0.log"
Jul 28 11:53:08.583247 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:53:08.583221    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-tlmfp_f1643f06-8d61-4ace-b0b1-f3082ae86210/manager/0.log"
Jul 28 11:53:08.599627 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:53:08.599599    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-8bdq9_6b5d8a26-4656-441f-8bfa-7579a7956ed0/manager/0.log"
Jul 28 11:53:08.776356 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:53:08.776315    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-2z85v_523cd2de-0f16-403c-be29-876f77b2b2b1/manager/0.log"
Jul 28 11:53:09.850837 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:53:09.850802    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-tlmfp_f1643f06-8d61-4ace-b0b1-f3082ae86210/manager/0.log"
Jul 28 11:53:09.867285 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:53:09.867256    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-8bdq9_6b5d8a26-4656-441f-8bfa-7579a7956ed0/manager/0.log"
Jul 28 11:53:10.035653 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:53:10.035622    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-2z85v_523cd2de-0f16-403c-be29-876f77b2b2b1/manager/0.log"
Jul 28 11:53:11.094036 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:53:11.094005    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-tlmfp_f1643f06-8d61-4ace-b0b1-f3082ae86210/manager/0.log"
Jul 28 11:53:11.109753 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:53:11.109707    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-8bdq9_6b5d8a26-4656-441f-8bfa-7579a7956ed0/manager/0.log"
Jul 28 11:53:11.282340 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:53:11.282304    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-2z85v_523cd2de-0f16-403c-be29-876f77b2b2b1/manager/0.log"
Jul 28 11:53:12.358296 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:53:12.358264    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-tlmfp_f1643f06-8d61-4ace-b0b1-f3082ae86210/manager/0.log"
Jul 28 11:53:12.373108 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:53:12.373079    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-8bdq9_6b5d8a26-4656-441f-8bfa-7579a7956ed0/manager/0.log"
Jul 28 11:53:12.547052 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:53:12.547016    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-2z85v_523cd2de-0f16-403c-be29-876f77b2b2b1/manager/0.log"
Jul 28 11:53:13.610039 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:53:13.610008    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-tlmfp_f1643f06-8d61-4ace-b0b1-f3082ae86210/manager/0.log"
Jul 28 11:53:13.625266 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:53:13.625237    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-8bdq9_6b5d8a26-4656-441f-8bfa-7579a7956ed0/manager/0.log"
Jul 28 11:53:13.800461 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:53:13.800423    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-2z85v_523cd2de-0f16-403c-be29-876f77b2b2b1/manager/0.log"
Jul 28 11:53:14.933786 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:53:14.933756    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-tlmfp_f1643f06-8d61-4ace-b0b1-f3082ae86210/manager/0.log"
Jul 28 11:53:14.948532 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:53:14.948502    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-8bdq9_6b5d8a26-4656-441f-8bfa-7579a7956ed0/manager/0.log"
Jul 28 11:53:15.117154 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:53:15.117121    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-2z85v_523cd2de-0f16-403c-be29-876f77b2b2b1/manager/0.log"
Jul 28 11:53:16.193132 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:53:16.193099    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-tlmfp_f1643f06-8d61-4ace-b0b1-f3082ae86210/manager/0.log"
Jul 28 11:53:16.206976 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:53:16.206944    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-8bdq9_6b5d8a26-4656-441f-8bfa-7579a7956ed0/manager/0.log"
Jul 28 11:53:16.377771 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:53:16.377737    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-2z85v_523cd2de-0f16-403c-be29-876f77b2b2b1/manager/0.log"
Jul 28 11:53:17.456037 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:53:17.456006    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-tlmfp_f1643f06-8d61-4ace-b0b1-f3082ae86210/manager/0.log"
Jul 28 11:53:17.474263 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:53:17.474239    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-8bdq9_6b5d8a26-4656-441f-8bfa-7579a7956ed0/manager/0.log"
Jul 28 11:53:17.641882 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:53:17.641840    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-2z85v_523cd2de-0f16-403c-be29-876f77b2b2b1/manager/0.log"
Jul 28 11:53:18.746997 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:53:18.746957    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-tlmfp_f1643f06-8d61-4ace-b0b1-f3082ae86210/manager/0.log"
Jul 28 11:53:18.762484 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:53:18.762455    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-8bdq9_6b5d8a26-4656-441f-8bfa-7579a7956ed0/manager/0.log"
Jul 28 11:53:18.934841 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:53:18.934811    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-2z85v_523cd2de-0f16-403c-be29-876f77b2b2b1/manager/0.log"
Jul 28 11:53:20.019555 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:53:20.019504    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-tlmfp_f1643f06-8d61-4ace-b0b1-f3082ae86210/manager/0.log"
Jul 28 11:53:20.034066 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:53:20.034037    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-8bdq9_6b5d8a26-4656-441f-8bfa-7579a7956ed0/manager/0.log"
Jul 28 11:53:20.201755 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:53:20.201720    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-2z85v_523cd2de-0f16-403c-be29-876f77b2b2b1/manager/0.log"
Jul 28 11:53:21.264987 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:53:21.264950    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-tlmfp_f1643f06-8d61-4ace-b0b1-f3082ae86210/manager/0.log"
Jul 28 11:53:21.291476 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:53:21.291446    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-8bdq9_6b5d8a26-4656-441f-8bfa-7579a7956ed0/manager/0.log"
Jul 28 11:53:21.477737 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:53:21.477703    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-2z85v_523cd2de-0f16-403c-be29-876f77b2b2b1/manager/0.log"
Jul 28 11:53:22.558738 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:53:22.558708    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-tlmfp_f1643f06-8d61-4ace-b0b1-f3082ae86210/manager/0.log"
Jul 28 11:53:22.574969 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:53:22.574943    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-8bdq9_6b5d8a26-4656-441f-8bfa-7579a7956ed0/manager/0.log"
Jul 28 11:53:22.745903 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:53:22.745867    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-2z85v_523cd2de-0f16-403c-be29-876f77b2b2b1/manager/0.log"
Jul 28 11:53:23.837278 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:53:23.837246    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-tlmfp_f1643f06-8d61-4ace-b0b1-f3082ae86210/manager/0.log"
Jul 28 11:53:23.863395 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:53:23.863365    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-8bdq9_6b5d8a26-4656-441f-8bfa-7579a7956ed0/manager/0.log"
Jul 28 11:53:24.054351 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:53:24.054311    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-2z85v_523cd2de-0f16-403c-be29-876f77b2b2b1/manager/0.log"
Jul 28 11:53:26.123169 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:53:26.123134    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:53:27.182926 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:53:27.182892    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:53:29.318962 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:53:29.318924    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-bffsx_1fb2411c-4da6-44e6-882e-4f36fb1a26ea/global-pull-secret-syncer/0.log"
Jul 28 11:53:29.446679 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:53:29.446647    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log"
Jul 28 11:53:29.672793 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:53:29.672716    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-141-50.ec2.internal_2efc845f7ae190bc5a73ce1b56253af6/haproxy/0.log"
Jul 28 11:53:31.963550 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:53:31.963513    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:53:32.023933 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:53:32.023898    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2w2gl_aff18ade-5ca4-40b1-bee1-a593dc42d38c/konnectivity-agent/0.log" limit=1024
Jul 28 11:53:34.457931 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:53:34.457905    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-tlmfp_f1643f06-8d61-4ace-b0b1-f3082ae86210/manager/0.log"
Jul 28 11:53:34.486610 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:53:34.486582    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-8bdq9_6b5d8a26-4656-441f-8bfa-7579a7956ed0/manager/0.log"
Jul 28 11:53:34.691997 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:53:34.691969    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-2z85v_523cd2de-0f16-403c-be29-876f77b2b2b1/manager/0.log"
Jul 28 11:53:36.253421 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:53:36.253387    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mttd2_382407f5-7905-49ae-b6df-e4458fd28bd6/node-exporter/0.log"
Jul 28 11:53:36.278254 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:53:36.278227    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mttd2_382407f5-7905-49ae-b6df-e4458fd28bd6/kube-rbac-proxy/0.log"
Jul 28 11:53:36.301494 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:53:36.301463    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mttd2_382407f5-7905-49ae-b6df-e4458fd28bd6/init-textfile/0.log"
Jul 28 11:53:40.515824 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:53:40.515795    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-x9gk6_69a2e04d-f363-4cfe-9dc0-61efcb33fc75/dns/0.log"
Jul 28 11:53:40.538890 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:53:40.538866    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-x9gk6_69a2e04d-f363-4cfe-9dc0-61efcb33fc75/kube-rbac-proxy/0.log"
Jul 28 11:53:40.563962 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:53:40.563930    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-2rf9h_ba296f38-5de8-4d11-b789-7a727a01b9cc/dns-node-resolver/0.log"
Jul 28 11:53:41.261972 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:53:41.261942    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-mkf4n_beb002bc-8a51-44cf-8274-fd83f8d1d237/node-ca/0.log"
Jul 28 11:53:42.457083 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:53:42.457050    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-w6tmc_348441b2-cf7f-43e6-9f8a-1d45d48301d2/discovery/0.log"
Jul 28 11:53:42.483680 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:53:42.483649    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-zf8mf_490ca628-de57-478e-b4b8-af249144d0d2/istio-proxy/0.log"
Jul 28 11:53:42.510613 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:53:42.510579    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-zf8mf_490ca628-de57-478e-b4b8-af249144d0d2/istio-proxy/1.log"
Jul 28 11:53:43.277337 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:53:43.277306    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-2qvgv_985feeac-ca2d-40aa-8ee6-53797559bc72/serve-healthcheck-canary/0.log"
Jul 28 11:53:44.278803 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:53:44.278777    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-mtcd6_3fd0cb5a-eb68-45ec-89f2-df8fcf761def/kube-rbac-proxy/0.log"
Jul 28 11:53:44.305482 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:53:44.305404    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-mtcd6_3fd0cb5a-eb68-45ec-89f2-df8fcf761def/exporter/0.log"
Jul 28 11:53:44.331833 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:53:44.331808    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-mtcd6_3fd0cb5a-eb68-45ec-89f2-df8fcf761def/extractor/0.log"
Jul 28 11:53:45.314054 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:53:45.314011    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-wxwvv/perf-node-gather-daemonset-22bxt"]
Jul 28 11:53:45.314587 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:53:45.314527    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8107ec9e-840d-40d3-8355-aede968b720c" containerName="main"
Jul 28 11:53:45.314587 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:53:45.314567    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="8107ec9e-840d-40d3-8355-aede968b720c" containerName="main"
Jul 28 11:53:45.314719 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:53:45.314667    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="8107ec9e-840d-40d3-8355-aede968b720c" containerName="main"
Jul 28 11:53:45.317802 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:53:45.317779    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wxwvv/perf-node-gather-daemonset-22bxt"
Jul 28 11:53:45.319677 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:53:45.319651    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-wxwvv\"/\"kube-root-ca.crt\""
Jul 28 11:53:45.319779 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:53:45.319699    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-wxwvv\"/\"openshift-service-ca.crt\""
Jul 28 11:53:45.320289 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:53:45.320274    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-wxwvv\"/\"default-dockercfg-crk46\""
Jul 28 11:53:45.325290 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:53:45.325262    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wxwvv/perf-node-gather-daemonset-22bxt"]
Jul 28 11:53:45.390657 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:53:45.390612    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f59c85d1-a4ef-4a69-8766-fc670c61e4fe-podres\") pod \"perf-node-gather-daemonset-22bxt\" (UID: \"f59c85d1-a4ef-4a69-8766-fc670c61e4fe\") " pod="openshift-must-gather-wxwvv/perf-node-gather-daemonset-22bxt"
Jul 28 11:53:45.390657 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:53:45.390663    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f59c85d1-a4ef-4a69-8766-fc670c61e4fe-lib-modules\") pod \"perf-node-gather-daemonset-22bxt\" (UID: \"f59c85d1-a4ef-4a69-8766-fc670c61e4fe\") " pod="openshift-must-gather-wxwvv/perf-node-gather-daemonset-22bxt"
Jul 28 11:53:45.390945 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:53:45.390713    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pqxj\" (UniqueName: \"kubernetes.io/projected/f59c85d1-a4ef-4a69-8766-fc670c61e4fe-kube-api-access-6pqxj\") pod \"perf-node-gather-daemonset-22bxt\" (UID: \"f59c85d1-a4ef-4a69-8766-fc670c61e4fe\") " pod="openshift-must-gather-wxwvv/perf-node-gather-daemonset-22bxt"
Jul 28 11:53:45.390945 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:53:45.390794    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f59c85d1-a4ef-4a69-8766-fc670c61e4fe-proc\") pod \"perf-node-gather-daemonset-22bxt\" (UID: \"f59c85d1-a4ef-4a69-8766-fc670c61e4fe\") " pod="openshift-must-gather-wxwvv/perf-node-gather-daemonset-22bxt"
Jul 28 11:53:45.390945 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:53:45.390856    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f59c85d1-a4ef-4a69-8766-fc670c61e4fe-sys\") pod \"perf-node-gather-daemonset-22bxt\" (UID: \"f59c85d1-a4ef-4a69-8766-fc670c61e4fe\") " pod="openshift-must-gather-wxwvv/perf-node-gather-daemonset-22bxt"
Jul 28 11:53:45.491886 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:53:45.491847    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f59c85d1-a4ef-4a69-8766-fc670c61e4fe-podres\") pod \"perf-node-gather-daemonset-22bxt\" (UID: \"f59c85d1-a4ef-4a69-8766-fc670c61e4fe\") " pod="openshift-must-gather-wxwvv/perf-node-gather-daemonset-22bxt"
Jul 28 11:53:45.492099 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:53:45.491902    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f59c85d1-a4ef-4a69-8766-fc670c61e4fe-lib-modules\") pod \"perf-node-gather-daemonset-22bxt\" (UID: \"f59c85d1-a4ef-4a69-8766-fc670c61e4fe\") " pod="openshift-must-gather-wxwvv/perf-node-gather-daemonset-22bxt"
Jul 28 11:53:45.492099 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:53:45.491932    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pqxj\" (UniqueName: \"kubernetes.io/projected/f59c85d1-a4ef-4a69-8766-fc670c61e4fe-kube-api-access-6pqxj\") pod \"perf-node-gather-daemonset-22bxt\" (UID: \"f59c85d1-a4ef-4a69-8766-fc670c61e4fe\") " pod="openshift-must-gather-wxwvv/perf-node-gather-daemonset-22bxt"
Jul 28 11:53:45.492099 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:53:45.491963    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f59c85d1-a4ef-4a69-8766-fc670c61e4fe-proc\") pod \"perf-node-gather-daemonset-22bxt\" (UID: \"f59c85d1-a4ef-4a69-8766-fc670c61e4fe\") " pod="openshift-must-gather-wxwvv/perf-node-gather-daemonset-22bxt"
Jul 28 11:53:45.492099 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:53:45.492004    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f59c85d1-a4ef-4a69-8766-fc670c61e4fe-sys\") pod \"perf-node-gather-daemonset-22bxt\" (UID: \"f59c85d1-a4ef-4a69-8766-fc670c61e4fe\") " pod="openshift-must-gather-wxwvv/perf-node-gather-daemonset-22bxt"
Jul 28 11:53:45.492099 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:53:45.492053    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f59c85d1-a4ef-4a69-8766-fc670c61e4fe-podres\") pod \"perf-node-gather-daemonset-22bxt\" (UID: \"f59c85d1-a4ef-4a69-8766-fc670c61e4fe\") " pod="openshift-must-gather-wxwvv/perf-node-gather-daemonset-22bxt"
Jul 28 11:53:45.492099 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:53:45.492070    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f59c85d1-a4ef-4a69-8766-fc670c61e4fe-lib-modules\") pod \"perf-node-gather-daemonset-22bxt\" (UID: \"f59c85d1-a4ef-4a69-8766-fc670c61e4fe\") " pod="openshift-must-gather-wxwvv/perf-node-gather-daemonset-22bxt"
Jul 28 11:53:45.492329 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:53:45.492097    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f59c85d1-a4ef-4a69-8766-fc670c61e4fe-sys\") pod \"perf-node-gather-daemonset-22bxt\" (UID: \"f59c85d1-a4ef-4a69-8766-fc670c61e4fe\") " pod="openshift-must-gather-wxwvv/perf-node-gather-daemonset-22bxt"
Jul 28 11:53:45.492329 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:53:45.492118    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f59c85d1-a4ef-4a69-8766-fc670c61e4fe-proc\") pod \"perf-node-gather-daemonset-22bxt\" (UID: \"f59c85d1-a4ef-4a69-8766-fc670c61e4fe\") " pod="openshift-must-gather-wxwvv/perf-node-gather-daemonset-22bxt"
Jul 28 11:53:45.500658 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:53:45.500624    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pqxj\" (UniqueName: \"kubernetes.io/projected/f59c85d1-a4ef-4a69-8766-fc670c61e4fe-kube-api-access-6pqxj\") pod \"perf-node-gather-daemonset-22bxt\" (UID: \"f59c85d1-a4ef-4a69-8766-fc670c61e4fe\") " pod="openshift-must-gather-wxwvv/perf-node-gather-daemonset-22bxt"
Jul 28 11:53:45.629431 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:53:45.629338    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wxwvv/perf-node-gather-daemonset-22bxt"
Jul 28 11:53:45.760896 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:53:45.760862    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wxwvv/perf-node-gather-daemonset-22bxt"]
Jul 28 11:53:45.762679 ip-10-0-141-50 kubenswrapper[2579]: W0728 11:53:45.762632    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podf59c85d1_a4ef_4a69_8766_fc670c61e4fe.slice/crio-9d7658608f298e79cb8ee948fad609777eab9cc107ac8cdb90acda0880156d6b WatchSource:0}: Error finding container 9d7658608f298e79cb8ee948fad609777eab9cc107ac8cdb90acda0880156d6b: Status 404 returned error can't find the container with id 9d7658608f298e79cb8ee948fad609777eab9cc107ac8cdb90acda0880156d6b
Jul 28 11:53:46.633688 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:53:46.633648    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wxwvv/perf-node-gather-daemonset-22bxt" event={"ID":"f59c85d1-a4ef-4a69-8766-fc670c61e4fe","Type":"ContainerStarted","Data":"a989d0deb6bb8f05a59399abc341c23a25adc122375c47827c70f6460fe5cd1d"}
Jul 28 11:53:46.634104 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:53:46.633741    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wxwvv/perf-node-gather-daemonset-22bxt" event={"ID":"f59c85d1-a4ef-4a69-8766-fc670c61e4fe","Type":"ContainerStarted","Data":"9d7658608f298e79cb8ee948fad609777eab9cc107ac8cdb90acda0880156d6b"}
Jul 28 11:53:46.634104 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:53:46.633778    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-wxwvv/perf-node-gather-daemonset-22bxt"
Jul 28 11:53:46.649856 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:53:46.649806    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-wxwvv/perf-node-gather-daemonset-22bxt" podStartSLOduration=1.649791658 podStartE2EDuration="1.649791658s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 11:53:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 11:53:46.647641413 +0000 UTC m=+6360.911037743" watchObservedRunningTime="2026-07-28 11:53:46.649791658 +0000 UTC m=+6360.913187986"
Jul 28 11:53:48.098452 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:53:48.098422    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-6968749498-brk56_6c21258e-66df-4a31-a1b4-8d10fcb6ea5d/manager/0.log"
Jul 28 11:53:48.132687 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:53:48.132658    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-6fdfcc6654-ltdmm_306f4879-2b49-4fe6-a4ee-f86e354c7017/manager/0.log"
Jul 28 11:53:48.154107 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:53:48.154072    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-7b954ffc95-tr22l_41f55d40-f073-4037-bde6-fd85c2e58cf1/server/0.log"
Jul 28 11:53:48.775672 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:53:48.775633    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-75d9cdbddf-vgnpz_f4b1f552-5478-4e5e-8fd5-6659b9b61ba2/manager/0.log"
Jul 28 11:53:48.809799 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:53:48.809761    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-vs6wx_9aad1d78-7972-42e1-b53b-a7fc64944590/s3-init/0.log"
Jul 28 11:53:48.827368 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:53:48.827342    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-vs6wx_9aad1d78-7972-42e1-b53b-a7fc64944590/download-hf-model/0.log"
Jul 28 11:53:48.855255 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:53:48.855223    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-dk5fj_ec836d14-324b-4bfc-979f-dd6967b5603a/seaweedfs/0.log"
Jul 28 11:53:52.648411 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:53:52.648377    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-wxwvv/perf-node-gather-daemonset-22bxt"
Jul 28 11:53:53.773522 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:53:53.773494    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-wdgll_e1e132cc-95bb-466a-a573-f40cf5cf8e60/migrator/0.log"
Jul 28 11:53:53.796046 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:53:53.796008    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-wdgll_e1e132cc-95bb-466a-a573-f40cf5cf8e60/graceful-termination/0.log"
Jul 28 11:53:55.778056 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:53:55.778030    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fftp9_4f0d1f6f-a98f-4c02-a856-2465e90ce76e/kube-multus-additional-cni-plugins/0.log"
Jul 28 11:53:55.802996 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:53:55.802922    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fftp9_4f0d1f6f-a98f-4c02-a856-2465e90ce76e/egress-router-binary-copy/0.log"
Jul 28 11:53:55.826980 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:53:55.826955    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fftp9_4f0d1f6f-a98f-4c02-a856-2465e90ce76e/cni-plugins/0.log"
Jul 28 11:53:55.852703 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:53:55.852682    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fftp9_4f0d1f6f-a98f-4c02-a856-2465e90ce76e/bond-cni-plugin/0.log"
Jul 28 11:53:55.878395 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:53:55.878367    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fftp9_4f0d1f6f-a98f-4c02-a856-2465e90ce76e/routeoverride-cni/0.log"
Jul 28 11:53:55.904451 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:53:55.904420    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fftp9_4f0d1f6f-a98f-4c02-a856-2465e90ce76e/whereabouts-cni-bincopy/0.log"
Jul 28 11:53:55.930830 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:53:55.930806    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fftp9_4f0d1f6f-a98f-4c02-a856-2465e90ce76e/whereabouts-cni/0.log"
Jul 28 11:53:56.174796 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:53:56.174718    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-gtxhl_1d9e06b8-3c8a-41bd-8368-435c66ba1a52/kube-multus/0.log"
Jul 28 11:53:56.372864 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:53:56.372837    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-vtxgv_d67ab7ea-ff97-432b-b10a-df89bb204785/network-metrics-daemon/0.log"
Jul 28 11:53:56.399639 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:53:56.399614    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-vtxgv_d67ab7ea-ff97-432b-b10a-df89bb204785/kube-rbac-proxy/0.log"
Jul 28 11:53:56.755406 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:53:56.755376    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-w6tmc_348441b2-cf7f-43e6-9f8a-1d45d48301d2/discovery/0.log"
Jul 28 11:53:56.771567 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:53:56.771518    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-zf8mf_490ca628-de57-478e-b4b8-af249144d0d2/istio-proxy/0.log"
Jul 28 11:53:56.799421 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:53:56.799387    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-zf8mf_490ca628-de57-478e-b4b8-af249144d0d2/istio-proxy/1.log"
Jul 28 11:53:57.729820 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:53:57.729795    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cjlcq_f237d44f-f0d7-4d96-82b1-6a75f8c22704/ovn-controller/0.log"
Jul 28 11:53:57.746915 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:53:57.746884    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cjlcq_f237d44f-f0d7-4d96-82b1-6a75f8c22704/ovn-acl-logging/0.log"
Jul 28 11:53:57.772645 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:53:57.772612    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cjlcq_f237d44f-f0d7-4d96-82b1-6a75f8c22704/kube-rbac-proxy-node/0.log"
Jul 28 11:53:57.794591 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:53:57.794566    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cjlcq_f237d44f-f0d7-4d96-82b1-6a75f8c22704/kube-rbac-proxy-ovn-metrics/0.log"
Jul 28 11:53:57.816884 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:53:57.816845    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cjlcq_f237d44f-f0d7-4d96-82b1-6a75f8c22704/northd/0.log"
Jul 28 11:53:57.839978 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:53:57.839953    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cjlcq_f237d44f-f0d7-4d96-82b1-6a75f8c22704/nbdb/0.log"
Jul 28 11:53:57.865345 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:53:57.865319    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cjlcq_f237d44f-f0d7-4d96-82b1-6a75f8c22704/sbdb/0.log"
Jul 28 11:53:57.954456 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:53:57.954425    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cjlcq_f237d44f-f0d7-4d96-82b1-6a75f8c22704/ovnkube-controller/0.log"
Jul 28 11:53:58.073376 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:53:58.073340    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-w6tmc_348441b2-cf7f-43e6-9f8a-1d45d48301d2/discovery/0.log"
Jul 28 11:53:58.094207 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:53:58.094178    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-zf8mf_490ca628-de57-478e-b4b8-af249144d0d2/istio-proxy/0.log"
Jul 28 11:53:58.121587 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:53:58.121560    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-zf8mf_490ca628-de57-478e-b4b8-af249144d0d2/istio-proxy/1.log"
Jul 28 11:53:59.744443 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:53:59.744416    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-4msch_92436f2b-eb78-4e84-ab81-be7d1102f5ff/network-check-target-container/0.log"
Jul 28 11:54:00.989641 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:54:00.989607    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-whcvk_b00fc78c-64e8-4700-ba67-06a656c2d399/iptables-alerter/0.log"
Jul 28 11:54:01.792094 ip-10-0-141-50 kubenswrapper[2579]: I0728 11:54:01.792061    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-lcrsz_20a3d596-59ed-46ce-b8f2-a9be7dc51b41/tuned/0.log"