Sep 09 20:50:53.378315 ip-10-0-134-72 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 09 20:50:53.378329 ip-10-0-134-72 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 09 20:50:53.378338 ip-10-0-134-72 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 09 20:50:53.378650 ip-10-0-134-72 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 09 20:51:02.980663 ip-10-0-134-72 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 09 20:51:02.980678 ip-10-0-134-72 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot e6a02596136f4390bde7535f9fd4de19 --
Sep 09 20:52:36.769698 ip-10-0-134-72 systemd[1]: Starting Kubernetes Kubelet...
Sep 09 20:52:37.194248 ip-10-0-134-72 kubenswrapper[2576]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 20:52:37.194248 ip-10-0-134-72 kubenswrapper[2576]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 09 20:52:37.194248 ip-10-0-134-72 kubenswrapper[2576]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 20:52:37.194248 ip-10-0-134-72 kubenswrapper[2576]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 09 20:52:37.194248 ip-10-0-134-72 kubenswrapper[2576]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 20:52:37.195819 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.195734    2576 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 09 20:52:37.199761 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.199745    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 20:52:37.199761 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.199761    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 20:52:37.199856 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.199765    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 20:52:37.199856 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.199768    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 20:52:37.199856 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.199777    2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 20:52:37.199856 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.199780    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 20:52:37.199856 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.199783    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 20:52:37.199856 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.199786    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 20:52:37.199856 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.199788    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 20:52:37.199856 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.199791    2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 20:52:37.199856 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.199793    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 20:52:37.199856 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.199796    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 20:52:37.199856 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.199798    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 20:52:37.199856 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.199801    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 20:52:37.199856 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.199805    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 20:52:37.199856 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.199808    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 20:52:37.199856 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.199810    2576 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 20:52:37.199856 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.199813    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 20:52:37.199856 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.199815    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 20:52:37.199856 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.199818    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 20:52:37.199856 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.199820    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 20:52:37.199856 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.199824    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 20:52:37.200324 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.199826    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 20:52:37.200324 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.199829    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 20:52:37.200324 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.199831    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 20:52:37.200324 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.199834    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 20:52:37.200324 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.199836    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 20:52:37.200324 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.199839    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 20:52:37.200324 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.199842    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 20:52:37.200324 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.199844    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 20:52:37.200324 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.199847    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 20:52:37.200324 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.199849    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 20:52:37.200324 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.199852    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 20:52:37.200324 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.199854    2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 20:52:37.200324 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.199857    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 20:52:37.200324 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.199860    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 20:52:37.200324 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.199863    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 20:52:37.200324 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.199866    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 20:52:37.200324 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.199868    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 20:52:37.200324 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.199871    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 20:52:37.200324 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.199874    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 20:52:37.200324 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.199876    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 20:52:37.200822 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.199879    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 20:52:37.200822 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.199882    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 20:52:37.200822 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.199884    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 20:52:37.200822 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.199887    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 20:52:37.200822 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.199889    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 20:52:37.200822 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.199892    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 20:52:37.200822 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.199896    2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 20:52:37.200822 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.199900    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 20:52:37.200822 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.199904    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 20:52:37.200822 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.199907    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 20:52:37.200822 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.199910    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 20:52:37.200822 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.199912    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 20:52:37.200822 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.199915    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 20:52:37.200822 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.199918    2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 20:52:37.200822 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.199921    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 20:52:37.200822 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.199924    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 20:52:37.200822 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.199927    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 20:52:37.200822 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.199930    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 20:52:37.200822 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.199934    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 20:52:37.201293 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.199936    2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 20:52:37.201293 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.199944    2576 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 20:52:37.201293 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.199948    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 20:52:37.201293 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.199950    2576 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 20:52:37.201293 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.199953    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 20:52:37.201293 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.199956    2576 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 20:52:37.201293 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.199958    2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 20:52:37.201293 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.199960    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 20:52:37.201293 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.199963    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 20:52:37.201293 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.199965    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 20:52:37.201293 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.199969    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 20:52:37.201293 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.199972    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 20:52:37.201293 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.199974    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 20:52:37.201293 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.199977    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 20:52:37.201293 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.199979    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 20:52:37.201293 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.199982    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 20:52:37.201293 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.199985    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 20:52:37.201293 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.199989    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 20:52:37.201293 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.199992    2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 20:52:37.201293 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.199994    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 20:52:37.201790 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.199997    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 20:52:37.201790 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.199999    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 20:52:37.201790 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.200001    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 20:52:37.201790 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.200004    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 20:52:37.201790 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.200006    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 20:52:37.201790 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.200009    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 20:52:37.201790 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.200012    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 20:52:37.201790 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.200015    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 20:52:37.201790 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.200017    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 20:52:37.201790 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.200019    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 20:52:37.201790 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.200022    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 20:52:37.201790 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.200025    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 20:52:37.201790 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.200027    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 20:52:37.201790 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.200030    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 20:52:37.201790 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.200035    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 20:52:37.201790 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.200038    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 20:52:37.201790 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.200041    2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 20:52:37.201790 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.200043    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 20:52:37.201790 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.200838    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 20:52:37.202346 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.200844    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 20:52:37.202346 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.200847    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 20:52:37.202346 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.200853    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 20:52:37.202346 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.200857    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 20:52:37.202346 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.200860    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 20:52:37.202346 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.200863    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 20:52:37.202346 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.200866    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 20:52:37.202346 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.200869    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 20:52:37.202346 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.200872    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 20:52:37.202346 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.200875    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 20:52:37.202346 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.200879    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 20:52:37.202346 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.200881    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 20:52:37.202346 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.200884    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 20:52:37.202346 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.200887    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 20:52:37.202346 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.200890    2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 20:52:37.202346 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.200892    2576 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 20:52:37.202346 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.200895    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 20:52:37.202346 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.200898    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 20:52:37.202346 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.200900    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 20:52:37.202889 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.200903    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 20:52:37.202889 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.200906    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 20:52:37.202889 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.200908    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 20:52:37.202889 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.200910    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 20:52:37.202889 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.200914    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 20:52:37.202889 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.200917    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 20:52:37.202889 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.200919    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 20:52:37.202889 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.200922    2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 20:52:37.202889 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.200925    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 20:52:37.202889 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.200937    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 20:52:37.202889 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.200940    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 20:52:37.202889 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.200942    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 20:52:37.202889 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.200945    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 20:52:37.202889 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.200947    2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 20:52:37.202889 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.200950    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 20:52:37.202889 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.200952    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 20:52:37.202889 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.200955    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 20:52:37.202889 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.200957    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 20:52:37.202889 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.200959    2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 20:52:37.202889 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.200962    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 20:52:37.203376 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.200964    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 20:52:37.203376 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.200967    2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 20:52:37.203376 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.200969    2576 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 20:52:37.203376 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.200972    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 20:52:37.203376 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.200974    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 20:52:37.203376 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.200977    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 20:52:37.203376 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.200979    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 20:52:37.203376 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.200982    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 20:52:37.203376 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.200984    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 20:52:37.203376 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.200986    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 20:52:37.203376 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.200989    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 20:52:37.203376 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.200991    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 20:52:37.203376 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.200994    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 20:52:37.203376 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.200998    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 20:52:37.203376 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.201000    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 20:52:37.203376 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.201002    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 20:52:37.203376 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.201006    2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 20:52:37.203376 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.201008    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 20:52:37.203376 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.201011    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 20:52:37.203376 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.201014    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 20:52:37.203864 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.201016    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 20:52:37.203864 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.201019    2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 20:52:37.203864 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.201027    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 20:52:37.203864 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.201029    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 20:52:37.203864 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.201031    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 20:52:37.203864 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.201034    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 20:52:37.203864 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.201037    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 20:52:37.203864 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.201039    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 20:52:37.203864 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.201042    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 20:52:37.203864 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.201044    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 20:52:37.203864 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.201047    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 20:52:37.203864 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.201049    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 20:52:37.203864 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.201052    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 20:52:37.203864 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.201055    2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 20:52:37.203864 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.201057    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 20:52:37.203864 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.201060    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 20:52:37.203864 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.201063    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 20:52:37.203864 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.201065    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 20:52:37.203864 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.201068    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 20:52:37.203864 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.201070    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 20:52:37.204355 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.201073    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 20:52:37.204355 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.201076    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 20:52:37.204355 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.201078    2576 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 20:52:37.204355 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.201081    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 20:52:37.204355 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.201083    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 20:52:37.204355 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.201087    2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 20:52:37.204355 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.201090    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 20:52:37.204355 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.201093    2576 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 20:52:37.204355 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.201096    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 20:52:37.204355 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.201098    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 20:52:37.204355 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.201101    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 20:52:37.204355 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.201104    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 20:52:37.204355 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.201106    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 20:52:37.204355 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.201108    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 20:52:37.204355 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.201111    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 20:52:37.204355 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.201119    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 20:52:37.204355 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.201122    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 20:52:37.204355 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.201125    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 20:52:37.204355 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.201129    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 20:52:37.204833 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.202102    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 20:52:37.204833 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.202108    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 20:52:37.204833 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.202112    2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 20:52:37.204833 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.202115    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 20:52:37.204833 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.202118    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 20:52:37.204833 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.202121    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 20:52:37.204833 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.202124    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 20:52:37.204833 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.202126    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 20:52:37.204833 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.202131    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 20:52:37.204833 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.202134    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 20:52:37.204833 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.202137    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 20:52:37.204833 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.202141    2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 20:52:37.204833 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.202144    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 20:52:37.204833 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.202147    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 20:52:37.204833 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.202150    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 20:52:37.204833 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.202153    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 20:52:37.204833 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.202155    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 20:52:37.204833 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.202158    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 20:52:37.204833 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.202160    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 20:52:37.205287 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.202163    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 20:52:37.205287 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.202165    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 20:52:37.205287 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.202168    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 20:52:37.205287 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.202170    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 20:52:37.205287 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.202173    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 20:52:37.205287 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.202175    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 20:52:37.205287 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.202178    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 20:52:37.205287 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.202183    2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 20:52:37.205287 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.202185    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 20:52:37.205287 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.202194    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 20:52:37.205287 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.202197    2576 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 20:52:37.205287 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.202200    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 20:52:37.205287 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.202202    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 20:52:37.205287 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.202205    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 20:52:37.205287 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.202207    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 20:52:37.205287 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.202211    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 20:52:37.205287 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.202213    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 20:52:37.205287 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.202215    2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 20:52:37.205287 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.202218    2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 20:52:37.205287 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.202220    2576 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 20:52:37.205794 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.202222    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 20:52:37.205794 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.202225    2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 20:52:37.205794 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.202227    2576 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 20:52:37.205794 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.202230    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 20:52:37.205794 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.202232    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 20:52:37.205794 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.202234    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 20:52:37.205794 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.202237    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 20:52:37.205794 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.202239    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 20:52:37.205794 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.202241    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 20:52:37.205794 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.202244    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 20:52:37.205794 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.202246    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 20:52:37.205794 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.202249    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 20:52:37.205794 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.202251    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 20:52:37.205794 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.202254    2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 20:52:37.205794 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.202256    2576 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 20:52:37.205794 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.202259    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 20:52:37.205794 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.202261    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 20:52:37.205794 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.202264    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 20:52:37.205794 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.202266    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 20:52:37.205794 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.202270    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 20:52:37.206272 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.202273    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 20:52:37.206272 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.202275    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 20:52:37.206272 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.202283    2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 20:52:37.206272 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.202286    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 20:52:37.206272 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.202288    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 20:52:37.206272 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.202290    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 20:52:37.206272 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.202293    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 20:52:37.206272 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.202295    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 20:52:37.206272 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.202297    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 20:52:37.206272 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.202300    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 20:52:37.206272 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.202302    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 20:52:37.206272 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.202305    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 20:52:37.206272 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.202307    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 20:52:37.206272 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.202309    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 20:52:37.206272 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.202312    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 20:52:37.206272 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.202315    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 20:52:37.206272 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.202318    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 20:52:37.206272 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.202321    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 20:52:37.206272 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.202324    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 20:52:37.206272 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.202326    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 20:52:37.206775 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.202329    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 20:52:37.206775 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.202331    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 20:52:37.206775 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.202334    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 20:52:37.206775 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.202336    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 20:52:37.206775 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.202339    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 20:52:37.206775 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.202341    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 20:52:37.206775 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.202344    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 20:52:37.206775 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.202346    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 20:52:37.206775 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.202349    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 20:52:37.206775 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.202352    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 20:52:37.206775 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.202354    2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 20:52:37.206775 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.202358    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 20:52:37.206775 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.202373    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 20:52:37.206775 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.202376    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 20:52:37.206775 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.202379    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 20:52:37.206775 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.202389    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 20:52:37.206775 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.202391    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 20:52:37.206775 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.202394    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 20:52:37.206775 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.202396    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 20:52:37.207233 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.202399    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 20:52:37.207233 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.202474    2576 flags.go:64] FLAG: --address="0.0.0.0"
Sep 09 20:52:37.207233 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.202509    2576 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 09 20:52:37.207233 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.202525    2576 flags.go:64] FLAG: --anonymous-auth="true"
Sep 09 20:52:37.207233 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.202531    2576 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 09 20:52:37.207233 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.202535    2576 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 09 20:52:37.207233 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.202538    2576 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 09 20:52:37.207233 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.202542    2576 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 09 20:52:37.207233 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.202547    2576 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 09 20:52:37.207233 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.202550    2576 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 09 20:52:37.207233 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.202553    2576 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 09 20:52:37.207233 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.202556    2576 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 09 20:52:37.207233 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.202559    2576 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 09 20:52:37.207233 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.202562    2576 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 09 20:52:37.207233 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.202565    2576 flags.go:64] FLAG: --cgroup-root=""
Sep 09 20:52:37.207233 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.202568    2576 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 09 20:52:37.207233 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.202571    2576 flags.go:64] FLAG: --client-ca-file=""
Sep 09 20:52:37.207233 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.202574    2576 flags.go:64] FLAG: --cloud-provider="external"
Sep 09 20:52:37.207233 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.202576    2576 flags.go:64] FLAG: --cluster-dns="[]"
Sep 09 20:52:37.207233 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.202589    2576 flags.go:64] FLAG: --cluster-domain=""
Sep 09 20:52:37.207233 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.202592    2576 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 09 20:52:37.207233 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.202595    2576 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 09 20:52:37.207233 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.202598    2576 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 09 20:52:37.207233 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.202601    2576 flags.go:64] FLAG: --container-log-max-files="5"
Sep 09 20:52:37.207839 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.202607    2576 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 09 20:52:37.207839 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.202610    2576 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 09 20:52:37.207839 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.202613    2576 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 09 20:52:37.207839 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.202616    2576 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 09 20:52:37.207839 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.202619    2576 flags.go:64] FLAG: --contention-profiling="false"
Sep 09 20:52:37.207839 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.202628    2576 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 09 20:52:37.207839 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.202631    2576 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 09 20:52:37.207839 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.202634    2576 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 09 20:52:37.207839 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.202637    2576 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 09 20:52:37.207839 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.202640    2576 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 09 20:52:37.207839 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.202643    2576 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 09 20:52:37.207839 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.202646    2576 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 09 20:52:37.207839 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.202649    2576 flags.go:64] FLAG: --enable-load-reader="false"
Sep 09 20:52:37.207839 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.202652    2576 flags.go:64] FLAG: --enable-server="true"
Sep 09 20:52:37.207839 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.202654    2576 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 09 20:52:37.207839 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.202661    2576 flags.go:64] FLAG: --event-burst="100"
Sep 09 20:52:37.207839 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.202664    2576 flags.go:64] FLAG: --event-qps="50"
Sep 09 20:52:37.207839 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.202667    2576 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 09 20:52:37.207839 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.202670    2576 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 09 20:52:37.207839 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.202673    2576 flags.go:64] FLAG: --eviction-hard=""
Sep 09 20:52:37.207839 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.202676    2576 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 09 20:52:37.207839 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.202679    2576 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 09 20:52:37.207839 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.202682    2576 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 09 20:52:37.207839 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.202684    2576 flags.go:64] FLAG: --eviction-soft=""
Sep 09 20:52:37.207839 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.202687    2576 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 09 20:52:37.208484 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.202690    2576 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 09 20:52:37.208484 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.202693    2576 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 09 20:52:37.208484 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.202695    2576 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 09 20:52:37.208484 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.202698    2576 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 09 20:52:37.208484 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.202701    2576 flags.go:64] FLAG: --fail-swap-on="true"
Sep 09 20:52:37.208484 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.202704    2576 flags.go:64] FLAG: --feature-gates=""
Sep 09 20:52:37.208484 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.202707    2576 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 09 20:52:37.208484 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.202711    2576 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 09 20:52:37.208484 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.202714    2576 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 09 20:52:37.208484 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.202717    2576 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 09 20:52:37.208484 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.202720    2576 flags.go:64] FLAG: --healthz-port="10248"
Sep 09 20:52:37.208484 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.202723    2576 flags.go:64] FLAG: --help="false"
Sep 09 20:52:37.208484 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.202726    2576 flags.go:64] FLAG: --hostname-override="ip-10-0-134-72.ec2.internal"
Sep 09 20:52:37.208484 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.202734    2576 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 09 20:52:37.208484 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.202737    2576 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 09 20:52:37.208484 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.202740    2576 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 09 20:52:37.208484 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.202751    2576 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 09 20:52:37.208484 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.202754    2576 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 09 20:52:37.208484 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.202757    2576 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 09 20:52:37.208484 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.202760    2576 flags.go:64] FLAG: --image-service-endpoint=""
Sep 09 20:52:37.208484 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.202762    2576 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 09 20:52:37.208484 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.202765    2576 flags.go:64] FLAG: --kube-api-burst="100"
Sep 09 20:52:37.208484 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.202768    2576 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 09 20:52:37.209060 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.202771    2576 flags.go:64] FLAG: --kube-api-qps="50"
Sep 09 20:52:37.209060 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.202773    2576 flags.go:64] FLAG: --kube-reserved=""
Sep 09 20:52:37.209060 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.202776    2576 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 09 20:52:37.209060 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.202780    2576 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 09 20:52:37.209060 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.202783    2576 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 09 20:52:37.209060 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.202786    2576 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 09 20:52:37.209060 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.202789    2576 flags.go:64] FLAG: --lock-file=""
Sep 09 20:52:37.209060 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.202791    2576 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 09 20:52:37.209060 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.202794    2576 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 09 20:52:37.209060 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.202797    2576 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 09 20:52:37.209060 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.202803    2576 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 09 20:52:37.209060 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.202805    2576 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 09 20:52:37.209060 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.202808    2576 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 09 20:52:37.209060 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.202811    2576 flags.go:64] FLAG: --logging-format="text"
Sep 09 20:52:37.209060 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.202814    2576 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 09 20:52:37.209060 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.202817    2576 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 09 20:52:37.209060 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.202821    2576 flags.go:64] FLAG: --manifest-url=""
Sep 09 20:52:37.209060 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.202824    2576 flags.go:64] FLAG: --manifest-url-header=""
Sep 09 20:52:37.209060 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.202828    2576 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 09 20:52:37.209060 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.202831    2576 flags.go:64] FLAG: --max-open-files="1000000"
Sep 09 20:52:37.209060 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.202835    2576 flags.go:64] FLAG: --max-pods="110"
Sep 09 20:52:37.209060 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.202838    2576 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 09 20:52:37.209060 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.202841    2576 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 09 20:52:37.209060 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.202849    2576 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 09 20:52:37.209060 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.202852    2576 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 09 20:52:37.209687 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.202854    2576 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 09 20:52:37.209687 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.202857    2576 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 09 20:52:37.209687 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.202860    2576 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 09 20:52:37.209687 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.202867    2576 flags.go:64] FLAG: --node-status-max-images="50"
Sep 09 20:52:37.209687 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.202870    2576 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 09 20:52:37.209687 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.202873    2576 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 09 20:52:37.209687 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.202876    2576 flags.go:64] FLAG: --pod-cidr=""
Sep 09 20:52:37.209687 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.202879    2576 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1"
Sep 09 20:52:37.209687 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.202883    2576 flags.go:64] FLAG: --pod-manifest-path=""
Sep 09 20:52:37.209687 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.202886    2576 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 09 20:52:37.209687 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.202889    2576 flags.go:64] FLAG: --pods-per-core="0"
Sep 09 20:52:37.209687 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.202893    2576 flags.go:64] FLAG: --port="10250"
Sep 09 20:52:37.209687 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.202896    2576 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 09 20:52:37.209687 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.202899    2576 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-00a35d16b9f49f0b5"
Sep 09 20:52:37.209687 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.202902    2576 flags.go:64] FLAG: --qos-reserved=""
Sep 09 20:52:37.209687 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.202905    2576 flags.go:64] FLAG: --read-only-port="10255"
Sep 09 20:52:37.209687 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.202908    2576 flags.go:64] FLAG: --register-node="true"
Sep 09 20:52:37.209687 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.202911    2576 flags.go:64] FLAG: --register-with-taints=""
Sep 09 20:52:37.209687 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.202915    2576 flags.go:64] FLAG: --registry-burst="10"
Sep 09 20:52:37.209687 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.202917    2576 flags.go:64] FLAG: --registry-qps="5"
Sep 09 20:52:37.209687 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.202920    2576 flags.go:64] FLAG: --reserved-cpus=""
Sep 09 20:52:37.209687 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.202923    2576 flags.go:64] FLAG: --reserved-memory=""
Sep 09 20:52:37.209687 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.202926    2576 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 09 20:52:37.209687 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.202930    2576 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 09 20:52:37.209687 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.202933    2576 flags.go:64] FLAG: --rotate-certificates="false"
Sep 09 20:52:37.210312 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.202936    2576 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 09 20:52:37.210312 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.202939    2576 flags.go:64] FLAG: --runonce="false"
Sep 09 20:52:37.210312 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.202942    2576 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 09 20:52:37.210312 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.202945    2576 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 09 20:52:37.210312 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.202948    2576 flags.go:64] FLAG: --seccomp-default="false"
Sep 09 20:52:37.210312 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.202951    2576 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 09 20:52:37.210312 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.202959    2576 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 09 20:52:37.210312 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.202962    2576 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 09 20:52:37.210312 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.202965    2576 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 09 20:52:37.210312 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.202968    2576 flags.go:64] FLAG: --storage-driver-password="root"
Sep 09 20:52:37.210312 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.202970    2576 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 09 20:52:37.210312 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.202974    2576 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 09 20:52:37.210312 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.202976    2576 flags.go:64] FLAG: --storage-driver-user="root"
Sep 09 20:52:37.210312 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.202979    2576 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 09 20:52:37.210312 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.202982    2576 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 09 20:52:37.210312 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.202985    2576 flags.go:64] FLAG: --system-cgroups=""
Sep 09 20:52:37.210312 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.202988    2576 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 09 20:52:37.210312 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.202993    2576 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 09 20:52:37.210312 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.202997    2576 flags.go:64] FLAG: --tls-cert-file=""
Sep 09 20:52:37.210312 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.203000    2576 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 09 20:52:37.210312 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.203006    2576 flags.go:64] FLAG: --tls-min-version=""
Sep 09 20:52:37.210312 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.203009    2576 flags.go:64] FLAG: --tls-private-key-file=""
Sep 09 20:52:37.210312 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.203011    2576 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 09 20:52:37.210312 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.203014    2576 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 09 20:52:37.210312 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.203017    2576 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 09 20:52:37.210942 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.203020    2576 flags.go:64] FLAG: --v="2"
Sep 09 20:52:37.210942 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.203023    2576 flags.go:64] FLAG: --version="false"
Sep 09 20:52:37.210942 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.203028    2576 flags.go:64] FLAG: --vmodule=""
Sep 09 20:52:37.210942 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.203031    2576 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 09 20:52:37.210942 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.203034    2576 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 09 20:52:37.210942 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.203182    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 20:52:37.210942 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.203185    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 20:52:37.210942 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.203189    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 20:52:37.210942 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.203192    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 20:52:37.210942 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.203195    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 20:52:37.210942 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.203198    2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 20:52:37.210942 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.203200    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 20:52:37.210942 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.203203    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 20:52:37.210942 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.203206    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 20:52:37.210942 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.203209    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 20:52:37.210942 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.203212    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 20:52:37.210942 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.203214    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 20:52:37.210942 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.203217    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 20:52:37.210942 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.203219    2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 20:52:37.210942 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.203222    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 20:52:37.210942 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.203224    2576 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 20:52:37.211820 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.203227    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 20:52:37.211820 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.203229    2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 20:52:37.211820 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.203232    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 20:52:37.211820 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.203234    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 20:52:37.211820 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.203238    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 20:52:37.211820 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.203241    2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 20:52:37.211820 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.203243    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 20:52:37.211820 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.203246    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 20:52:37.211820 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.203248    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 20:52:37.211820 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.203251    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 20:52:37.211820 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.203253    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 20:52:37.211820 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.203256    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 20:52:37.211820 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.203258    2576 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 20:52:37.211820 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.203260    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 20:52:37.211820 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.203263    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 20:52:37.211820 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.203266    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 20:52:37.211820 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.203269    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 20:52:37.211820 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.203272    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 20:52:37.211820 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.203276    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 20:52:37.212474 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.203279    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 20:52:37.212474 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.203282    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 20:52:37.212474 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.203285    2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 20:52:37.212474 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.203288    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 20:52:37.212474 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.203291    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 20:52:37.212474 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.203294    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 20:52:37.212474 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.203296    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 20:52:37.212474 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.203299    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 20:52:37.212474 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.203302    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 20:52:37.212474 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.203304    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 20:52:37.212474 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.203307    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 20:52:37.212474 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.203309    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 20:52:37.212474 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.203311    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 20:52:37.212474 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.203314    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 20:52:37.212474 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.203316    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 20:52:37.212474 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.203319    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 20:52:37.212474 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.203323    2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 20:52:37.212474 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.203327    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 20:52:37.212474 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.203330    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 20:52:37.212940 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.203333    2576 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 20:52:37.212940 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.203335    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 20:52:37.212940 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.203338    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 20:52:37.212940 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.203340    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 20:52:37.212940 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.203343    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 20:52:37.212940 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.203345    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 20:52:37.212940 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.203348    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 20:52:37.212940 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.203350    2576 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 20:52:37.212940 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.203353    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 20:52:37.212940 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.203356    2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 20:52:37.212940 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.203360    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 20:52:37.212940 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.203379    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 20:52:37.212940 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.203383    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 20:52:37.212940 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.203387    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 20:52:37.212940 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.203392    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 20:52:37.212940 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.203394    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 20:52:37.212940 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.203397    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 20:52:37.212940 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.203400    2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 20:52:37.212940 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.203402    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 20:52:37.212940 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.203405    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 20:52:37.213477 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.203409    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 20:52:37.213477 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.203414    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 20:52:37.213477 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.203417    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 20:52:37.213477 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.203420    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 20:52:37.213477 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.203423    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 20:52:37.213477 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.203425    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 20:52:37.213477 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.203428    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 20:52:37.213477 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.203430    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 20:52:37.213477 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.203432    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 20:52:37.213477 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.203435    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 20:52:37.213477 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.203439    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 20:52:37.213477 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.203442    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 20:52:37.213477 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.203444    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 20:52:37.213477 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.203446    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 20:52:37.213477 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.203449    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 20:52:37.213477 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.203451    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 20:52:37.213477 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.203454    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 20:52:37.213477 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.203456    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 20:52:37.213477 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.203458    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 20:52:37.213477 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.203461    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 20:52:37.213954 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.203464    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 20:52:37.213954 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.203466    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 20:52:37.213954 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.203470    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 20:52:37.213954 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.203473    2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 20:52:37.213954 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.203475    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 20:52:37.213954 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.204183    2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 20:52:37.213954 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.213147    2576 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 09 20:52:37.213954 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.213164    2576 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 09 20:52:37.214828 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.214816    2576 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 09 20:52:37.214867 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.214832    2576 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 09 20:52:37.214900 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.214894    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 20:52:37.214900 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.214899    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 20:52:37.214953 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.214903    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 20:52:37.214953 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.214907    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 20:52:37.214953 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.214911    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 20:52:37.214953 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.214915    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 20:52:37.214953 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.214919    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 20:52:37.214953 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.214922    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 20:52:37.214953 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.214925    2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 20:52:37.214953 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.214927    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 20:52:37.214953 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.214930    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 20:52:37.214953 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.214933    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 20:52:37.214953 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.214936    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 20:52:37.214953 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.214938    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 20:52:37.214953 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.214941    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 20:52:37.214953 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.214943    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 20:52:37.214953 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.214946    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 20:52:37.214953 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.214948    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 20:52:37.214953 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.214951    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 20:52:37.214953 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.214953    2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 20:52:37.214953 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.214956    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 20:52:37.215443 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.214960    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 20:52:37.215443 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.214963    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 20:52:37.215443 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.214966    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 20:52:37.215443 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.214969    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 20:52:37.215443 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.214971    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 20:52:37.215443 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.214974    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 20:52:37.215443 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.214976    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 20:52:37.215443 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.214979    2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 20:52:37.215443 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.214981    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 20:52:37.215443 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.214984    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 20:52:37.215443 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.214986    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 20:52:37.215443 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.214988    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 20:52:37.215443 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.214991    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 20:52:37.215443 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.214993    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 20:52:37.215443 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.214996    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 20:52:37.215443 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.214998    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 20:52:37.215443 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.215001    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 20:52:37.215443 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.215003    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 20:52:37.215443 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.215006    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 20:52:37.215443 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.215008    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 20:52:37.215956 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.215010    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 20:52:37.215956 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.215013    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 20:52:37.215956 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.215015    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 20:52:37.215956 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.215018    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 20:52:37.215956 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.215020    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 20:52:37.215956 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.215023    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 20:52:37.215956 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.215025    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 20:52:37.215956 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.215028    2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 20:52:37.215956 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.215030    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 20:52:37.215956 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.215032    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 20:52:37.215956 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.215035    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 20:52:37.215956 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.215038    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 20:52:37.215956 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.215040    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 20:52:37.215956 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.215044    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 20:52:37.215956 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.215047    2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 20:52:37.215956 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.215051    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 20:52:37.215956 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.215055    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 20:52:37.215956 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.215057    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 20:52:37.215956 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.215060    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 20:52:37.215956 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.215062    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 20:52:37.216463 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.215065    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 20:52:37.216463 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.215067    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 20:52:37.216463 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.215070    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 20:52:37.216463 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.215072    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 20:52:37.216463 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.215074    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 20:52:37.216463 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.215077    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 20:52:37.216463 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.215079    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 20:52:37.216463 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.215082    2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 20:52:37.216463 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.215084    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 20:52:37.216463 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.215086    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 20:52:37.216463 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.215089    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 20:52:37.216463 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.215092    2576 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 20:52:37.216463 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.215094    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 20:52:37.216463 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.215096    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 20:52:37.216463 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.215099    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 20:52:37.216463 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.215101    2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 20:52:37.216463 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.215104    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 20:52:37.216463 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.215106    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 20:52:37.216463 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.215116    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 20:52:37.216917 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.215118    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 20:52:37.216917 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.215121    2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 20:52:37.216917 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.215123    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 20:52:37.216917 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.215126    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 20:52:37.216917 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.215129    2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 20:52:37.216917 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.215133    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 20:52:37.216917 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.215136    2576 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 20:52:37.216917 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.215139    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 20:52:37.216917 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.215142    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 20:52:37.216917 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.215145    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 20:52:37.216917 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.215147    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 20:52:37.216917 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.215150    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 20:52:37.216917 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.215152    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 20:52:37.216917 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.215154    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 20:52:37.216917 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.215157    2576 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 20:52:37.216917 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.215159    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 20:52:37.216917 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.215162    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 20:52:37.216917 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.215164    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 20:52:37.216917 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.215167    2576 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 20:52:37.217397 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.215172    2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 20:52:37.217397 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.215324    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 20:52:37.217397 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.215330    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 20:52:37.217397 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.215333    2576 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 20:52:37.217397 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.215336    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 20:52:37.217397 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.215339    2576 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 20:52:37.217397 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.215341    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 20:52:37.217397 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.215344    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 20:52:37.217397 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.215347    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 20:52:37.217397 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.215350    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 20:52:37.217397 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.215353    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 20:52:37.217397 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.215355    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 20:52:37.217397 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.215358    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 20:52:37.217397 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.215380    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 20:52:37.217397 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.215383    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 20:52:37.217775 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.215386    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 20:52:37.217775 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.215388    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 20:52:37.217775 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.215391    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 20:52:37.217775 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.215393    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 20:52:37.217775 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.215396    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 20:52:37.217775 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.215400    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 20:52:37.217775 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.215404    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 20:52:37.217775 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.215406    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 20:52:37.217775 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.215409    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 20:52:37.217775 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.215411    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 20:52:37.217775 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.215414    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 20:52:37.217775 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.215416    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 20:52:37.217775 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.215419    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 20:52:37.217775 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.215421    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 20:52:37.217775 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.215425    2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 20:52:37.217775 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.215429    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 20:52:37.217775 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.215432    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 20:52:37.217775 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.215435    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 20:52:37.217775 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.215437    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 20:52:37.218226 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.215440    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 20:52:37.218226 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.215442    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 20:52:37.218226 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.215444    2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 20:52:37.218226 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.215447    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 20:52:37.218226 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.215450    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 20:52:37.218226 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.215452    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 20:52:37.218226 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.215455    2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 20:52:37.218226 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.215457    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 20:52:37.218226 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.215460    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 20:52:37.218226 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.215463    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 20:52:37.218226 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.215465    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 20:52:37.218226 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.215467    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 20:52:37.218226 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.215476    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 20:52:37.218226 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.215478    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 20:52:37.218226 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.215481    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 20:52:37.218226 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.215483    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 20:52:37.218226 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.215487    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 20:52:37.218226 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.215489    2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 20:52:37.218226 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.215492    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 20:52:37.218226 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.215495    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 20:52:37.218785 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.215498    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 20:52:37.218785 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.215500    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 20:52:37.218785 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.215502    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 20:52:37.218785 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.215505    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 20:52:37.218785 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.215507    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 20:52:37.218785 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.215510    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 20:52:37.218785 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.215512    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 20:52:37.218785 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.215515    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 20:52:37.218785 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.215517    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 20:52:37.218785 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.215519    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 20:52:37.218785 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.215522    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 20:52:37.218785 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.215524    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 20:52:37.218785 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.215527    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 20:52:37.218785 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.215529    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 20:52:37.218785 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.215531    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 20:52:37.218785 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.215534    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 20:52:37.218785 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.215537    2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 20:52:37.218785 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.215539    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 20:52:37.218785 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.215541    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 20:52:37.218785 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.215544    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 20:52:37.219268 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.215546    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 20:52:37.219268 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.215549    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 20:52:37.219268 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.215551    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 20:52:37.219268 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.215553    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 20:52:37.219268 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.215556    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 20:52:37.219268 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.215559    2576 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 20:52:37.219268 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.215562    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 20:52:37.219268 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.215564    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 20:52:37.219268 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.215567    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 20:52:37.219268 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.215569    2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 20:52:37.219268 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.215572    2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 20:52:37.219268 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.215575    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 20:52:37.219268 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.215577    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 20:52:37.219268 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.215579    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 20:52:37.219268 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.215582    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 20:52:37.219268 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.215584    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 20:52:37.219268 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.215588    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 20:52:37.219268 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.215591    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 20:52:37.219268 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.215594    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 20:52:37.219268 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.215597    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 20:52:37.219754 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.215599    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 20:52:37.219754 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.215602    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 20:52:37.219754 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.215604    2576 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 20:52:37.219754 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.215607    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 20:52:37.219754 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.215609    2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 20:52:37.219754 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:37.215612    2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 20:52:37.219754 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.215616    2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 20:52:37.219754 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.215742    2576 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 09 20:52:37.219950 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.219828    2576 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 09 20:52:37.220840 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.220828    2576 server.go:1019] "Starting client certificate rotation"
Sep 09 20:52:37.220972 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.220953    2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 09 20:52:37.221036 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.220995    2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 09 20:52:37.223739 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.223720    2576 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 09 20:52:37.224940 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.224922    2576 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 09 20:52:37.240335 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.240312    2576 log.go:25] "Validated CRI v1 runtime API"
Sep 09 20:52:37.244105 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.244083    2576 log.go:25] "Validated CRI v1 image API"
Sep 09 20:52:37.244976 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.244957    2576 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 09 20:52:37.249812 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.249789    2576 fs.go:135] Filesystem UUIDs: map[2403a00d-98db-4d25-911c-f1818cbdfe5a:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 c738c062-149a-4639-bf61-484dd1a76e3b:/dev/nvme0n1p4]
Sep 09 20:52:37.249895 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.249808    2576 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 09 20:52:37.256772 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.256268    2576 manager.go:216] Machine: {Timestamp:2026-09-09 20:52:37.254107599 +0000 UTC m=+0.367793452 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101744 MemoryCapacity:32812154880 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec22fa4ec63da0a4b484df7006693fb3 SystemUUID:ec22fa4e-c63d-a0a4-b484-df7006693fb3 BootID:e6a02596-136f-4390-bde7-535f9fd4de19 Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406077440 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406077440 Type:vfs Inodes:4005390 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562430976 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:71:64:f4:ac:ed Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:71:64:f4:ac:ed Speed:0 Mtu:9001} {Name:ovs-system MacAddress:fa:6d:d2:8f:77:4c Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812154880 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 09 20:52:37.256886 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.256769    2576 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 09 20:52:37.256990 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.256947    2576 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 09 20:52:37.259324 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.259303    2576 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 09 20:52:37.261587 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.261382    2576 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 09 20:52:37.261727 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.261595    2576 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-72.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 09 20:52:37.262399 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.262388    2576 topology_manager.go:138] "Creating topology manager with none policy"
Sep 09 20:52:37.262439 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.262402    2576 container_manager_linux.go:309] "Creating device plugin manager"
Sep 09 20:52:37.262439 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.262410    2576 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 09 20:52:37.263663 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.263653    2576 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 09 20:52:37.264486 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.264476    2576 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 09 20:52:37.267276 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.267266    2576 state_mem.go:36] "Initialized new in-memory state store"
Sep 09 20:52:37.267410 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.267400    2576 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 09 20:52:37.267441 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.267432    2576 kubelet.go:486] "Attempting to sync node with API server"
Sep 09 20:52:37.267475 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.267441    2576 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 09 20:52:37.267475 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.267455    2576 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 09 20:52:37.267475 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.267465    2576 kubelet.go:398] "Adding apiserver pod source"
Sep 09 20:52:37.267475 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.267473    2576 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 09 20:52:37.267828 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.267818    2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 09 20:52:37.267875 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.267838    2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 09 20:52:37.271675 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.271648    2576 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 09 20:52:37.276547 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.276533    2576 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 09 20:52:37.276625 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.276563    2576 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 09 20:52:37.277794 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:52:37.277770    2576 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-134-72.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 09 20:52:37.277841 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:52:37.277809    2576 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 09 20:52:37.278798 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.278787    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 09 20:52:37.278832 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.278807    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 09 20:52:37.278832 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.278814    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 09 20:52:37.278832 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.278822    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 09 20:52:37.278927 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.278834    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 09 20:52:37.278927 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.278842    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 09 20:52:37.278927 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.278869    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 09 20:52:37.278927 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.278878    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 09 20:52:37.278927 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.278885    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 09 20:52:37.278927 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.278891    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 09 20:52:37.278927 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.278900    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 09 20:52:37.278927 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.278905    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 09 20:52:37.280168 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.280159    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 09 20:52:37.280168 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.280168    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 09 20:52:37.284221 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.284206    2576 server.go:1268] "Started kubelet"
Sep 09 20:52:37.284313 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.284269    2576 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 09 20:52:37.284351 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.284303    2576 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 09 20:52:37.284398 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.284382    2576 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 09 20:52:37.285190 ip-10-0-134-72 systemd[1]: Started Kubernetes Kubelet.
Sep 09 20:52:37.285724 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.285666    2576 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 09 20:52:37.286639 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.286535    2576 server.go:310] "Adding debug handlers to kubelet server"
Sep 09 20:52:37.291381 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:52:37.291345    2576 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 09 20:52:37.292556 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.292540    2576 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 09 20:52:37.292627 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.292544    2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 09 20:52:37.292747 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:52:37.292729    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-72.ec2.internal\" not found"
Sep 09 20:52:37.293251 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.293227    2576 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 09 20:52:37.293384 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.293262    2576 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 09 20:52:37.293384 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.293293    2576 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 09 20:52:37.293508 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.293417    2576 reconstruct.go:97] "Volume reconstruction finished"
Sep 09 20:52:37.293508 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.293424    2576 reconciler.go:29] "Reconciler: start to sync state"
Sep 09 20:52:37.293508 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.293468    2576 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 09 20:52:37.293508 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.293482    2576 factory.go:55] Registering systemd factory
Sep 09 20:52:37.293508 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.293492    2576 factory.go:223] Registration of the systemd container factory successfully
Sep 09 20:52:37.293653 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.293558    2576 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-134-72.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 09 20:52:37.294209 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.294192    2576 factory.go:153] Registering CRI-O factory
Sep 09 20:52:37.294209 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.294210    2576 factory.go:223] Registration of the crio container factory successfully
Sep 09 20:52:37.294323 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.294228    2576 factory.go:103] Registering Raw factory
Sep 09 20:52:37.294323 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.294245    2576 manager.go:1196] Started watching for new ooms in manager
Sep 09 20:52:37.294691 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.294677    2576 manager.go:349] Starting recovery of all containers
Sep 09 20:52:37.294744 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:52:37.293080    2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-72.ec2.internal.18d3c25751a8ce4a  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-72.ec2.internal,UID:ip-10-0-134-72.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-134-72.ec2.internal,},FirstTimestamp:2026-09-09 20:52:37.28417953 +0000 UTC m=+0.397865380,LastTimestamp:2026-09-09 20:52:37.28417953 +0000 UTC m=+0.397865380,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-72.ec2.internal,}"
Sep 09 20:52:37.295900 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:52:37.295877    2576 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 09 20:52:37.295975 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:52:37.295951    2576 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-134-72.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 09 20:52:37.307544 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.307351    2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-4qwx9"
Sep 09 20:52:37.307629 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.307476    2576 manager.go:354] Recovery completed
Sep 09 20:52:37.311821 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.311809    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 20:52:37.313607 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.313593    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-72.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 20:52:37.313656 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.313625    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-72.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 20:52:37.313656 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.313638    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-72.ec2.internal" event="NodeHasSufficientPID"
Sep 09 20:52:37.314134 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.314120    2576 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 09 20:52:37.314191 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.314136    2576 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 09 20:52:37.314191 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.314156    2576 state_mem.go:36] "Initialized new in-memory state store"
Sep 09 20:52:37.314191 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.314127    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 20:52:37.314846 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.314832    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-72.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 20:52:37.314897 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.314856    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-72.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 20:52:37.314897 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.314865    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-72.ec2.internal" event="NodeHasSufficientPID"
Sep 09 20:52:37.315381 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.315351    2576 policy_none.go:49] "None policy: Start"
Sep 09 20:52:37.315422 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.315404    2576 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 09 20:52:37.315422 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.315416    2576 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 09 20:52:37.315709 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:52:37.315637    2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-72.ec2.internal.18d3c2575369e00d  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-72.ec2.internal,UID:ip-10-0-134-72.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-134-72.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-134-72.ec2.internal,},FirstTimestamp:2026-09-09 20:52:37.313609741 +0000 UTC m=+0.427295592,LastTimestamp:2026-09-09 20:52:37.313609741 +0000 UTC m=+0.427295592,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-72.ec2.internal,}"
Sep 09 20:52:37.315834 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.315820    2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-4qwx9"
Sep 09 20:52:37.316710 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.316697    2576 policy_none.go:47] "Start"
Sep 09 20:52:37.358203 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.358182    2576 manager.go:337] "Starting Device Plugin manager"
Sep 09 20:52:37.362022 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:52:37.358228    2576 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 09 20:52:37.362022 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.358240    2576 server.go:87] "Starting device plugin registration server"
Sep 09 20:52:37.362022 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.358568    2576 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 09 20:52:37.362022 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.358580    2576 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 09 20:52:37.362022 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.358696    2576 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 09 20:52:37.362022 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.358838    2576 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 09 20:52:37.362022 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.358847    2576 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 09 20:52:37.362022 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:52:37.359267    2576 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 09 20:52:37.362022 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:52:37.359312    2576 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-72.ec2.internal\" not found"
Sep 09 20:52:37.401419 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.401382    2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 09 20:52:37.402748 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.402729    2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 09 20:52:37.402875 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.402794    2576 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 09 20:52:37.402875 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.402857    2576 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 09 20:52:37.402969 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:52:37.402906    2576 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 09 20:52:37.406132 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.406113    2576 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 20:52:37.459244 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.459159    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 20:52:37.460202 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.460186    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-72.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 20:52:37.460276 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.460222    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-72.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 20:52:37.460276 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.460237    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-72.ec2.internal" event="NodeHasSufficientPID"
Sep 09 20:52:37.460276 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.460267    2576 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-134-72.ec2.internal"
Sep 09 20:52:37.469721 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.469701    2576 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-134-72.ec2.internal"
Sep 09 20:52:37.469797 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:52:37.469726    2576 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-134-72.ec2.internal\": node \"ip-10-0-134-72.ec2.internal\" not found"
Sep 09 20:52:37.486689 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:52:37.486663    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-72.ec2.internal\" not found"
Sep 09 20:52:37.503182 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.503158    2576 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-72.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-72.ec2.internal"]
Sep 09 20:52:37.503257 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.503224    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 20:52:37.504859 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.504838    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-72.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 20:52:37.504975 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.504871    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-72.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 20:52:37.504975 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.504893    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-72.ec2.internal" event="NodeHasSufficientPID"
Sep 09 20:52:37.506104 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.506093    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 20:52:37.506221 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.506206    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-72.ec2.internal"
Sep 09 20:52:37.506270 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.506260    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 20:52:37.506812 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.506797    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-72.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 20:52:37.506887 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.506826    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-72.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 20:52:37.506887 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.506836    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-72.ec2.internal" event="NodeHasSufficientPID"
Sep 09 20:52:37.508291 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.508268    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-72.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 20:52:37.508395 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.508302    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-72.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 20:52:37.508395 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.508313    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-72.ec2.internal" event="NodeHasSufficientPID"
Sep 09 20:52:37.509435 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.509422    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-72.ec2.internal"
Sep 09 20:52:37.509492 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.509445    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 20:52:37.510113 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.510095    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-72.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 20:52:37.510212 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.510122    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-72.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 20:52:37.510212 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.510134    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-72.ec2.internal" event="NodeHasSufficientPID"
Sep 09 20:52:37.538030 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:52:37.538011    2576 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-72.ec2.internal\" not found" node="ip-10-0-134-72.ec2.internal"
Sep 09 20:52:37.542636 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:52:37.542621    2576 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-72.ec2.internal\" not found" node="ip-10-0-134-72.ec2.internal"
Sep 09 20:52:37.586977 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:52:37.586942    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-72.ec2.internal\" not found"
Sep 09 20:52:37.594867 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.594846    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8d0d54b6d026f1113a292c2b25b7bd73-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-72.ec2.internal\" (UID: \"8d0d54b6d026f1113a292c2b25b7bd73\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-72.ec2.internal"
Sep 09 20:52:37.594924 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.594881    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8d0d54b6d026f1113a292c2b25b7bd73-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-72.ec2.internal\" (UID: \"8d0d54b6d026f1113a292c2b25b7bd73\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-72.ec2.internal"
Sep 09 20:52:37.594924 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.594899    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2d40b80525651e1d9758e04c441ca328-config\") pod \"kube-apiserver-proxy-ip-10-0-134-72.ec2.internal\" (UID: \"2d40b80525651e1d9758e04c441ca328\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-72.ec2.internal"
Sep 09 20:52:37.687692 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:52:37.687667    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-72.ec2.internal\" not found"
Sep 09 20:52:37.695967 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.695949    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2d40b80525651e1d9758e04c441ca328-config\") pod \"kube-apiserver-proxy-ip-10-0-134-72.ec2.internal\" (UID: \"2d40b80525651e1d9758e04c441ca328\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-72.ec2.internal"
Sep 09 20:52:37.696016 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.695973    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8d0d54b6d026f1113a292c2b25b7bd73-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-72.ec2.internal\" (UID: \"8d0d54b6d026f1113a292c2b25b7bd73\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-72.ec2.internal"
Sep 09 20:52:37.696016 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.695989    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8d0d54b6d026f1113a292c2b25b7bd73-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-72.ec2.internal\" (UID: \"8d0d54b6d026f1113a292c2b25b7bd73\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-72.ec2.internal"
Sep 09 20:52:37.696077 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.696015    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8d0d54b6d026f1113a292c2b25b7bd73-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-72.ec2.internal\" (UID: \"8d0d54b6d026f1113a292c2b25b7bd73\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-72.ec2.internal"
Sep 09 20:52:37.696077 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.696052    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8d0d54b6d026f1113a292c2b25b7bd73-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-72.ec2.internal\" (UID: \"8d0d54b6d026f1113a292c2b25b7bd73\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-72.ec2.internal"
Sep 09 20:52:37.696077 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.696052    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2d40b80525651e1d9758e04c441ca328-config\") pod \"kube-apiserver-proxy-ip-10-0-134-72.ec2.internal\" (UID: \"2d40b80525651e1d9758e04c441ca328\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-72.ec2.internal"
Sep 09 20:52:37.788496 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:52:37.788432    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-72.ec2.internal\" not found"
Sep 09 20:52:37.842889 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.842864    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-72.ec2.internal"
Sep 09 20:52:37.848685 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:37.848669    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-72.ec2.internal"
Sep 09 20:52:37.889434 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:52:37.889409    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-72.ec2.internal\" not found"
Sep 09 20:52:37.989951 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:52:37.989903    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-72.ec2.internal\" not found"
Sep 09 20:52:38.090495 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:52:38.090423    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-72.ec2.internal\" not found"
Sep 09 20:52:38.190973 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:52:38.190934    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-72.ec2.internal\" not found"
Sep 09 20:52:38.220435 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:38.220411    2576 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 09 20:52:38.220792 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:38.220562    2576 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 09 20:52:38.291944 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:52:38.291912    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-72.ec2.internal\" not found"
Sep 09 20:52:38.320134 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:38.320106    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-08 20:47:37 +0000 UTC" deadline="2028-04-23 19:14:27.363066835 +0000 UTC"
Sep 09 20:52:38.320134 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:38.320130    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14206h21m49.042939702s"
Sep 09 20:52:38.392618 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:52:38.392550    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-72.ec2.internal\" not found"
Sep 09 20:52:38.393955 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:38.393938    2576 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 20:52:38.493358 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:52:38.493335    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-72.ec2.internal\" not found"
Sep 09 20:52:38.572464 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:38.572424    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d40b80525651e1d9758e04c441ca328.slice/crio-c7be44e3480c59861d22979e8c1c797eb379373f90a5876a105675800f0549c7 WatchSource:0}: Error finding container c7be44e3480c59861d22979e8c1c797eb379373f90a5876a105675800f0549c7: Status 404 returned error can't find the container with id c7be44e3480c59861d22979e8c1c797eb379373f90a5876a105675800f0549c7
Sep 09 20:52:38.577889 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:38.577876    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 20:52:38.593606 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:52:38.593586    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-72.ec2.internal\" not found"
Sep 09 20:52:38.685133 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:38.685066    2576 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 20:52:38.693343 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:38.693319    2576 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-72.ec2.internal"
Sep 09 20:52:38.702669 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:38.702650    2576 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 20:52:38.707062 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:38.707048    2576 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 09 20:52:38.707954 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:38.707940    2576 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-72.ec2.internal"
Sep 09 20:52:38.715535 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:38.715519    2576 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 09 20:52:38.774493 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:38.774467    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d0d54b6d026f1113a292c2b25b7bd73.slice/crio-0a8a7e9c707607796cda15ec2971201166d4c9ef92705bf41d654f48578d3fb9 WatchSource:0}: Error finding container 0a8a7e9c707607796cda15ec2971201166d4c9ef92705bf41d654f48578d3fb9: Status 404 returned error can't find the container with id 0a8a7e9c707607796cda15ec2971201166d4c9ef92705bf41d654f48578d3fb9
Sep 09 20:52:39.268520 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.268482    2576 apiserver.go:52] "Watching apiserver"
Sep 09 20:52:39.280729 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.280702    2576 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 09 20:52:39.281248 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.281221    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-tkbs9","kube-system/konnectivity-agent-5v6vg","kube-system/kube-apiserver-proxy-ip-10-0-134-72.ec2.internal","openshift-cluster-node-tuning-operator/tuned-6mr4s","openshift-dns/node-resolver-wvw9h","openshift-image-registry/node-ca-dcv48","openshift-multus/multus-additional-cni-plugins-d8z7g","openshift-multus/network-metrics-daemon-v74xn","kube-system/global-pull-secret-syncer-2c9kq","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5vpsh","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-72.ec2.internal","openshift-multus/multus-5vzc7","openshift-network-diagnostics/network-check-target-t8tk8","openshift-network-operator/iptables-alerter-x9txm"]
Sep 09 20:52:39.283382 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.283358    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-d8z7g"
Sep 09 20:52:39.285373 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.285352    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t8tk8"
Sep 09 20:52:39.285497 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:52:39.285441    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t8tk8" podUID="82d55335-0036-4166-a35c-65adec33932d"
Sep 09 20:52:39.286254 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.286169    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 09 20:52:39.286254 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.286231    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 09 20:52:39.286464 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.286331    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 09 20:52:39.286574 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.286554    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 09 20:52:39.286635 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.286559    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 09 20:52:39.286692 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.286682    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-7wx6b\""
Sep 09 20:52:39.287148 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.287115    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-x9txm"
Sep 09 20:52:39.289453 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.289429    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 09 20:52:39.289773 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.289754    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 09 20:52:39.289976 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.289961    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 09 20:52:39.290176 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.290156    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-64qrd\""
Sep 09 20:52:39.290672 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.290405    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-6mr4s"
Sep 09 20:52:39.291871 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.291851    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-wvw9h"
Sep 09 20:52:39.293007 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.292987    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 09 20:52:39.293110 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.293076    2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 09 20:52:39.293110 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.293093    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 09 20:52:39.293228 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.293178    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-8bnxn\""
Sep 09 20:52:39.294720 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.294523    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 09 20:52:39.294720 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.294619    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 09 20:52:39.294862 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.294755    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-bpqbc\""
Sep 09 20:52:39.295240 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.295106    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-dcv48"
Sep 09 20:52:39.298295 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.298274    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2c9kq"
Sep 09 20:52:39.298396 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:52:39.298357    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2c9kq" podUID="b906e5b3-61d3-41da-9776-795376a73811"
Sep 09 20:52:39.298795 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.298777    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-zt2cb\""
Sep 09 20:52:39.299009 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.298995    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 09 20:52:39.299159 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.299137    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 09 20:52:39.299306 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.299288    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 09 20:52:39.300120 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.300059    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-tkbs9"
Sep 09 20:52:39.302748 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.302726    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 09 20:52:39.302748 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.302744    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-5v6vg"
Sep 09 20:52:39.303485 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.302955    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 09 20:52:39.304556 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.303594    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-bvs95\""
Sep 09 20:52:39.304556 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.303760    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 09 20:52:39.304556 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.303765    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 09 20:52:39.304556 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.303940    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/af099159-41cf-43d9-8ffb-6e5fb9aeea23-var-lib-kubelet\") pod \"tuned-6mr4s\" (UID: \"af099159-41cf-43d9-8ffb-6e5fb9aeea23\") " pod="openshift-cluster-node-tuning-operator/tuned-6mr4s"
Sep 09 20:52:39.304556 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.303959    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3ede0a95-3907-4d9c-aa45-acf822104e28-system-cni-dir\") pod \"multus-additional-cni-plugins-d8z7g\" (UID: \"3ede0a95-3907-4d9c-aa45-acf822104e28\") " pod="openshift-multus/multus-additional-cni-plugins-d8z7g"
Sep 09 20:52:39.304556 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.303971    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 09 20:52:39.304556 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.303977    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3ede0a95-3907-4d9c-aa45-acf822104e28-cni-binary-copy\") pod \"multus-additional-cni-plugins-d8z7g\" (UID: \"3ede0a95-3907-4d9c-aa45-acf822104e28\") " pod="openshift-multus/multus-additional-cni-plugins-d8z7g"
Sep 09 20:52:39.304556 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.303994    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qbd6\" (UniqueName: \"kubernetes.io/projected/cc3e4e7d-c7ef-462c-963a-bbcb4931a486-kube-api-access-6qbd6\") pod \"iptables-alerter-x9txm\" (UID: \"cc3e4e7d-c7ef-462c-963a-bbcb4931a486\") " pod="openshift-network-operator/iptables-alerter-x9txm"
Sep 09 20:52:39.304556 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.304008    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/af099159-41cf-43d9-8ffb-6e5fb9aeea23-etc-sysconfig\") pod \"tuned-6mr4s\" (UID: \"af099159-41cf-43d9-8ffb-6e5fb9aeea23\") " pod="openshift-cluster-node-tuning-operator/tuned-6mr4s"
Sep 09 20:52:39.304556 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.304023    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/af099159-41cf-43d9-8ffb-6e5fb9aeea23-etc-tuned\") pod \"tuned-6mr4s\" (UID: \"af099159-41cf-43d9-8ffb-6e5fb9aeea23\") " pod="openshift-cluster-node-tuning-operator/tuned-6mr4s"
Sep 09 20:52:39.304556 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.304040    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3ede0a95-3907-4d9c-aa45-acf822104e28-tuning-conf-dir\") pod \"multus-additional-cni-plugins-d8z7g\" (UID: \"3ede0a95-3907-4d9c-aa45-acf822104e28\") " pod="openshift-multus/multus-additional-cni-plugins-d8z7g"
Sep 09 20:52:39.304556 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.304057    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68xzx\" (UniqueName: \"kubernetes.io/projected/af099159-41cf-43d9-8ffb-6e5fb9aeea23-kube-api-access-68xzx\") pod \"tuned-6mr4s\" (UID: \"af099159-41cf-43d9-8ffb-6e5fb9aeea23\") " pod="openshift-cluster-node-tuning-operator/tuned-6mr4s"
Sep 09 20:52:39.304556 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.304074    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3ede0a95-3907-4d9c-aa45-acf822104e28-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-d8z7g\" (UID: \"3ede0a95-3907-4d9c-aa45-acf822104e28\") " pod="openshift-multus/multus-additional-cni-plugins-d8z7g"
Sep 09 20:52:39.304556 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.304091    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 09 20:52:39.304556 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.304089    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3ede0a95-3907-4d9c-aa45-acf822104e28-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-d8z7g\" (UID: \"3ede0a95-3907-4d9c-aa45-acf822104e28\") " pod="openshift-multus/multus-additional-cni-plugins-d8z7g"
Sep 09 20:52:39.304556 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.304192    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/af099159-41cf-43d9-8ffb-6e5fb9aeea23-run\") pod \"tuned-6mr4s\" (UID: \"af099159-41cf-43d9-8ffb-6e5fb9aeea23\") " pod="openshift-cluster-node-tuning-operator/tuned-6mr4s"
Sep 09 20:52:39.304556 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.304215    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/af099159-41cf-43d9-8ffb-6e5fb9aeea23-host\") pod \"tuned-6mr4s\" (UID: \"af099159-41cf-43d9-8ffb-6e5fb9aeea23\") " pod="openshift-cluster-node-tuning-operator/tuned-6mr4s"
Sep 09 20:52:39.304556 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.304228    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/af099159-41cf-43d9-8ffb-6e5fb9aeea23-sys\") pod \"tuned-6mr4s\" (UID: \"af099159-41cf-43d9-8ffb-6e5fb9aeea23\") " pod="openshift-cluster-node-tuning-operator/tuned-6mr4s"
Sep 09 20:52:39.304556 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.304258    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2x8m7\" (UniqueName: \"kubernetes.io/projected/3ede0a95-3907-4d9c-aa45-acf822104e28-kube-api-access-2x8m7\") pod \"multus-additional-cni-plugins-d8z7g\" (UID: \"3ede0a95-3907-4d9c-aa45-acf822104e28\") " pod="openshift-multus/multus-additional-cni-plugins-d8z7g"
Sep 09 20:52:39.305266 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.304279    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/cc3e4e7d-c7ef-462c-963a-bbcb4931a486-iptables-alerter-script\") pod \"iptables-alerter-x9txm\" (UID: \"cc3e4e7d-c7ef-462c-963a-bbcb4931a486\") " pod="openshift-network-operator/iptables-alerter-x9txm"
Sep 09 20:52:39.305266 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.304293    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/af099159-41cf-43d9-8ffb-6e5fb9aeea23-etc-sysctl-conf\") pod \"tuned-6mr4s\" (UID: \"af099159-41cf-43d9-8ffb-6e5fb9aeea23\") " pod="openshift-cluster-node-tuning-operator/tuned-6mr4s"
Sep 09 20:52:39.305266 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.304306    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/af099159-41cf-43d9-8ffb-6e5fb9aeea23-etc-systemd\") pod \"tuned-6mr4s\" (UID: \"af099159-41cf-43d9-8ffb-6e5fb9aeea23\") " pod="openshift-cluster-node-tuning-operator/tuned-6mr4s"
Sep 09 20:52:39.305266 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.304339    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e706fb03-415d-407d-b105-bd06474f6882-serviceca\") pod \"node-ca-dcv48\" (UID: \"e706fb03-415d-407d-b105-bd06474f6882\") " pod="openshift-image-registry/node-ca-dcv48"
Sep 09 20:52:39.305266 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.304372    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e706fb03-415d-407d-b105-bd06474f6882-host\") pod \"node-ca-dcv48\" (UID: \"e706fb03-415d-407d-b105-bd06474f6882\") " pod="openshift-image-registry/node-ca-dcv48"
Sep 09 20:52:39.305266 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.304400    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xk99w\" (UniqueName: \"kubernetes.io/projected/e706fb03-415d-407d-b105-bd06474f6882-kube-api-access-xk99w\") pod \"node-ca-dcv48\" (UID: \"e706fb03-415d-407d-b105-bd06474f6882\") " pod="openshift-image-registry/node-ca-dcv48"
Sep 09 20:52:39.305266 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.304464    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cc3e4e7d-c7ef-462c-963a-bbcb4931a486-host-slash\") pod \"iptables-alerter-x9txm\" (UID: \"cc3e4e7d-c7ef-462c-963a-bbcb4931a486\") " pod="openshift-network-operator/iptables-alerter-x9txm"
Sep 09 20:52:39.305266 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.304506    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/af099159-41cf-43d9-8ffb-6e5fb9aeea23-etc-sysctl-d\") pod \"tuned-6mr4s\" (UID: \"af099159-41cf-43d9-8ffb-6e5fb9aeea23\") " pod="openshift-cluster-node-tuning-operator/tuned-6mr4s"
Sep 09 20:52:39.305266 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.304539    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/09f40168-2bb2-4eff-8a3f-7ef28a149eed-hosts-file\") pod \"node-resolver-wvw9h\" (UID: \"09f40168-2bb2-4eff-8a3f-7ef28a149eed\") " pod="openshift-dns/node-resolver-wvw9h"
Sep 09 20:52:39.305266 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.304568    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krdp6\" (UniqueName: \"kubernetes.io/projected/82d55335-0036-4166-a35c-65adec33932d-kube-api-access-krdp6\") pod \"network-check-target-t8tk8\" (UID: \"82d55335-0036-4166-a35c-65adec33932d\") " pod="openshift-network-diagnostics/network-check-target-t8tk8"
Sep 09 20:52:39.305266 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.304601    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/af099159-41cf-43d9-8ffb-6e5fb9aeea23-lib-modules\") pod \"tuned-6mr4s\" (UID: \"af099159-41cf-43d9-8ffb-6e5fb9aeea23\") " pod="openshift-cluster-node-tuning-operator/tuned-6mr4s"
Sep 09 20:52:39.305266 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.304625    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/af099159-41cf-43d9-8ffb-6e5fb9aeea23-tmp\") pod \"tuned-6mr4s\" (UID: \"af099159-41cf-43d9-8ffb-6e5fb9aeea23\") " pod="openshift-cluster-node-tuning-operator/tuned-6mr4s"
Sep 09 20:52:39.305266 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.304658    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09f40168-2bb2-4eff-8a3f-7ef28a149eed-tmp-dir\") pod \"node-resolver-wvw9h\" (UID: \"09f40168-2bb2-4eff-8a3f-7ef28a149eed\") " pod="openshift-dns/node-resolver-wvw9h"
Sep 09 20:52:39.305266 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.304684    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7f9rv\" (UniqueName: \"kubernetes.io/projected/09f40168-2bb2-4eff-8a3f-7ef28a149eed-kube-api-access-7f9rv\") pod \"node-resolver-wvw9h\" (UID: \"09f40168-2bb2-4eff-8a3f-7ef28a149eed\") " pod="openshift-dns/node-resolver-wvw9h"
Sep 09 20:52:39.305266 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.304709    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3ede0a95-3907-4d9c-aa45-acf822104e28-cnibin\") pod \"multus-additional-cni-plugins-d8z7g\" (UID: \"3ede0a95-3907-4d9c-aa45-acf822104e28\") " pod="openshift-multus/multus-additional-cni-plugins-d8z7g"
Sep 09 20:52:39.305266 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.304733    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/af099159-41cf-43d9-8ffb-6e5fb9aeea23-etc-modprobe-d\") pod \"tuned-6mr4s\" (UID: \"af099159-41cf-43d9-8ffb-6e5fb9aeea23\") " pod="openshift-cluster-node-tuning-operator/tuned-6mr4s"
Sep 09 20:52:39.305266 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.304757    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/af099159-41cf-43d9-8ffb-6e5fb9aeea23-etc-kubernetes\") pod \"tuned-6mr4s\" (UID: \"af099159-41cf-43d9-8ffb-6e5fb9aeea23\") " pod="openshift-cluster-node-tuning-operator/tuned-6mr4s"
Sep 09 20:52:39.305893 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.305014    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 09 20:52:39.305893 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.305158    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-4zzvr\""
Sep 09 20:52:39.305893 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.305445    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 09 20:52:39.305893 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.305460    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5vpsh"
Sep 09 20:52:39.306601 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.306329    2576 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 09 20:52:39.307701 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.307682    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 09 20:52:39.307913 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.307816    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 09 20:52:39.309509 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.308549    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v74xn"
Sep 09 20:52:39.309509 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:52:39.308627    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v74xn" podUID="5b815b2b-983b-4afe-900d-13dd791e4778"
Sep 09 20:52:39.309509 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.308951    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 09 20:52:39.309509 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.309054    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-qfkpq\""
Sep 09 20:52:39.314104 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.314081    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-5vzc7"
Sep 09 20:52:39.316578 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.316560    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 09 20:52:39.316858 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.316838    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-rpp6z\""
Sep 09 20:52:39.337623 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.337598    2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-bfmzl"
Sep 09 20:52:39.344308 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.344289    2576 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 20:52:39.348258 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.348228    2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-bfmzl"
Sep 09 20:52:39.394601 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.394574    2576 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 09 20:52:39.405458 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.405414    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09f40168-2bb2-4eff-8a3f-7ef28a149eed-tmp-dir\") pod \"node-resolver-wvw9h\" (UID: \"09f40168-2bb2-4eff-8a3f-7ef28a149eed\") " pod="openshift-dns/node-resolver-wvw9h"
Sep 09 20:52:39.405573 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.405459    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7f9rv\" (UniqueName: \"kubernetes.io/projected/09f40168-2bb2-4eff-8a3f-7ef28a149eed-kube-api-access-7f9rv\") pod \"node-resolver-wvw9h\" (UID: \"09f40168-2bb2-4eff-8a3f-7ef28a149eed\") " pod="openshift-dns/node-resolver-wvw9h"
Sep 09 20:52:39.405614 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.405580    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3ede0a95-3907-4d9c-aa45-acf822104e28-cnibin\") pod \"multus-additional-cni-plugins-d8z7g\" (UID: \"3ede0a95-3907-4d9c-aa45-acf822104e28\") " pod="openshift-multus/multus-additional-cni-plugins-d8z7g"
Sep 09 20:52:39.405650 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.405613    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/855262cd-a32b-4283-9482-cd25b26b0433-multus-cni-dir\") pod \"multus-5vzc7\" (UID: \"855262cd-a32b-4283-9482-cd25b26b0433\") " pod="openshift-multus/multus-5vzc7"
Sep 09 20:52:39.405693 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.405649    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/855262cd-a32b-4283-9482-cd25b26b0433-host-run-k8s-cni-cncf-io\") pod \"multus-5vzc7\" (UID: \"855262cd-a32b-4283-9482-cd25b26b0433\") " pod="openshift-multus/multus-5vzc7"
Sep 09 20:52:39.405760 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.405720    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3ede0a95-3907-4d9c-aa45-acf822104e28-cnibin\") pod \"multus-additional-cni-plugins-d8z7g\" (UID: \"3ede0a95-3907-4d9c-aa45-acf822104e28\") " pod="openshift-multus/multus-additional-cni-plugins-d8z7g"
Sep 09 20:52:39.405821 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.405770    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09f40168-2bb2-4eff-8a3f-7ef28a149eed-tmp-dir\") pod \"node-resolver-wvw9h\" (UID: \"09f40168-2bb2-4eff-8a3f-7ef28a149eed\") " pod="openshift-dns/node-resolver-wvw9h"
Sep 09 20:52:39.405821 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.405774    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f0db116c-804d-4ce3-b4f9-135288ad7cea-log-socket\") pod \"ovnkube-node-tkbs9\" (UID: \"f0db116c-804d-4ce3-b4f9-135288ad7cea\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkbs9"
Sep 09 20:52:39.405960 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.405852    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3ede0a95-3907-4d9c-aa45-acf822104e28-cni-binary-copy\") pod \"multus-additional-cni-plugins-d8z7g\" (UID: \"3ede0a95-3907-4d9c-aa45-acf822104e28\") " pod="openshift-multus/multus-additional-cni-plugins-d8z7g"
Sep 09 20:52:39.405960 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.405880    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qbd6\" (UniqueName: \"kubernetes.io/projected/cc3e4e7d-c7ef-462c-963a-bbcb4931a486-kube-api-access-6qbd6\") pod \"iptables-alerter-x9txm\" (UID: \"cc3e4e7d-c7ef-462c-963a-bbcb4931a486\") " pod="openshift-network-operator/iptables-alerter-x9txm"
Sep 09 20:52:39.405960 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.405909    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/af099159-41cf-43d9-8ffb-6e5fb9aeea23-etc-sysconfig\") pod \"tuned-6mr4s\" (UID: \"af099159-41cf-43d9-8ffb-6e5fb9aeea23\") " pod="openshift-cluster-node-tuning-operator/tuned-6mr4s"
Sep 09 20:52:39.405960 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.405942    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/af099159-41cf-43d9-8ffb-6e5fb9aeea23-etc-tuned\") pod \"tuned-6mr4s\" (UID: \"af099159-41cf-43d9-8ffb-6e5fb9aeea23\") " pod="openshift-cluster-node-tuning-operator/tuned-6mr4s"
Sep 09 20:52:39.406167 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.405970    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/855262cd-a32b-4283-9482-cd25b26b0433-cnibin\") pod \"multus-5vzc7\" (UID: \"855262cd-a32b-4283-9482-cd25b26b0433\") " pod="openshift-multus/multus-5vzc7"
Sep 09 20:52:39.406167 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.405992    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/af099159-41cf-43d9-8ffb-6e5fb9aeea23-etc-sysconfig\") pod \"tuned-6mr4s\" (UID: \"af099159-41cf-43d9-8ffb-6e5fb9aeea23\") " pod="openshift-cluster-node-tuning-operator/tuned-6mr4s"
Sep 09 20:52:39.406167 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.405997    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b906e5b3-61d3-41da-9776-795376a73811-original-pull-secret\") pod \"global-pull-secret-syncer-2c9kq\" (UID: \"b906e5b3-61d3-41da-9776-795376a73811\") " pod="kube-system/global-pull-secret-syncer-2c9kq"
Sep 09 20:52:39.406167 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.406061    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f0db116c-804d-4ce3-b4f9-135288ad7cea-host-run-ovn-kubernetes\") pod \"ovnkube-node-tkbs9\" (UID: \"f0db116c-804d-4ce3-b4f9-135288ad7cea\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkbs9"
Sep 09 20:52:39.406167 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.406089    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f0db116c-804d-4ce3-b4f9-135288ad7cea-host-cni-bin\") pod \"ovnkube-node-tkbs9\" (UID: \"f0db116c-804d-4ce3-b4f9-135288ad7cea\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkbs9"
Sep 09 20:52:39.406167 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.406119    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3ede0a95-3907-4d9c-aa45-acf822104e28-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-d8z7g\" (UID: \"3ede0a95-3907-4d9c-aa45-acf822104e28\") " pod="openshift-multus/multus-additional-cni-plugins-d8z7g"
Sep 09 20:52:39.406167 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.406149    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3ede0a95-3907-4d9c-aa45-acf822104e28-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-d8z7g\" (UID: \"3ede0a95-3907-4d9c-aa45-acf822104e28\") " pod="openshift-multus/multus-additional-cni-plugins-d8z7g"
Sep 09 20:52:39.406557 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.406178    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/af099159-41cf-43d9-8ffb-6e5fb9aeea23-host\") pod \"tuned-6mr4s\" (UID: \"af099159-41cf-43d9-8ffb-6e5fb9aeea23\") " pod="openshift-cluster-node-tuning-operator/tuned-6mr4s"
Sep 09 20:52:39.406557 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.406202    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b815b2b-983b-4afe-900d-13dd791e4778-metrics-certs\") pod \"network-metrics-daemon-v74xn\" (UID: \"5b815b2b-983b-4afe-900d-13dd791e4778\") " pod="openshift-multus/network-metrics-daemon-v74xn"
Sep 09 20:52:39.406557 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.406228    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/855262cd-a32b-4283-9482-cd25b26b0433-hostroot\") pod \"multus-5vzc7\" (UID: \"855262cd-a32b-4283-9482-cd25b26b0433\") " pod="openshift-multus/multus-5vzc7"
Sep 09 20:52:39.406557 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.406254    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f0db116c-804d-4ce3-b4f9-135288ad7cea-run-openvswitch\") pod \"ovnkube-node-tkbs9\" (UID: \"f0db116c-804d-4ce3-b4f9-135288ad7cea\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkbs9"
Sep 09 20:52:39.406557 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.406322    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f0db116c-804d-4ce3-b4f9-135288ad7cea-node-log\") pod \"ovnkube-node-tkbs9\" (UID: \"f0db116c-804d-4ce3-b4f9-135288ad7cea\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkbs9"
Sep 09 20:52:39.406557 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.406361    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f0db116c-804d-4ce3-b4f9-135288ad7cea-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tkbs9\" (UID: \"f0db116c-804d-4ce3-b4f9-135288ad7cea\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkbs9"
Sep 09 20:52:39.406557 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.406392    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/cc3e4e7d-c7ef-462c-963a-bbcb4931a486-iptables-alerter-script\") pod \"iptables-alerter-x9txm\" (UID: \"cc3e4e7d-c7ef-462c-963a-bbcb4931a486\") " pod="openshift-network-operator/iptables-alerter-x9txm"
Sep 09 20:52:39.406557 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.406424    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/af099159-41cf-43d9-8ffb-6e5fb9aeea23-host\") pod \"tuned-6mr4s\" (UID: \"af099159-41cf-43d9-8ffb-6e5fb9aeea23\") " pod="openshift-cluster-node-tuning-operator/tuned-6mr4s"
Sep 09 20:52:39.406557 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.406446    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/af099159-41cf-43d9-8ffb-6e5fb9aeea23-etc-sysctl-conf\") pod \"tuned-6mr4s\" (UID: \"af099159-41cf-43d9-8ffb-6e5fb9aeea23\") " pod="openshift-cluster-node-tuning-operator/tuned-6mr4s"
Sep 09 20:52:39.406557 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.406504    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f30a6570-2579-4271-ba77-d75fd2fd8390-konnectivity-ca\") pod \"konnectivity-agent-5v6vg\" (UID: \"f30a6570-2579-4271-ba77-d75fd2fd8390\") " pod="kube-system/konnectivity-agent-5v6vg"
Sep 09 20:52:39.406557 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.406551    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/855262cd-a32b-4283-9482-cd25b26b0433-multus-socket-dir-parent\") pod \"multus-5vzc7\" (UID: \"855262cd-a32b-4283-9482-cd25b26b0433\") " pod="openshift-multus/multus-5vzc7"
Sep 09 20:52:39.407131 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.406575    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/855262cd-a32b-4283-9482-cd25b26b0433-host-var-lib-cni-bin\") pod \"multus-5vzc7\" (UID: \"855262cd-a32b-4283-9482-cd25b26b0433\") " pod="openshift-multus/multus-5vzc7"
Sep 09 20:52:39.407131 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.406582    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3ede0a95-3907-4d9c-aa45-acf822104e28-cni-binary-copy\") pod \"multus-additional-cni-plugins-d8z7g\" (UID: \"3ede0a95-3907-4d9c-aa45-acf822104e28\") " pod="openshift-multus/multus-additional-cni-plugins-d8z7g"
Sep 09 20:52:39.407131 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.406603    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b3a7af8e-62e0-43d9-8845-a71f6b10c551-etc-selinux\") pod \"aws-ebs-csi-driver-node-5vpsh\" (UID: \"b3a7af8e-62e0-43d9-8845-a71f6b10c551\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5vpsh"
Sep 09 20:52:39.407131 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.406633    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b3a7af8e-62e0-43d9-8845-a71f6b10c551-sys-fs\") pod \"aws-ebs-csi-driver-node-5vpsh\" (UID: \"b3a7af8e-62e0-43d9-8845-a71f6b10c551\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5vpsh"
Sep 09 20:52:39.407131 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.406668    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e706fb03-415d-407d-b105-bd06474f6882-serviceca\") pod \"node-ca-dcv48\" (UID: \"e706fb03-415d-407d-b105-bd06474f6882\") " pod="openshift-image-registry/node-ca-dcv48"
Sep 09 20:52:39.407131 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.406685    2576 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 09 20:52:39.407131 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.406717    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/af099159-41cf-43d9-8ffb-6e5fb9aeea23-etc-sysctl-conf\") pod \"tuned-6mr4s\" (UID: \"af099159-41cf-43d9-8ffb-6e5fb9aeea23\") " pod="openshift-cluster-node-tuning-operator/tuned-6mr4s"
Sep 09 20:52:39.407131 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.406724    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76vjl\" (UniqueName: \"kubernetes.io/projected/855262cd-a32b-4283-9482-cd25b26b0433-kube-api-access-76vjl\") pod \"multus-5vzc7\" (UID: \"855262cd-a32b-4283-9482-cd25b26b0433\") " pod="openshift-multus/multus-5vzc7"
Sep 09 20:52:39.407131 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.406774    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f0db116c-804d-4ce3-b4f9-135288ad7cea-systemd-units\") pod \"ovnkube-node-tkbs9\" (UID: \"f0db116c-804d-4ce3-b4f9-135288ad7cea\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkbs9"
Sep 09 20:52:39.407131 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.406805    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3ede0a95-3907-4d9c-aa45-acf822104e28-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-d8z7g\" (UID: \"3ede0a95-3907-4d9c-aa45-acf822104e28\") " pod="openshift-multus/multus-additional-cni-plugins-d8z7g"
Sep 09 20:52:39.407131 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.406866    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f0db116c-804d-4ce3-b4f9-135288ad7cea-var-lib-openvswitch\") pod \"ovnkube-node-tkbs9\" (UID: \"f0db116c-804d-4ce3-b4f9-135288ad7cea\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkbs9"
Sep 09 20:52:39.407131 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.406900    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f0db116c-804d-4ce3-b4f9-135288ad7cea-ovnkube-config\") pod \"ovnkube-node-tkbs9\" (UID: \"f0db116c-804d-4ce3-b4f9-135288ad7cea\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkbs9"
Sep 09 20:52:39.407131 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.406936    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xk99w\" (UniqueName: \"kubernetes.io/projected/e706fb03-415d-407d-b105-bd06474f6882-kube-api-access-xk99w\") pod \"node-ca-dcv48\" (UID: \"e706fb03-415d-407d-b105-bd06474f6882\") " pod="openshift-image-registry/node-ca-dcv48"
Sep 09 20:52:39.407131 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.406970    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cc3e4e7d-c7ef-462c-963a-bbcb4931a486-host-slash\") pod \"iptables-alerter-x9txm\" (UID: \"cc3e4e7d-c7ef-462c-963a-bbcb4931a486\") " pod="openshift-network-operator/iptables-alerter-x9txm"
Sep 09 20:52:39.407131 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.407055    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/855262cd-a32b-4283-9482-cd25b26b0433-etc-kubernetes\") pod \"multus-5vzc7\" (UID: \"855262cd-a32b-4283-9482-cd25b26b0433\") " pod="openshift-multus/multus-5vzc7"
Sep 09 20:52:39.407131 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.407085    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f0db116c-804d-4ce3-b4f9-135288ad7cea-run-systemd\") pod \"ovnkube-node-tkbs9\" (UID: \"f0db116c-804d-4ce3-b4f9-135288ad7cea\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkbs9"
Sep 09 20:52:39.407131 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.407101    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/cc3e4e7d-c7ef-462c-963a-bbcb4931a486-iptables-alerter-script\") pod \"iptables-alerter-x9txm\" (UID: \"cc3e4e7d-c7ef-462c-963a-bbcb4931a486\") " pod="openshift-network-operator/iptables-alerter-x9txm"
Sep 09 20:52:39.407995 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.407172    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cc3e4e7d-c7ef-462c-963a-bbcb4931a486-host-slash\") pod \"iptables-alerter-x9txm\" (UID: \"cc3e4e7d-c7ef-462c-963a-bbcb4931a486\") " pod="openshift-network-operator/iptables-alerter-x9txm"
Sep 09 20:52:39.407995 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.407188    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e706fb03-415d-407d-b105-bd06474f6882-serviceca\") pod \"node-ca-dcv48\" (UID: \"e706fb03-415d-407d-b105-bd06474f6882\") " pod="openshift-image-registry/node-ca-dcv48"
Sep 09 20:52:39.407995 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.407200    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/09f40168-2bb2-4eff-8a3f-7ef28a149eed-hosts-file\") pod \"node-resolver-wvw9h\" (UID: \"09f40168-2bb2-4eff-8a3f-7ef28a149eed\") " pod="openshift-dns/node-resolver-wvw9h"
Sep 09 20:52:39.407995 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.407270    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3ede0a95-3907-4d9c-aa45-acf822104e28-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-d8z7g\" (UID: \"3ede0a95-3907-4d9c-aa45-acf822104e28\") " pod="openshift-multus/multus-additional-cni-plugins-d8z7g"
Sep 09 20:52:39.407995 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.407271    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krdp6\" (UniqueName: \"kubernetes.io/projected/82d55335-0036-4166-a35c-65adec33932d-kube-api-access-krdp6\") pod \"network-check-target-t8tk8\" (UID: \"82d55335-0036-4166-a35c-65adec33932d\") " pod="openshift-network-diagnostics/network-check-target-t8tk8"
Sep 09 20:52:39.407995 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.407326    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/09f40168-2bb2-4eff-8a3f-7ef28a149eed-hosts-file\") pod \"node-resolver-wvw9h\" (UID: \"09f40168-2bb2-4eff-8a3f-7ef28a149eed\") " pod="openshift-dns/node-resolver-wvw9h"
Sep 09 20:52:39.407995 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.407330    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/af099159-41cf-43d9-8ffb-6e5fb9aeea23-lib-modules\") pod \"tuned-6mr4s\" (UID: \"af099159-41cf-43d9-8ffb-6e5fb9aeea23\") " pod="openshift-cluster-node-tuning-operator/tuned-6mr4s"
Sep 09 20:52:39.407995 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.407388    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/855262cd-a32b-4283-9482-cd25b26b0433-multus-conf-dir\") pod \"multus-5vzc7\" (UID: \"855262cd-a32b-4283-9482-cd25b26b0433\") " pod="openshift-multus/multus-5vzc7"
Sep 09 20:52:39.407995 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.407490    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f0db116c-804d-4ce3-b4f9-135288ad7cea-host-cni-netd\") pod \"ovnkube-node-tkbs9\" (UID: \"f0db116c-804d-4ce3-b4f9-135288ad7cea\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkbs9"
Sep 09 20:52:39.407995 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.407539    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/af099159-41cf-43d9-8ffb-6e5fb9aeea23-lib-modules\") pod \"tuned-6mr4s\" (UID: \"af099159-41cf-43d9-8ffb-6e5fb9aeea23\") " pod="openshift-cluster-node-tuning-operator/tuned-6mr4s"
Sep 09 20:52:39.407995 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.407556    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b3a7af8e-62e0-43d9-8845-a71f6b10c551-socket-dir\") pod \"aws-ebs-csi-driver-node-5vpsh\" (UID: \"b3a7af8e-62e0-43d9-8845-a71f6b10c551\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5vpsh"
Sep 09 20:52:39.407995 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.407591    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/af099159-41cf-43d9-8ffb-6e5fb9aeea23-etc-modprobe-d\") pod \"tuned-6mr4s\" (UID: \"af099159-41cf-43d9-8ffb-6e5fb9aeea23\") " pod="openshift-cluster-node-tuning-operator/tuned-6mr4s"
Sep 09 20:52:39.407995 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.407656    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/af099159-41cf-43d9-8ffb-6e5fb9aeea23-etc-kubernetes\") pod \"tuned-6mr4s\" (UID: \"af099159-41cf-43d9-8ffb-6e5fb9aeea23\") " pod="openshift-cluster-node-tuning-operator/tuned-6mr4s"
Sep 09 20:52:39.407995 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.407690    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/af099159-41cf-43d9-8ffb-6e5fb9aeea23-var-lib-kubelet\") pod \"tuned-6mr4s\" (UID: \"af099159-41cf-43d9-8ffb-6e5fb9aeea23\") " pod="openshift-cluster-node-tuning-operator/tuned-6mr4s"
Sep 09 20:52:39.407995 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.407715    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f30a6570-2579-4271-ba77-d75fd2fd8390-agent-certs\") pod \"konnectivity-agent-5v6vg\" (UID: \"f30a6570-2579-4271-ba77-d75fd2fd8390\") " pod="kube-system/konnectivity-agent-5v6vg"
Sep 09 20:52:39.407995 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.407766    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/855262cd-a32b-4283-9482-cd25b26b0433-host-var-lib-cni-multus\") pod \"multus-5vzc7\" (UID: \"855262cd-a32b-4283-9482-cd25b26b0433\") " pod="openshift-multus/multus-5vzc7"
Sep 09 20:52:39.407995 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.407766    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/af099159-41cf-43d9-8ffb-6e5fb9aeea23-etc-modprobe-d\") pod \"tuned-6mr4s\" (UID: \"af099159-41cf-43d9-8ffb-6e5fb9aeea23\") " pod="openshift-cluster-node-tuning-operator/tuned-6mr4s"
Sep 09 20:52:39.408851 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.407818    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/af099159-41cf-43d9-8ffb-6e5fb9aeea23-etc-kubernetes\") pod \"tuned-6mr4s\" (UID: \"af099159-41cf-43d9-8ffb-6e5fb9aeea23\") " pod="openshift-cluster-node-tuning-operator/tuned-6mr4s"
Sep 09 20:52:39.408851 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.407824    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/af099159-41cf-43d9-8ffb-6e5fb9aeea23-var-lib-kubelet\") pod \"tuned-6mr4s\" (UID: \"af099159-41cf-43d9-8ffb-6e5fb9aeea23\") " pod="openshift-cluster-node-tuning-operator/tuned-6mr4s"
Sep 09 20:52:39.408851 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.407857    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f0db116c-804d-4ce3-b4f9-135288ad7cea-host-kubelet\") pod \"ovnkube-node-tkbs9\" (UID: \"f0db116c-804d-4ce3-b4f9-135288ad7cea\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkbs9"
Sep 09 20:52:39.408851 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.407884    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f0db116c-804d-4ce3-b4f9-135288ad7cea-ovn-node-metrics-cert\") pod \"ovnkube-node-tkbs9\" (UID: \"f0db116c-804d-4ce3-b4f9-135288ad7cea\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkbs9"
Sep 09 20:52:39.408851 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.407912    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f0db116c-804d-4ce3-b4f9-135288ad7cea-ovnkube-script-lib\") pod \"ovnkube-node-tkbs9\" (UID: \"f0db116c-804d-4ce3-b4f9-135288ad7cea\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkbs9"
Sep 09 20:52:39.408851 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.407939    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3ede0a95-3907-4d9c-aa45-acf822104e28-system-cni-dir\") pod \"multus-additional-cni-plugins-d8z7g\" (UID: \"3ede0a95-3907-4d9c-aa45-acf822104e28\") " pod="openshift-multus/multus-additional-cni-plugins-d8z7g"
Sep 09 20:52:39.408851 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.407975    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3ede0a95-3907-4d9c-aa45-acf822104e28-tuning-conf-dir\") pod \"multus-additional-cni-plugins-d8z7g\" (UID: \"3ede0a95-3907-4d9c-aa45-acf822104e28\") " pod="openshift-multus/multus-additional-cni-plugins-d8z7g"
Sep 09 20:52:39.408851 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.408033    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68xzx\" (UniqueName: \"kubernetes.io/projected/af099159-41cf-43d9-8ffb-6e5fb9aeea23-kube-api-access-68xzx\") pod \"tuned-6mr4s\" (UID: \"af099159-41cf-43d9-8ffb-6e5fb9aeea23\") " pod="openshift-cluster-node-tuning-operator/tuned-6mr4s"
Sep 09 20:52:39.408851 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.408075    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4z2xp\" (UniqueName: \"kubernetes.io/projected/5b815b2b-983b-4afe-900d-13dd791e4778-kube-api-access-4z2xp\") pod \"network-metrics-daemon-v74xn\" (UID: \"5b815b2b-983b-4afe-900d-13dd791e4778\") " pod="openshift-multus/network-metrics-daemon-v74xn"
Sep 09 20:52:39.408851 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.408037    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3ede0a95-3907-4d9c-aa45-acf822104e28-system-cni-dir\") pod \"multus-additional-cni-plugins-d8z7g\" (UID: \"3ede0a95-3907-4d9c-aa45-acf822104e28\") " pod="openshift-multus/multus-additional-cni-plugins-d8z7g"
Sep 09 20:52:39.408851 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.408106    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f0db116c-804d-4ce3-b4f9-135288ad7cea-etc-openvswitch\") pod \"ovnkube-node-tkbs9\" (UID: \"f0db116c-804d-4ce3-b4f9-135288ad7cea\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkbs9"
Sep 09 20:52:39.408851 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.408134    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f0db116c-804d-4ce3-b4f9-135288ad7cea-env-overrides\") pod \"ovnkube-node-tkbs9\" (UID: \"f0db116c-804d-4ce3-b4f9-135288ad7cea\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkbs9"
Sep 09 20:52:39.408851 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.408160    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3ede0a95-3907-4d9c-aa45-acf822104e28-tuning-conf-dir\") pod \"multus-additional-cni-plugins-d8z7g\" (UID: \"3ede0a95-3907-4d9c-aa45-acf822104e28\") " pod="openshift-multus/multus-additional-cni-plugins-d8z7g"
Sep 09 20:52:39.408851 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.408210    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b3a7af8e-62e0-43d9-8845-a71f6b10c551-device-dir\") pod \"aws-ebs-csi-driver-node-5vpsh\" (UID: \"b3a7af8e-62e0-43d9-8845-a71f6b10c551\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5vpsh"
Sep 09 20:52:39.408851 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.408289    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/af099159-41cf-43d9-8ffb-6e5fb9aeea23-run\") pod \"tuned-6mr4s\" (UID: \"af099159-41cf-43d9-8ffb-6e5fb9aeea23\") " pod="openshift-cluster-node-tuning-operator/tuned-6mr4s"
Sep 09 20:52:39.408851 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.408307    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/855262cd-a32b-4283-9482-cd25b26b0433-host-var-lib-kubelet\") pod \"multus-5vzc7\" (UID: \"855262cd-a32b-4283-9482-cd25b26b0433\") " pod="openshift-multus/multus-5vzc7"
Sep 09 20:52:39.408851 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.408332    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/855262cd-a32b-4283-9482-cd25b26b0433-multus-daemon-config\") pod \"multus-5vzc7\" (UID: \"855262cd-a32b-4283-9482-cd25b26b0433\") " pod="openshift-multus/multus-5vzc7"
Sep 09 20:52:39.409718 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.408364    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f0db116c-804d-4ce3-b4f9-135288ad7cea-host-slash\") pod \"ovnkube-node-tkbs9\" (UID: \"f0db116c-804d-4ce3-b4f9-135288ad7cea\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkbs9"
Sep 09 20:52:39.409718 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.408415    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxn2t\" (UniqueName: \"kubernetes.io/projected/b3a7af8e-62e0-43d9-8845-a71f6b10c551-kube-api-access-fxn2t\") pod \"aws-ebs-csi-driver-node-5vpsh\" (UID: \"b3a7af8e-62e0-43d9-8845-a71f6b10c551\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5vpsh"
Sep 09 20:52:39.409718 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.408438    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/af099159-41cf-43d9-8ffb-6e5fb9aeea23-run\") pod \"tuned-6mr4s\" (UID: \"af099159-41cf-43d9-8ffb-6e5fb9aeea23\") " pod="openshift-cluster-node-tuning-operator/tuned-6mr4s"
Sep 09 20:52:39.409718 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.408454    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/af099159-41cf-43d9-8ffb-6e5fb9aeea23-sys\") pod \"tuned-6mr4s\" (UID: \"af099159-41cf-43d9-8ffb-6e5fb9aeea23\") " pod="openshift-cluster-node-tuning-operator/tuned-6mr4s"
Sep 09 20:52:39.409718 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.408496    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2x8m7\" (UniqueName: \"kubernetes.io/projected/3ede0a95-3907-4d9c-aa45-acf822104e28-kube-api-access-2x8m7\") pod \"multus-additional-cni-plugins-d8z7g\" (UID: \"3ede0a95-3907-4d9c-aa45-acf822104e28\") " pod="openshift-multus/multus-additional-cni-plugins-d8z7g"
Sep 09 20:52:39.409718 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.408512    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/af099159-41cf-43d9-8ffb-6e5fb9aeea23-sys\") pod \"tuned-6mr4s\" (UID: \"af099159-41cf-43d9-8ffb-6e5fb9aeea23\") " pod="openshift-cluster-node-tuning-operator/tuned-6mr4s"
Sep 09 20:52:39.409718 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.408523    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/af099159-41cf-43d9-8ffb-6e5fb9aeea23-etc-systemd\") pod \"tuned-6mr4s\" (UID: \"af099159-41cf-43d9-8ffb-6e5fb9aeea23\") " pod="openshift-cluster-node-tuning-operator/tuned-6mr4s"
Sep 09 20:52:39.409718 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.408548    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b906e5b3-61d3-41da-9776-795376a73811-kubelet-config\") pod \"global-pull-secret-syncer-2c9kq\" (UID: \"b906e5b3-61d3-41da-9776-795376a73811\") " pod="kube-system/global-pull-secret-syncer-2c9kq"
Sep 09 20:52:39.409718 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.408564    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f0db116c-804d-4ce3-b4f9-135288ad7cea-host-run-netns\") pod \"ovnkube-node-tkbs9\" (UID: \"f0db116c-804d-4ce3-b4f9-135288ad7cea\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkbs9"
Sep 09 20:52:39.409718 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.408583    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdk6m\" (UniqueName: \"kubernetes.io/projected/f0db116c-804d-4ce3-b4f9-135288ad7cea-kube-api-access-gdk6m\") pod \"ovnkube-node-tkbs9\" (UID: \"f0db116c-804d-4ce3-b4f9-135288ad7cea\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkbs9"
Sep 09 20:52:39.409718 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.408620    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/af099159-41cf-43d9-8ffb-6e5fb9aeea23-etc-systemd\") pod \"tuned-6mr4s\" (UID: \"af099159-41cf-43d9-8ffb-6e5fb9aeea23\") " pod="openshift-cluster-node-tuning-operator/tuned-6mr4s"
Sep 09 20:52:39.409718 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.408688    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/855262cd-a32b-4283-9482-cd25b26b0433-host-run-netns\") pod \"multus-5vzc7\" (UID: \"855262cd-a32b-4283-9482-cd25b26b0433\") " pod="openshift-multus/multus-5vzc7"
Sep 09 20:52:39.409718 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.408738    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/855262cd-a32b-4283-9482-cd25b26b0433-host-run-multus-certs\") pod \"multus-5vzc7\" (UID: \"855262cd-a32b-4283-9482-cd25b26b0433\") " pod="openshift-multus/multus-5vzc7"
Sep 09 20:52:39.409718 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.408781    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b3a7af8e-62e0-43d9-8845-a71f6b10c551-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5vpsh\" (UID: \"b3a7af8e-62e0-43d9-8845-a71f6b10c551\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5vpsh"
Sep 09 20:52:39.409718 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.408808    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e706fb03-415d-407d-b105-bd06474f6882-host\") pod \"node-ca-dcv48\" (UID: \"e706fb03-415d-407d-b105-bd06474f6882\") " pod="openshift-image-registry/node-ca-dcv48"
Sep 09 20:52:39.409718 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.408862    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/af099159-41cf-43d9-8ffb-6e5fb9aeea23-etc-sysctl-d\") pod \"tuned-6mr4s\" (UID: \"af099159-41cf-43d9-8ffb-6e5fb9aeea23\") " pod="openshift-cluster-node-tuning-operator/tuned-6mr4s"
Sep 09 20:52:39.409718 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.408931    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e706fb03-415d-407d-b105-bd06474f6882-host\") pod \"node-ca-dcv48\" (UID: \"e706fb03-415d-407d-b105-bd06474f6882\") " pod="openshift-image-registry/node-ca-dcv48"
Sep 09 20:52:39.410525 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.408956    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/855262cd-a32b-4283-9482-cd25b26b0433-system-cni-dir\") pod \"multus-5vzc7\" (UID: \"855262cd-a32b-4283-9482-cd25b26b0433\") " pod="openshift-multus/multus-5vzc7"
Sep 09 20:52:39.410525 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.408971    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/af099159-41cf-43d9-8ffb-6e5fb9aeea23-etc-sysctl-d\") pod \"tuned-6mr4s\" (UID: \"af099159-41cf-43d9-8ffb-6e5fb9aeea23\") " pod="openshift-cluster-node-tuning-operator/tuned-6mr4s"
Sep 09 20:52:39.410525 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.408976    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/855262cd-a32b-4283-9482-cd25b26b0433-cni-binary-copy\") pod \"multus-5vzc7\" (UID: \"855262cd-a32b-4283-9482-cd25b26b0433\") " pod="openshift-multus/multus-5vzc7"
Sep 09 20:52:39.410525 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.408990    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b906e5b3-61d3-41da-9776-795376a73811-dbus\") pod \"global-pull-secret-syncer-2c9kq\" (UID: \"b906e5b3-61d3-41da-9776-795376a73811\") " pod="kube-system/global-pull-secret-syncer-2c9kq"
Sep 09 20:52:39.410525 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.409008    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/af099159-41cf-43d9-8ffb-6e5fb9aeea23-tmp\") pod \"tuned-6mr4s\" (UID: \"af099159-41cf-43d9-8ffb-6e5fb9aeea23\") " pod="openshift-cluster-node-tuning-operator/tuned-6mr4s"
Sep 09 20:52:39.410525 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.409023    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f0db116c-804d-4ce3-b4f9-135288ad7cea-run-ovn\") pod \"ovnkube-node-tkbs9\" (UID: \"f0db116c-804d-4ce3-b4f9-135288ad7cea\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkbs9"
Sep 09 20:52:39.410525 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.409053    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b3a7af8e-62e0-43d9-8845-a71f6b10c551-registration-dir\") pod \"aws-ebs-csi-driver-node-5vpsh\" (UID: \"b3a7af8e-62e0-43d9-8845-a71f6b10c551\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5vpsh"
Sep 09 20:52:39.410525 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.409485    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-72.ec2.internal" event={"ID":"8d0d54b6d026f1113a292c2b25b7bd73","Type":"ContainerStarted","Data":"0a8a7e9c707607796cda15ec2971201166d4c9ef92705bf41d654f48578d3fb9"}
Sep 09 20:52:39.410916 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.410806    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-72.ec2.internal" event={"ID":"2d40b80525651e1d9758e04c441ca328","Type":"ContainerStarted","Data":"c7be44e3480c59861d22979e8c1c797eb379373f90a5876a105675800f0549c7"}
Sep 09 20:52:39.411197 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.411173    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/af099159-41cf-43d9-8ffb-6e5fb9aeea23-etc-tuned\") pod \"tuned-6mr4s\" (UID: \"af099159-41cf-43d9-8ffb-6e5fb9aeea23\") " pod="openshift-cluster-node-tuning-operator/tuned-6mr4s"
Sep 09 20:52:39.412012 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.411989    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/af099159-41cf-43d9-8ffb-6e5fb9aeea23-tmp\") pod \"tuned-6mr4s\" (UID: \"af099159-41cf-43d9-8ffb-6e5fb9aeea23\") " pod="openshift-cluster-node-tuning-operator/tuned-6mr4s"
Sep 09 20:52:39.415419 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.415391    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qbd6\" (UniqueName: \"kubernetes.io/projected/cc3e4e7d-c7ef-462c-963a-bbcb4931a486-kube-api-access-6qbd6\") pod \"iptables-alerter-x9txm\" (UID: \"cc3e4e7d-c7ef-462c-963a-bbcb4931a486\") " pod="openshift-network-operator/iptables-alerter-x9txm"
Sep 09 20:52:39.415562 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:52:39.415527    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 20:52:39.415562 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:52:39.415549    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 20:52:39.415562 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:52:39.415563    2576 projected.go:196] Error preparing data for projected volume kube-api-access-krdp6 for pod openshift-network-diagnostics/network-check-target-t8tk8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:52:39.415748 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.415584    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7f9rv\" (UniqueName: \"kubernetes.io/projected/09f40168-2bb2-4eff-8a3f-7ef28a149eed-kube-api-access-7f9rv\") pod \"node-resolver-wvw9h\" (UID: \"09f40168-2bb2-4eff-8a3f-7ef28a149eed\") " pod="openshift-dns/node-resolver-wvw9h"
Sep 09 20:52:39.415748 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:52:39.415643    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/82d55335-0036-4166-a35c-65adec33932d-kube-api-access-krdp6 podName:82d55335-0036-4166-a35c-65adec33932d nodeName:}" failed. No retries permitted until 2026-09-09 20:52:39.915606535 +0000 UTC m=+3.029292380 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-krdp6" (UniqueName: "kubernetes.io/projected/82d55335-0036-4166-a35c-65adec33932d-kube-api-access-krdp6") pod "network-check-target-t8tk8" (UID: "82d55335-0036-4166-a35c-65adec33932d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:52:39.416120 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.416101    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xk99w\" (UniqueName: \"kubernetes.io/projected/e706fb03-415d-407d-b105-bd06474f6882-kube-api-access-xk99w\") pod \"node-ca-dcv48\" (UID: \"e706fb03-415d-407d-b105-bd06474f6882\") " pod="openshift-image-registry/node-ca-dcv48"
Sep 09 20:52:39.416554 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.416493    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-68xzx\" (UniqueName: \"kubernetes.io/projected/af099159-41cf-43d9-8ffb-6e5fb9aeea23-kube-api-access-68xzx\") pod \"tuned-6mr4s\" (UID: \"af099159-41cf-43d9-8ffb-6e5fb9aeea23\") " pod="openshift-cluster-node-tuning-operator/tuned-6mr4s"
Sep 09 20:52:39.418361 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.418323    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2x8m7\" (UniqueName: \"kubernetes.io/projected/3ede0a95-3907-4d9c-aa45-acf822104e28-kube-api-access-2x8m7\") pod \"multus-additional-cni-plugins-d8z7g\" (UID: \"3ede0a95-3907-4d9c-aa45-acf822104e28\") " pod="openshift-multus/multus-additional-cni-plugins-d8z7g"
Sep 09 20:52:39.509377 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.509320    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f0db116c-804d-4ce3-b4f9-135288ad7cea-host-run-ovn-kubernetes\") pod \"ovnkube-node-tkbs9\" (UID: \"f0db116c-804d-4ce3-b4f9-135288ad7cea\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkbs9"
Sep 09 20:52:39.509377 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.509373    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f0db116c-804d-4ce3-b4f9-135288ad7cea-host-cni-bin\") pod \"ovnkube-node-tkbs9\" (UID: \"f0db116c-804d-4ce3-b4f9-135288ad7cea\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkbs9"
Sep 09 20:52:39.509646 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.509391    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f0db116c-804d-4ce3-b4f9-135288ad7cea-host-run-ovn-kubernetes\") pod \"ovnkube-node-tkbs9\" (UID: \"f0db116c-804d-4ce3-b4f9-135288ad7cea\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkbs9"
Sep 09 20:52:39.509646 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.509418    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b815b2b-983b-4afe-900d-13dd791e4778-metrics-certs\") pod \"network-metrics-daemon-v74xn\" (UID: \"5b815b2b-983b-4afe-900d-13dd791e4778\") " pod="openshift-multus/network-metrics-daemon-v74xn"
Sep 09 20:52:39.509646 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.509446    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/855262cd-a32b-4283-9482-cd25b26b0433-hostroot\") pod \"multus-5vzc7\" (UID: \"855262cd-a32b-4283-9482-cd25b26b0433\") " pod="openshift-multus/multus-5vzc7"
Sep 09 20:52:39.509646 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.509455    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f0db116c-804d-4ce3-b4f9-135288ad7cea-host-cni-bin\") pod \"ovnkube-node-tkbs9\" (UID: \"f0db116c-804d-4ce3-b4f9-135288ad7cea\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkbs9"
Sep 09 20:52:39.509646 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.509472    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f0db116c-804d-4ce3-b4f9-135288ad7cea-run-openvswitch\") pod \"ovnkube-node-tkbs9\" (UID: \"f0db116c-804d-4ce3-b4f9-135288ad7cea\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkbs9"
Sep 09 20:52:39.509646 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.509494    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f0db116c-804d-4ce3-b4f9-135288ad7cea-node-log\") pod \"ovnkube-node-tkbs9\" (UID: \"f0db116c-804d-4ce3-b4f9-135288ad7cea\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkbs9"
Sep 09 20:52:39.509646 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:52:39.509526    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:52:39.509646 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.509533    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f0db116c-804d-4ce3-b4f9-135288ad7cea-node-log\") pod \"ovnkube-node-tkbs9\" (UID: \"f0db116c-804d-4ce3-b4f9-135288ad7cea\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkbs9"
Sep 09 20:52:39.509646 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.509532    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/855262cd-a32b-4283-9482-cd25b26b0433-hostroot\") pod \"multus-5vzc7\" (UID: \"855262cd-a32b-4283-9482-cd25b26b0433\") " pod="openshift-multus/multus-5vzc7"
Sep 09 20:52:39.509646 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.509558    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f0db116c-804d-4ce3-b4f9-135288ad7cea-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tkbs9\" (UID: \"f0db116c-804d-4ce3-b4f9-135288ad7cea\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkbs9"
Sep 09 20:52:39.509646 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.509576    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f0db116c-804d-4ce3-b4f9-135288ad7cea-run-openvswitch\") pod \"ovnkube-node-tkbs9\" (UID: \"f0db116c-804d-4ce3-b4f9-135288ad7cea\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkbs9"
Sep 09 20:52:39.509646 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.509604    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f0db116c-804d-4ce3-b4f9-135288ad7cea-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tkbs9\" (UID: \"f0db116c-804d-4ce3-b4f9-135288ad7cea\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkbs9"
Sep 09 20:52:39.509646 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:52:39.509616    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b815b2b-983b-4afe-900d-13dd791e4778-metrics-certs podName:5b815b2b-983b-4afe-900d-13dd791e4778 nodeName:}" failed. No retries permitted until 2026-09-09 20:52:40.009575238 +0000 UTC m=+3.123261097 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5b815b2b-983b-4afe-900d-13dd791e4778-metrics-certs") pod "network-metrics-daemon-v74xn" (UID: "5b815b2b-983b-4afe-900d-13dd791e4778") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:52:39.510206 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.509657    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f30a6570-2579-4271-ba77-d75fd2fd8390-konnectivity-ca\") pod \"konnectivity-agent-5v6vg\" (UID: \"f30a6570-2579-4271-ba77-d75fd2fd8390\") " pod="kube-system/konnectivity-agent-5v6vg"
Sep 09 20:52:39.510206 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.509684    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/855262cd-a32b-4283-9482-cd25b26b0433-multus-socket-dir-parent\") pod \"multus-5vzc7\" (UID: \"855262cd-a32b-4283-9482-cd25b26b0433\") " pod="openshift-multus/multus-5vzc7"
Sep 09 20:52:39.510206 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.509749    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/855262cd-a32b-4283-9482-cd25b26b0433-host-var-lib-cni-bin\") pod \"multus-5vzc7\" (UID: \"855262cd-a32b-4283-9482-cd25b26b0433\") " pod="openshift-multus/multus-5vzc7"
Sep 09 20:52:39.510206 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.509777    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b3a7af8e-62e0-43d9-8845-a71f6b10c551-etc-selinux\") pod \"aws-ebs-csi-driver-node-5vpsh\" (UID: \"b3a7af8e-62e0-43d9-8845-a71f6b10c551\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5vpsh"
Sep 09 20:52:39.510206 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.509801    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b3a7af8e-62e0-43d9-8845-a71f6b10c551-sys-fs\") pod \"aws-ebs-csi-driver-node-5vpsh\" (UID: \"b3a7af8e-62e0-43d9-8845-a71f6b10c551\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5vpsh"
Sep 09 20:52:39.510206 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.509827    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76vjl\" (UniqueName: \"kubernetes.io/projected/855262cd-a32b-4283-9482-cd25b26b0433-kube-api-access-76vjl\") pod \"multus-5vzc7\" (UID: \"855262cd-a32b-4283-9482-cd25b26b0433\") " pod="openshift-multus/multus-5vzc7"
Sep 09 20:52:39.510206 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.509853    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f0db116c-804d-4ce3-b4f9-135288ad7cea-systemd-units\") pod \"ovnkube-node-tkbs9\" (UID: \"f0db116c-804d-4ce3-b4f9-135288ad7cea\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkbs9"
Sep 09 20:52:39.510206 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.509876    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f0db116c-804d-4ce3-b4f9-135288ad7cea-var-lib-openvswitch\") pod \"ovnkube-node-tkbs9\" (UID: \"f0db116c-804d-4ce3-b4f9-135288ad7cea\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkbs9"
Sep 09 20:52:39.510206 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.509900    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f0db116c-804d-4ce3-b4f9-135288ad7cea-ovnkube-config\") pod \"ovnkube-node-tkbs9\" (UID: \"f0db116c-804d-4ce3-b4f9-135288ad7cea\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkbs9"
Sep 09 20:52:39.510206 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.509927    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/855262cd-a32b-4283-9482-cd25b26b0433-etc-kubernetes\") pod \"multus-5vzc7\" (UID: \"855262cd-a32b-4283-9482-cd25b26b0433\") " pod="openshift-multus/multus-5vzc7"
Sep 09 20:52:39.510206 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.509931    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b3a7af8e-62e0-43d9-8845-a71f6b10c551-etc-selinux\") pod \"aws-ebs-csi-driver-node-5vpsh\" (UID: \"b3a7af8e-62e0-43d9-8845-a71f6b10c551\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5vpsh"
Sep 09 20:52:39.510206 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.509944    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f0db116c-804d-4ce3-b4f9-135288ad7cea-run-systemd\") pod \"ovnkube-node-tkbs9\" (UID: \"f0db116c-804d-4ce3-b4f9-135288ad7cea\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkbs9"
Sep 09 20:52:39.510206 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.509971    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/855262cd-a32b-4283-9482-cd25b26b0433-multus-conf-dir\") pod \"multus-5vzc7\" (UID: \"855262cd-a32b-4283-9482-cd25b26b0433\") " pod="openshift-multus/multus-5vzc7"
Sep 09 20:52:39.510206 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.509991    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/855262cd-a32b-4283-9482-cd25b26b0433-host-var-lib-cni-bin\") pod \"multus-5vzc7\" (UID: \"855262cd-a32b-4283-9482-cd25b26b0433\") " pod="openshift-multus/multus-5vzc7"
Sep 09 20:52:39.510206 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.509987    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f0db116c-804d-4ce3-b4f9-135288ad7cea-host-cni-netd\") pod \"ovnkube-node-tkbs9\" (UID: \"f0db116c-804d-4ce3-b4f9-135288ad7cea\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkbs9"
Sep 09 20:52:39.510206 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.510012    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b3a7af8e-62e0-43d9-8845-a71f6b10c551-socket-dir\") pod \"aws-ebs-csi-driver-node-5vpsh\" (UID: \"b3a7af8e-62e0-43d9-8845-a71f6b10c551\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5vpsh"
Sep 09 20:52:39.510206 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.510036    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f30a6570-2579-4271-ba77-d75fd2fd8390-agent-certs\") pod \"konnectivity-agent-5v6vg\" (UID: \"f30a6570-2579-4271-ba77-d75fd2fd8390\") " pod="kube-system/konnectivity-agent-5v6vg"
Sep 09 20:52:39.510774 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.510052    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/855262cd-a32b-4283-9482-cd25b26b0433-host-var-lib-cni-multus\") pod \"multus-5vzc7\" (UID: \"855262cd-a32b-4283-9482-cd25b26b0433\") " pod="openshift-multus/multus-5vzc7"
Sep 09 20:52:39.510774 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.510068    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f0db116c-804d-4ce3-b4f9-135288ad7cea-host-kubelet\") pod \"ovnkube-node-tkbs9\" (UID: \"f0db116c-804d-4ce3-b4f9-135288ad7cea\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkbs9"
Sep 09 20:52:39.510774 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.510082    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f0db116c-804d-4ce3-b4f9-135288ad7cea-ovn-node-metrics-cert\") pod \"ovnkube-node-tkbs9\" (UID: \"f0db116c-804d-4ce3-b4f9-135288ad7cea\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkbs9"
Sep 09 20:52:39.510774 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.510099    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f0db116c-804d-4ce3-b4f9-135288ad7cea-ovnkube-script-lib\") pod \"ovnkube-node-tkbs9\" (UID: \"f0db116c-804d-4ce3-b4f9-135288ad7cea\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkbs9"
Sep 09 20:52:39.510774 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.510122    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4z2xp\" (UniqueName: \"kubernetes.io/projected/5b815b2b-983b-4afe-900d-13dd791e4778-kube-api-access-4z2xp\") pod \"network-metrics-daemon-v74xn\" (UID: \"5b815b2b-983b-4afe-900d-13dd791e4778\") " pod="openshift-multus/network-metrics-daemon-v74xn"
Sep 09 20:52:39.510774 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.510138    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f0db116c-804d-4ce3-b4f9-135288ad7cea-etc-openvswitch\") pod \"ovnkube-node-tkbs9\" (UID: \"f0db116c-804d-4ce3-b4f9-135288ad7cea\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkbs9"
Sep 09 20:52:39.510774 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.510153    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f0db116c-804d-4ce3-b4f9-135288ad7cea-env-overrides\") pod \"ovnkube-node-tkbs9\" (UID: \"f0db116c-804d-4ce3-b4f9-135288ad7cea\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkbs9"
Sep 09 20:52:39.510774 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.509970    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/855262cd-a32b-4283-9482-cd25b26b0433-multus-socket-dir-parent\") pod \"multus-5vzc7\" (UID: \"855262cd-a32b-4283-9482-cd25b26b0433\") " pod="openshift-multus/multus-5vzc7"
Sep 09 20:52:39.510774 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.510175    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b3a7af8e-62e0-43d9-8845-a71f6b10c551-device-dir\") pod \"aws-ebs-csi-driver-node-5vpsh\" (UID: \"b3a7af8e-62e0-43d9-8845-a71f6b10c551\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5vpsh"
Sep 09 20:52:39.510774 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.510199    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b3a7af8e-62e0-43d9-8845-a71f6b10c551-sys-fs\") pod \"aws-ebs-csi-driver-node-5vpsh\" (UID: \"b3a7af8e-62e0-43d9-8845-a71f6b10c551\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5vpsh"
Sep 09 20:52:39.510774 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.510202    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/855262cd-a32b-4283-9482-cd25b26b0433-host-var-lib-kubelet\") pod \"multus-5vzc7\" (UID: \"855262cd-a32b-4283-9482-cd25b26b0433\") " pod="openshift-multus/multus-5vzc7"
Sep 09 20:52:39.510774 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.510217    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f30a6570-2579-4271-ba77-d75fd2fd8390-konnectivity-ca\") pod \"konnectivity-agent-5v6vg\" (UID: \"f30a6570-2579-4271-ba77-d75fd2fd8390\") " pod="kube-system/konnectivity-agent-5v6vg"
Sep 09 20:52:39.510774 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.510221    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/855262cd-a32b-4283-9482-cd25b26b0433-multus-daemon-config\") pod \"multus-5vzc7\" (UID: \"855262cd-a32b-4283-9482-cd25b26b0433\") " pod="openshift-multus/multus-5vzc7"
Sep 09 20:52:39.510774 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.510264    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f0db116c-804d-4ce3-b4f9-135288ad7cea-host-slash\") pod \"ovnkube-node-tkbs9\" (UID: \"f0db116c-804d-4ce3-b4f9-135288ad7cea\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkbs9"
Sep 09 20:52:39.510774 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.510285    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxn2t\" (UniqueName: \"kubernetes.io/projected/b3a7af8e-62e0-43d9-8845-a71f6b10c551-kube-api-access-fxn2t\") pod \"aws-ebs-csi-driver-node-5vpsh\" (UID: \"b3a7af8e-62e0-43d9-8845-a71f6b10c551\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5vpsh"
Sep 09 20:52:39.510774 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.510306    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b906e5b3-61d3-41da-9776-795376a73811-kubelet-config\") pod \"global-pull-secret-syncer-2c9kq\" (UID: \"b906e5b3-61d3-41da-9776-795376a73811\") " pod="kube-system/global-pull-secret-syncer-2c9kq"
Sep 09 20:52:39.510774 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.510322    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f0db116c-804d-4ce3-b4f9-135288ad7cea-host-run-netns\") pod \"ovnkube-node-tkbs9\" (UID: \"f0db116c-804d-4ce3-b4f9-135288ad7cea\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkbs9"
Sep 09 20:52:39.511822 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.510369    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdk6m\" (UniqueName: \"kubernetes.io/projected/f0db116c-804d-4ce3-b4f9-135288ad7cea-kube-api-access-gdk6m\") pod \"ovnkube-node-tkbs9\" (UID: \"f0db116c-804d-4ce3-b4f9-135288ad7cea\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkbs9"
Sep 09 20:52:39.511822 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.510371    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f0db116c-804d-4ce3-b4f9-135288ad7cea-systemd-units\") pod \"ovnkube-node-tkbs9\" (UID: \"f0db116c-804d-4ce3-b4f9-135288ad7cea\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkbs9"
Sep 09 20:52:39.511822 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.510395    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/855262cd-a32b-4283-9482-cd25b26b0433-host-run-netns\") pod \"multus-5vzc7\" (UID: \"855262cd-a32b-4283-9482-cd25b26b0433\") " pod="openshift-multus/multus-5vzc7"
Sep 09 20:52:39.511822 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.510406    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f0db116c-804d-4ce3-b4f9-135288ad7cea-host-slash\") pod \"ovnkube-node-tkbs9\" (UID: \"f0db116c-804d-4ce3-b4f9-135288ad7cea\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkbs9"
Sep 09 20:52:39.511822 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.510415    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/855262cd-a32b-4283-9482-cd25b26b0433-host-run-multus-certs\") pod \"multus-5vzc7\" (UID: \"855262cd-a32b-4283-9482-cd25b26b0433\") " pod="openshift-multus/multus-5vzc7"
Sep 09 20:52:39.511822 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.510432    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b3a7af8e-62e0-43d9-8845-a71f6b10c551-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5vpsh\" (UID: \"b3a7af8e-62e0-43d9-8845-a71f6b10c551\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5vpsh"
Sep 09 20:52:39.511822 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.510469    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/855262cd-a32b-4283-9482-cd25b26b0433-system-cni-dir\") pod \"multus-5vzc7\" (UID: \"855262cd-a32b-4283-9482-cd25b26b0433\") " pod="openshift-multus/multus-5vzc7"
Sep 09 20:52:39.511822 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.510492    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/855262cd-a32b-4283-9482-cd25b26b0433-cni-binary-copy\") pod \"multus-5vzc7\" (UID: \"855262cd-a32b-4283-9482-cd25b26b0433\") " pod="openshift-multus/multus-5vzc7"
Sep 09 20:52:39.511822 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.510513    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b906e5b3-61d3-41da-9776-795376a73811-dbus\") pod \"global-pull-secret-syncer-2c9kq\" (UID: \"b906e5b3-61d3-41da-9776-795376a73811\") " pod="kube-system/global-pull-secret-syncer-2c9kq"
Sep 09 20:52:39.511822 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.510538    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f0db116c-804d-4ce3-b4f9-135288ad7cea-run-ovn\") pod \"ovnkube-node-tkbs9\" (UID: \"f0db116c-804d-4ce3-b4f9-135288ad7cea\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkbs9"
Sep 09 20:52:39.511822 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.510566    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b3a7af8e-62e0-43d9-8845-a71f6b10c551-registration-dir\") pod \"aws-ebs-csi-driver-node-5vpsh\" (UID: \"b3a7af8e-62e0-43d9-8845-a71f6b10c551\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5vpsh"
Sep 09 20:52:39.511822 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.510563    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/855262cd-a32b-4283-9482-cd25b26b0433-multus-conf-dir\") pod \"multus-5vzc7\" (UID: \"855262cd-a32b-4283-9482-cd25b26b0433\") " pod="openshift-multus/multus-5vzc7"
Sep 09 20:52:39.511822 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.510599    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/855262cd-a32b-4283-9482-cd25b26b0433-multus-cni-dir\") pod \"multus-5vzc7\" (UID: \"855262cd-a32b-4283-9482-cd25b26b0433\") " pod="openshift-multus/multus-5vzc7"
Sep 09 20:52:39.511822 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.510625    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/855262cd-a32b-4283-9482-cd25b26b0433-host-run-k8s-cni-cncf-io\") pod \"multus-5vzc7\" (UID: \"855262cd-a32b-4283-9482-cd25b26b0433\") " pod="openshift-multus/multus-5vzc7"
Sep 09 20:52:39.511822 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.510630    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f0db116c-804d-4ce3-b4f9-135288ad7cea-var-lib-openvswitch\") pod \"ovnkube-node-tkbs9\" (UID: \"f0db116c-804d-4ce3-b4f9-135288ad7cea\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkbs9"
Sep 09 20:52:39.511822 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.510684    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/855262cd-a32b-4283-9482-cd25b26b0433-host-run-k8s-cni-cncf-io\") pod \"multus-5vzc7\" (UID: \"855262cd-a32b-4283-9482-cd25b26b0433\") " pod="openshift-multus/multus-5vzc7"
Sep 09 20:52:39.511822 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.510731    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/855262cd-a32b-4283-9482-cd25b26b0433-etc-kubernetes\") pod \"multus-5vzc7\" (UID: \"855262cd-a32b-4283-9482-cd25b26b0433\") " pod="openshift-multus/multus-5vzc7"
Sep 09 20:52:39.511822 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.510736    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f0db116c-804d-4ce3-b4f9-135288ad7cea-run-systemd\") pod \"ovnkube-node-tkbs9\" (UID: \"f0db116c-804d-4ce3-b4f9-135288ad7cea\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkbs9"
Sep 09 20:52:39.512737 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.510687    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b906e5b3-61d3-41da-9776-795376a73811-kubelet-config\") pod \"global-pull-secret-syncer-2c9kq\" (UID: \"b906e5b3-61d3-41da-9776-795376a73811\") " pod="kube-system/global-pull-secret-syncer-2c9kq"
Sep 09 20:52:39.512737 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.510748    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/855262cd-a32b-4283-9482-cd25b26b0433-multus-daemon-config\") pod \"multus-5vzc7\" (UID: \"855262cd-a32b-4283-9482-cd25b26b0433\") " pod="openshift-multus/multus-5vzc7"
Sep 09 20:52:39.512737 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.510767    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f0db116c-804d-4ce3-b4f9-135288ad7cea-log-socket\") pod \"ovnkube-node-tkbs9\" (UID: \"f0db116c-804d-4ce3-b4f9-135288ad7cea\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkbs9"
Sep 09 20:52:39.512737 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.510802    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/855262cd-a32b-4283-9482-cd25b26b0433-cnibin\") pod \"multus-5vzc7\" (UID: \"855262cd-a32b-4283-9482-cd25b26b0433\") " pod="openshift-multus/multus-5vzc7"
Sep 09 20:52:39.512737 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.510829    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b906e5b3-61d3-41da-9776-795376a73811-original-pull-secret\") pod \"global-pull-secret-syncer-2c9kq\" (UID: \"b906e5b3-61d3-41da-9776-795376a73811\") " pod="kube-system/global-pull-secret-syncer-2c9kq"
Sep 09 20:52:39.512737 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.510875    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/855262cd-a32b-4283-9482-cd25b26b0433-host-run-netns\") pod \"multus-5vzc7\" (UID: \"855262cd-a32b-4283-9482-cd25b26b0433\") " pod="openshift-multus/multus-5vzc7"
Sep 09 20:52:39.512737 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.510924    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/855262cd-a32b-4283-9482-cd25b26b0433-host-run-multus-certs\") pod \"multus-5vzc7\" (UID: \"855262cd-a32b-4283-9482-cd25b26b0433\") " pod="openshift-multus/multus-5vzc7"
Sep 09 20:52:39.512737 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:52:39.510938    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 20:52:39.512737 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.510977    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b3a7af8e-62e0-43d9-8845-a71f6b10c551-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5vpsh\" (UID: \"b3a7af8e-62e0-43d9-8845-a71f6b10c551\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5vpsh"
Sep 09 20:52:39.512737 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:52:39.510999    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b906e5b3-61d3-41da-9776-795376a73811-original-pull-secret podName:b906e5b3-61d3-41da-9776-795376a73811 nodeName:}" failed. No retries permitted until 2026-09-09 20:52:40.010981802 +0000 UTC m=+3.124667642 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b906e5b3-61d3-41da-9776-795376a73811-original-pull-secret") pod "global-pull-secret-syncer-2c9kq" (UID: "b906e5b3-61d3-41da-9776-795376a73811") : object "kube-system"/"original-pull-secret" not registered
Sep 09 20:52:39.512737 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.511027    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/855262cd-a32b-4283-9482-cd25b26b0433-system-cni-dir\") pod \"multus-5vzc7\" (UID: \"855262cd-a32b-4283-9482-cd25b26b0433\") " pod="openshift-multus/multus-5vzc7"
Sep 09 20:52:39.512737 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.511038    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f0db116c-804d-4ce3-b4f9-135288ad7cea-log-socket\") pod \"ovnkube-node-tkbs9\" (UID: \"f0db116c-804d-4ce3-b4f9-135288ad7cea\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkbs9"
Sep 09 20:52:39.512737 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.511096    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b906e5b3-61d3-41da-9776-795376a73811-dbus\") pod \"global-pull-secret-syncer-2c9kq\" (UID: \"b906e5b3-61d3-41da-9776-795376a73811\") " pod="kube-system/global-pull-secret-syncer-2c9kq"
Sep 09 20:52:39.512737 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.511136    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f0db116c-804d-4ce3-b4f9-135288ad7cea-run-ovn\") pod \"ovnkube-node-tkbs9\" (UID: \"f0db116c-804d-4ce3-b4f9-135288ad7cea\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkbs9"
Sep 09 20:52:39.512737 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.511180    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b3a7af8e-62e0-43d9-8845-a71f6b10c551-registration-dir\") pod \"aws-ebs-csi-driver-node-5vpsh\" (UID: \"b3a7af8e-62e0-43d9-8845-a71f6b10c551\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5vpsh"
Sep 09 20:52:39.512737 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.511218    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/855262cd-a32b-4283-9482-cd25b26b0433-multus-cni-dir\") pod \"multus-5vzc7\" (UID: \"855262cd-a32b-4283-9482-cd25b26b0433\") " pod="openshift-multus/multus-5vzc7"
Sep 09 20:52:39.512737 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.511257    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f0db116c-804d-4ce3-b4f9-135288ad7cea-host-run-netns\") pod \"ovnkube-node-tkbs9\" (UID: \"f0db116c-804d-4ce3-b4f9-135288ad7cea\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkbs9"
Sep 09 20:52:39.513532 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.511281    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f0db116c-804d-4ce3-b4f9-135288ad7cea-host-cni-netd\") pod \"ovnkube-node-tkbs9\" (UID: \"f0db116c-804d-4ce3-b4f9-135288ad7cea\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkbs9"
Sep 09 20:52:39.513532 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.511303    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f0db116c-804d-4ce3-b4f9-135288ad7cea-ovnkube-config\") pod \"ovnkube-node-tkbs9\" (UID: \"f0db116c-804d-4ce3-b4f9-135288ad7cea\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkbs9"
Sep 09 20:52:39.513532 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.511375    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b3a7af8e-62e0-43d9-8845-a71f6b10c551-device-dir\") pod \"aws-ebs-csi-driver-node-5vpsh\" (UID: \"b3a7af8e-62e0-43d9-8845-a71f6b10c551\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5vpsh"
Sep 09 20:52:39.513532 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.511426    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/855262cd-a32b-4283-9482-cd25b26b0433-host-var-lib-cni-multus\") pod \"multus-5vzc7\" (UID: \"855262cd-a32b-4283-9482-cd25b26b0433\") " pod="openshift-multus/multus-5vzc7"
Sep 09 20:52:39.513532 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.511427    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b3a7af8e-62e0-43d9-8845-a71f6b10c551-socket-dir\") pod \"aws-ebs-csi-driver-node-5vpsh\" (UID: \"b3a7af8e-62e0-43d9-8845-a71f6b10c551\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5vpsh"
Sep 09 20:52:39.513532 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.511462    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f0db116c-804d-4ce3-b4f9-135288ad7cea-host-kubelet\") pod \"ovnkube-node-tkbs9\" (UID: \"f0db116c-804d-4ce3-b4f9-135288ad7cea\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkbs9"
Sep 09 20:52:39.513532 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.511496    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/855262cd-a32b-4283-9482-cd25b26b0433-host-var-lib-kubelet\") pod \"multus-5vzc7\" (UID: \"855262cd-a32b-4283-9482-cd25b26b0433\") " pod="openshift-multus/multus-5vzc7"
Sep 09 20:52:39.513532 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.511508    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/855262cd-a32b-4283-9482-cd25b26b0433-cni-binary-copy\") pod \"multus-5vzc7\" (UID: \"855262cd-a32b-4283-9482-cd25b26b0433\") " pod="openshift-multus/multus-5vzc7"
Sep 09 20:52:39.513532 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.511528    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/855262cd-a32b-4283-9482-cd25b26b0433-cnibin\") pod \"multus-5vzc7\" (UID: \"855262cd-a32b-4283-9482-cd25b26b0433\") " pod="openshift-multus/multus-5vzc7"
Sep 09 20:52:39.513532 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.511572    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f0db116c-804d-4ce3-b4f9-135288ad7cea-etc-openvswitch\") pod \"ovnkube-node-tkbs9\" (UID: \"f0db116c-804d-4ce3-b4f9-135288ad7cea\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkbs9"
Sep 09 20:52:39.513532 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.512237    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f0db116c-804d-4ce3-b4f9-135288ad7cea-env-overrides\") pod \"ovnkube-node-tkbs9\" (UID: \"f0db116c-804d-4ce3-b4f9-135288ad7cea\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkbs9"
Sep 09 20:52:39.513532 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.512803    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f0db116c-804d-4ce3-b4f9-135288ad7cea-ovnkube-script-lib\") pod \"ovnkube-node-tkbs9\" (UID: \"f0db116c-804d-4ce3-b4f9-135288ad7cea\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkbs9"
Sep 09 20:52:39.514052 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.513606    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f0db116c-804d-4ce3-b4f9-135288ad7cea-ovn-node-metrics-cert\") pod \"ovnkube-node-tkbs9\" (UID: \"f0db116c-804d-4ce3-b4f9-135288ad7cea\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkbs9"
Sep 09 20:52:39.514872 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.514846    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f30a6570-2579-4271-ba77-d75fd2fd8390-agent-certs\") pod \"konnectivity-agent-5v6vg\" (UID: \"f30a6570-2579-4271-ba77-d75fd2fd8390\") " pod="kube-system/konnectivity-agent-5v6vg"
Sep 09 20:52:39.527747 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.527722    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-76vjl\" (UniqueName: \"kubernetes.io/projected/855262cd-a32b-4283-9482-cd25b26b0433-kube-api-access-76vjl\") pod \"multus-5vzc7\" (UID: \"855262cd-a32b-4283-9482-cd25b26b0433\") " pod="openshift-multus/multus-5vzc7"
Sep 09 20:52:39.528394 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.528349    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4z2xp\" (UniqueName: \"kubernetes.io/projected/5b815b2b-983b-4afe-900d-13dd791e4778-kube-api-access-4z2xp\") pod \"network-metrics-daemon-v74xn\" (UID: \"5b815b2b-983b-4afe-900d-13dd791e4778\") " pod="openshift-multus/network-metrics-daemon-v74xn"
Sep 09 20:52:39.528783 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.528757    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxn2t\" (UniqueName: \"kubernetes.io/projected/b3a7af8e-62e0-43d9-8845-a71f6b10c551-kube-api-access-fxn2t\") pod \"aws-ebs-csi-driver-node-5vpsh\" (UID: \"b3a7af8e-62e0-43d9-8845-a71f6b10c551\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5vpsh"
Sep 09 20:52:39.529808 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.529769    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdk6m\" (UniqueName: \"kubernetes.io/projected/f0db116c-804d-4ce3-b4f9-135288ad7cea-kube-api-access-gdk6m\") pod \"ovnkube-node-tkbs9\" (UID: \"f0db116c-804d-4ce3-b4f9-135288ad7cea\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkbs9"
Sep 09 20:52:39.599210 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.599131    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-d8z7g"
Sep 09 20:52:39.606312 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.606276    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-x9txm"
Sep 09 20:52:39.616844 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.616823    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-6mr4s"
Sep 09 20:52:39.623674 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.623648    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-wvw9h"
Sep 09 20:52:39.631580 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.631556    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-dcv48"
Sep 09 20:52:39.640460 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.640422    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-tkbs9"
Sep 09 20:52:39.648662 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.648635    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-5v6vg"
Sep 09 20:52:39.657037 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.657017    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5vpsh"
Sep 09 20:52:39.663989 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:39.663967    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-5vzc7"
Sep 09 20:52:40.015080 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:40.014983    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krdp6\" (UniqueName: \"kubernetes.io/projected/82d55335-0036-4166-a35c-65adec33932d-kube-api-access-krdp6\") pod \"network-check-target-t8tk8\" (UID: \"82d55335-0036-4166-a35c-65adec33932d\") " pod="openshift-network-diagnostics/network-check-target-t8tk8"
Sep 09 20:52:40.015080 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:40.015063    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b906e5b3-61d3-41da-9776-795376a73811-original-pull-secret\") pod \"global-pull-secret-syncer-2c9kq\" (UID: \"b906e5b3-61d3-41da-9776-795376a73811\") " pod="kube-system/global-pull-secret-syncer-2c9kq"
Sep 09 20:52:40.015321 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:40.015100    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b815b2b-983b-4afe-900d-13dd791e4778-metrics-certs\") pod \"network-metrics-daemon-v74xn\" (UID: \"5b815b2b-983b-4afe-900d-13dd791e4778\") " pod="openshift-multus/network-metrics-daemon-v74xn"
Sep 09 20:52:40.015321 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:52:40.015189    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 20:52:40.015321 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:52:40.015197    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:52:40.015321 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:52:40.015192    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 20:52:40.015321 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:52:40.015256    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b906e5b3-61d3-41da-9776-795376a73811-original-pull-secret podName:b906e5b3-61d3-41da-9776-795376a73811 nodeName:}" failed. No retries permitted until 2026-09-09 20:52:41.015240351 +0000 UTC m=+4.128926194 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b906e5b3-61d3-41da-9776-795376a73811-original-pull-secret") pod "global-pull-secret-syncer-2c9kq" (UID: "b906e5b3-61d3-41da-9776-795376a73811") : object "kube-system"/"original-pull-secret" not registered
Sep 09 20:52:40.015321 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:52:40.015263    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 20:52:40.015321 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:52:40.015277    2576 projected.go:196] Error preparing data for projected volume kube-api-access-krdp6 for pod openshift-network-diagnostics/network-check-target-t8tk8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:52:40.015321 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:52:40.015278    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b815b2b-983b-4afe-900d-13dd791e4778-metrics-certs podName:5b815b2b-983b-4afe-900d-13dd791e4778 nodeName:}" failed. No retries permitted until 2026-09-09 20:52:41.015266286 +0000 UTC m=+4.128952125 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5b815b2b-983b-4afe-900d-13dd791e4778-metrics-certs") pod "network-metrics-daemon-v74xn" (UID: "5b815b2b-983b-4afe-900d-13dd791e4778") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:52:40.015321 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:52:40.015317    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/82d55335-0036-4166-a35c-65adec33932d-kube-api-access-krdp6 podName:82d55335-0036-4166-a35c-65adec33932d nodeName:}" failed. No retries permitted until 2026-09-09 20:52:41.015304516 +0000 UTC m=+4.128990354 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-krdp6" (UniqueName: "kubernetes.io/projected/82d55335-0036-4166-a35c-65adec33932d-kube-api-access-krdp6") pod "network-check-target-t8tk8" (UID: "82d55335-0036-4166-a35c-65adec33932d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:52:40.352059 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:40.351953    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-08 20:47:39 +0000 UTC" deadline="2028-02-17 00:42:28.558155113 +0000 UTC"
Sep 09 20:52:40.352059 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:40.352002    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12603h49m48.206156704s"
Sep 09 20:52:40.403832 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:40.403797    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t8tk8"
Sep 09 20:52:40.404032 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:52:40.403926    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t8tk8" podUID="82d55335-0036-4166-a35c-65adec33932d"
Sep 09 20:52:40.404032 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:40.403986    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v74xn"
Sep 09 20:52:40.404162 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:52:40.404101    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v74xn" podUID="5b815b2b-983b-4afe-900d-13dd791e4778"
Sep 09 20:52:40.691164 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:52:40.691129    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0db116c_804d_4ce3_b4f9_135288ad7cea.slice/crio-bfee3f9b043cb58fd39551b6ad78ea13a5df809dd7b942cf58acd774d834e57e WatchSource:0}: Error finding container bfee3f9b043cb58fd39551b6ad78ea13a5df809dd7b942cf58acd774d834e57e: Status 404 returned error can't find the container with id bfee3f9b043cb58fd39551b6ad78ea13a5df809dd7b942cf58acd774d834e57e
Sep 09 20:52:41.023000 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:41.022968    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b906e5b3-61d3-41da-9776-795376a73811-original-pull-secret\") pod \"global-pull-secret-syncer-2c9kq\" (UID: \"b906e5b3-61d3-41da-9776-795376a73811\") " pod="kube-system/global-pull-secret-syncer-2c9kq"
Sep 09 20:52:41.023087 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:41.023017    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b815b2b-983b-4afe-900d-13dd791e4778-metrics-certs\") pod \"network-metrics-daemon-v74xn\" (UID: \"5b815b2b-983b-4afe-900d-13dd791e4778\") " pod="openshift-multus/network-metrics-daemon-v74xn"
Sep 09 20:52:41.023087 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:41.023053    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krdp6\" (UniqueName: \"kubernetes.io/projected/82d55335-0036-4166-a35c-65adec33932d-kube-api-access-krdp6\") pod \"network-check-target-t8tk8\" (UID: \"82d55335-0036-4166-a35c-65adec33932d\") " pod="openshift-network-diagnostics/network-check-target-t8tk8"
Sep 09 20:52:41.023162 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:52:41.023119    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 20:52:41.023197 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:52:41.023181    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:52:41.023197 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:52:41.023189    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 20:52:41.023269 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:52:41.023207    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 20:52:41.023269 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:52:41.023219    2576 projected.go:196] Error preparing data for projected volume kube-api-access-krdp6 for pod openshift-network-diagnostics/network-check-target-t8tk8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:52:41.023269 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:52:41.023189    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b906e5b3-61d3-41da-9776-795376a73811-original-pull-secret podName:b906e5b3-61d3-41da-9776-795376a73811 nodeName:}" failed. No retries permitted until 2026-09-09 20:52:43.023171152 +0000 UTC m=+6.136856995 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b906e5b3-61d3-41da-9776-795376a73811-original-pull-secret") pod "global-pull-secret-syncer-2c9kq" (UID: "b906e5b3-61d3-41da-9776-795376a73811") : object "kube-system"/"original-pull-secret" not registered
Sep 09 20:52:41.023364 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:52:41.023274    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b815b2b-983b-4afe-900d-13dd791e4778-metrics-certs podName:5b815b2b-983b-4afe-900d-13dd791e4778 nodeName:}" failed. No retries permitted until 2026-09-09 20:52:43.02325991 +0000 UTC m=+6.136945749 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5b815b2b-983b-4afe-900d-13dd791e4778-metrics-certs") pod "network-metrics-daemon-v74xn" (UID: "5b815b2b-983b-4afe-900d-13dd791e4778") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:52:41.023364 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:52:41.023288    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/82d55335-0036-4166-a35c-65adec33932d-kube-api-access-krdp6 podName:82d55335-0036-4166-a35c-65adec33932d nodeName:}" failed. No retries permitted until 2026-09-09 20:52:43.023280253 +0000 UTC m=+6.136966094 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-krdp6" (UniqueName: "kubernetes.io/projected/82d55335-0036-4166-a35c-65adec33932d-kube-api-access-krdp6") pod "network-check-target-t8tk8" (UID: "82d55335-0036-4166-a35c-65adec33932d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:52:41.352532 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:41.352464    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-08 20:47:39 +0000 UTC" deadline="2028-05-19 23:15:02.54808976 +0000 UTC"
Sep 09 20:52:41.352532 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:41.352533    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14834h22m21.195563951s"
Sep 09 20:52:41.406507 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:41.406114    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2c9kq"
Sep 09 20:52:41.406507 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:52:41.406251    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2c9kq" podUID="b906e5b3-61d3-41da-9776-795376a73811"
Sep 09 20:52:41.428022 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:41.427944    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-6mr4s" event={"ID":"af099159-41cf-43d9-8ffb-6e5fb9aeea23","Type":"ContainerStarted","Data":"a7713fcb504ad8c85258bcb46f5e6cfdbea067be8e8771cd3e47a82c11f749c6"}
Sep 09 20:52:41.445014 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:41.444975    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5vpsh" event={"ID":"b3a7af8e-62e0-43d9-8845-a71f6b10c551","Type":"ContainerStarted","Data":"d4f5e447672ca89b2bb80dc8b96611b4ab706aa251e0f4a9b262b0a47d7715f0"}
Sep 09 20:52:41.460019 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:41.459987    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-dcv48" event={"ID":"e706fb03-415d-407d-b105-bd06474f6882","Type":"ContainerStarted","Data":"54e9e976a07ef6de70be86a02f28ffb251bc7df514d92aae83fae42080199bb3"}
Sep 09 20:52:41.468294 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:41.468260    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-x9txm" event={"ID":"cc3e4e7d-c7ef-462c-963a-bbcb4931a486","Type":"ContainerStarted","Data":"6f23063a8bb3ac4b2dd8140391838c2a57dcb53bac2d795ea86ba1f97a2f9ddc"}
Sep 09 20:52:41.471432 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:41.471352    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-5v6vg" event={"ID":"f30a6570-2579-4271-ba77-d75fd2fd8390","Type":"ContainerStarted","Data":"9b85d4642ed456eb987a58d0f67cfda1474b8320259f46c6f8585a5ea9f949bc"}
Sep 09 20:52:41.474669 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:41.474611    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-5vzc7" event={"ID":"855262cd-a32b-4283-9482-cd25b26b0433","Type":"ContainerStarted","Data":"a18e179ac4b61594f1fb4e8e832779437d7148e5ff21d222fe2a2fe53a6ea98d"}
Sep 09 20:52:41.479168 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:41.479132    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-wvw9h" event={"ID":"09f40168-2bb2-4eff-8a3f-7ef28a149eed","Type":"ContainerStarted","Data":"b410d0d7e621cdedef89adbb5c059b3d48d3e8bb36f7aa9f6c1122b49ef8b4c5"}
Sep 09 20:52:41.483896 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:41.483864    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d8z7g" event={"ID":"3ede0a95-3907-4d9c-aa45-acf822104e28","Type":"ContainerStarted","Data":"8d535042dff70451fa11dd0141711b38b81b5fbd84a428570b4981bbfdb6c63b"}
Sep 09 20:52:41.497795 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:41.497767    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tkbs9" event={"ID":"f0db116c-804d-4ce3-b4f9-135288ad7cea","Type":"ContainerStarted","Data":"bfee3f9b043cb58fd39551b6ad78ea13a5df809dd7b942cf58acd774d834e57e"}
Sep 09 20:52:41.519679 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:41.519649    2576 generic.go:356] "Generic (PLEG): container finished" podID="8d0d54b6d026f1113a292c2b25b7bd73" containerID="75c67526d9b01312bdb161cf2c9a6f5fa125dc02b6b4089c74c140ce4e0339a6" exitCode=0
Sep 09 20:52:41.519812 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:41.519750    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-72.ec2.internal" event={"ID":"8d0d54b6d026f1113a292c2b25b7bd73","Type":"ContainerDied","Data":"75c67526d9b01312bdb161cf2c9a6f5fa125dc02b6b4089c74c140ce4e0339a6"}
Sep 09 20:52:41.527829 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:41.527801    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-72.ec2.internal" event={"ID":"2d40b80525651e1d9758e04c441ca328","Type":"ContainerStarted","Data":"1d57e9a72c11ce2b650e1b281575218a1a33e9426612487295533e0bbbcbbd57"}
Sep 09 20:52:42.403940 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:42.403680    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t8tk8"
Sep 09 20:52:42.404424 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:42.403866    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v74xn"
Sep 09 20:52:42.404424 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:52:42.404058    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t8tk8" podUID="82d55335-0036-4166-a35c-65adec33932d"
Sep 09 20:52:42.404424 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:52:42.404156    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v74xn" podUID="5b815b2b-983b-4afe-900d-13dd791e4778"
Sep 09 20:52:42.542273 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:42.542238    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-72.ec2.internal" event={"ID":"8d0d54b6d026f1113a292c2b25b7bd73","Type":"ContainerStarted","Data":"e727885a07af384942182247e79dd4684b311fb84b8297b3303a126be60eea72"}
Sep 09 20:52:42.563677 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:42.563616    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-72.ec2.internal" podStartSLOduration=3.563592822 podStartE2EDuration="3.563592822s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:52:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:52:41.551254267 +0000 UTC m=+4.664940136" watchObservedRunningTime="2026-09-09 20:52:42.563592822 +0000 UTC m=+5.677278688"
Sep 09 20:52:43.039958 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:43.039913    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b906e5b3-61d3-41da-9776-795376a73811-original-pull-secret\") pod \"global-pull-secret-syncer-2c9kq\" (UID: \"b906e5b3-61d3-41da-9776-795376a73811\") " pod="kube-system/global-pull-secret-syncer-2c9kq"
Sep 09 20:52:43.040141 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:43.039976    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b815b2b-983b-4afe-900d-13dd791e4778-metrics-certs\") pod \"network-metrics-daemon-v74xn\" (UID: \"5b815b2b-983b-4afe-900d-13dd791e4778\") " pod="openshift-multus/network-metrics-daemon-v74xn"
Sep 09 20:52:43.040141 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:43.040019    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krdp6\" (UniqueName: \"kubernetes.io/projected/82d55335-0036-4166-a35c-65adec33932d-kube-api-access-krdp6\") pod \"network-check-target-t8tk8\" (UID: \"82d55335-0036-4166-a35c-65adec33932d\") " pod="openshift-network-diagnostics/network-check-target-t8tk8"
Sep 09 20:52:43.040283 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:52:43.040209    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 20:52:43.040283 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:52:43.040242    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 20:52:43.040283 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:52:43.040258    2576 projected.go:196] Error preparing data for projected volume kube-api-access-krdp6 for pod openshift-network-diagnostics/network-check-target-t8tk8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:52:43.040426 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:52:43.040322    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/82d55335-0036-4166-a35c-65adec33932d-kube-api-access-krdp6 podName:82d55335-0036-4166-a35c-65adec33932d nodeName:}" failed. No retries permitted until 2026-09-09 20:52:47.04030249 +0000 UTC m=+10.153988329 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-krdp6" (UniqueName: "kubernetes.io/projected/82d55335-0036-4166-a35c-65adec33932d-kube-api-access-krdp6") pod "network-check-target-t8tk8" (UID: "82d55335-0036-4166-a35c-65adec33932d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:52:43.040954 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:52:43.040802    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 20:52:43.040954 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:52:43.040859    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b906e5b3-61d3-41da-9776-795376a73811-original-pull-secret podName:b906e5b3-61d3-41da-9776-795376a73811 nodeName:}" failed. No retries permitted until 2026-09-09 20:52:47.040842561 +0000 UTC m=+10.154528404 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b906e5b3-61d3-41da-9776-795376a73811-original-pull-secret") pod "global-pull-secret-syncer-2c9kq" (UID: "b906e5b3-61d3-41da-9776-795376a73811") : object "kube-system"/"original-pull-secret" not registered
Sep 09 20:52:43.040954 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:52:43.040915    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:52:43.040954 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:52:43.040955    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b815b2b-983b-4afe-900d-13dd791e4778-metrics-certs podName:5b815b2b-983b-4afe-900d-13dd791e4778 nodeName:}" failed. No retries permitted until 2026-09-09 20:52:47.040943092 +0000 UTC m=+10.154628949 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5b815b2b-983b-4afe-900d-13dd791e4778-metrics-certs") pod "network-metrics-daemon-v74xn" (UID: "5b815b2b-983b-4afe-900d-13dd791e4778") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:52:43.404188 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:43.403828    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2c9kq"
Sep 09 20:52:43.404188 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:52:43.403992    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2c9kq" podUID="b906e5b3-61d3-41da-9776-795376a73811"
Sep 09 20:52:44.404001 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:44.403958    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t8tk8"
Sep 09 20:52:44.404199 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:52:44.404128    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t8tk8" podUID="82d55335-0036-4166-a35c-65adec33932d"
Sep 09 20:52:44.404591 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:44.404224    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v74xn"
Sep 09 20:52:44.404591 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:52:44.404311    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v74xn" podUID="5b815b2b-983b-4afe-900d-13dd791e4778"
Sep 09 20:52:45.404102 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:45.403584    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2c9kq"
Sep 09 20:52:45.404102 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:52:45.403736    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2c9kq" podUID="b906e5b3-61d3-41da-9776-795376a73811"
Sep 09 20:52:46.403494 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:46.403410    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v74xn"
Sep 09 20:52:46.403494 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:46.403410    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t8tk8"
Sep 09 20:52:46.403718 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:52:46.403569    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v74xn" podUID="5b815b2b-983b-4afe-900d-13dd791e4778"
Sep 09 20:52:46.403956 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:52:46.403899    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t8tk8" podUID="82d55335-0036-4166-a35c-65adec33932d"
Sep 09 20:52:47.074566 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:47.074530    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krdp6\" (UniqueName: \"kubernetes.io/projected/82d55335-0036-4166-a35c-65adec33932d-kube-api-access-krdp6\") pod \"network-check-target-t8tk8\" (UID: \"82d55335-0036-4166-a35c-65adec33932d\") " pod="openshift-network-diagnostics/network-check-target-t8tk8"
Sep 09 20:52:47.074989 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:47.074619    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b906e5b3-61d3-41da-9776-795376a73811-original-pull-secret\") pod \"global-pull-secret-syncer-2c9kq\" (UID: \"b906e5b3-61d3-41da-9776-795376a73811\") " pod="kube-system/global-pull-secret-syncer-2c9kq"
Sep 09 20:52:47.074989 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:47.074655    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b815b2b-983b-4afe-900d-13dd791e4778-metrics-certs\") pod \"network-metrics-daemon-v74xn\" (UID: \"5b815b2b-983b-4afe-900d-13dd791e4778\") " pod="openshift-multus/network-metrics-daemon-v74xn"
Sep 09 20:52:47.074989 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:52:47.074694    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 20:52:47.074989 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:52:47.074714    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 20:52:47.074989 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:52:47.074723    2576 projected.go:196] Error preparing data for projected volume kube-api-access-krdp6 for pod openshift-network-diagnostics/network-check-target-t8tk8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:52:47.074989 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:52:47.074757    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 20:52:47.074989 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:52:47.074767    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/82d55335-0036-4166-a35c-65adec33932d-kube-api-access-krdp6 podName:82d55335-0036-4166-a35c-65adec33932d nodeName:}" failed. No retries permitted until 2026-09-09 20:52:55.074754224 +0000 UTC m=+18.188440064 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-krdp6" (UniqueName: "kubernetes.io/projected/82d55335-0036-4166-a35c-65adec33932d-kube-api-access-krdp6") pod "network-check-target-t8tk8" (UID: "82d55335-0036-4166-a35c-65adec33932d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:52:47.074989 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:52:47.074770    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:52:47.074989 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:52:47.074802    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b906e5b3-61d3-41da-9776-795376a73811-original-pull-secret podName:b906e5b3-61d3-41da-9776-795376a73811 nodeName:}" failed. No retries permitted until 2026-09-09 20:52:55.074790148 +0000 UTC m=+18.188475985 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b906e5b3-61d3-41da-9776-795376a73811-original-pull-secret") pod "global-pull-secret-syncer-2c9kq" (UID: "b906e5b3-61d3-41da-9776-795376a73811") : object "kube-system"/"original-pull-secret" not registered
Sep 09 20:52:47.074989 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:52:47.074823    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b815b2b-983b-4afe-900d-13dd791e4778-metrics-certs podName:5b815b2b-983b-4afe-900d-13dd791e4778 nodeName:}" failed. No retries permitted until 2026-09-09 20:52:55.074807609 +0000 UTC m=+18.188493448 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5b815b2b-983b-4afe-900d-13dd791e4778-metrics-certs") pod "network-metrics-daemon-v74xn" (UID: "5b815b2b-983b-4afe-900d-13dd791e4778") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:52:47.404464 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:47.404396    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2c9kq"
Sep 09 20:52:47.404588 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:52:47.404510    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2c9kq" podUID="b906e5b3-61d3-41da-9776-795376a73811"
Sep 09 20:52:48.403983 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:48.403945    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t8tk8"
Sep 09 20:52:48.404474 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:48.403963    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v74xn"
Sep 09 20:52:48.404474 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:52:48.404088    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t8tk8" podUID="82d55335-0036-4166-a35c-65adec33932d"
Sep 09 20:52:48.404474 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:52:48.404164    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v74xn" podUID="5b815b2b-983b-4afe-900d-13dd791e4778"
Sep 09 20:52:49.404072 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:49.404008    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2c9kq"
Sep 09 20:52:49.404509 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:52:49.404148    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2c9kq" podUID="b906e5b3-61d3-41da-9776-795376a73811"
Sep 09 20:52:50.403781 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:50.403750    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t8tk8"
Sep 09 20:52:50.403973 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:50.403758    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v74xn"
Sep 09 20:52:50.403973 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:52:50.403887    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t8tk8" podUID="82d55335-0036-4166-a35c-65adec33932d"
Sep 09 20:52:50.403973 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:52:50.403963    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v74xn" podUID="5b815b2b-983b-4afe-900d-13dd791e4778"
Sep 09 20:52:51.403873 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:51.403353    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2c9kq"
Sep 09 20:52:51.403873 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:52:51.403515    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2c9kq" podUID="b906e5b3-61d3-41da-9776-795376a73811"
Sep 09 20:52:52.403448 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:52.403424    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t8tk8"
Sep 09 20:52:52.403587 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:52:52.403528    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t8tk8" podUID="82d55335-0036-4166-a35c-65adec33932d"
Sep 09 20:52:52.403629 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:52.403583    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v74xn"
Sep 09 20:52:52.403701 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:52:52.403684    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v74xn" podUID="5b815b2b-983b-4afe-900d-13dd791e4778"
Sep 09 20:52:53.404071 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:53.404039    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2c9kq"
Sep 09 20:52:53.404486 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:52:53.404224    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2c9kq" podUID="b906e5b3-61d3-41da-9776-795376a73811"
Sep 09 20:52:54.403271 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:54.403225    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v74xn"
Sep 09 20:52:54.403516 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:54.403281    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t8tk8"
Sep 09 20:52:54.403516 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:52:54.403408    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v74xn" podUID="5b815b2b-983b-4afe-900d-13dd791e4778"
Sep 09 20:52:54.403516 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:52:54.403499    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t8tk8" podUID="82d55335-0036-4166-a35c-65adec33932d"
Sep 09 20:52:55.133672 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:55.133631    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b906e5b3-61d3-41da-9776-795376a73811-original-pull-secret\") pod \"global-pull-secret-syncer-2c9kq\" (UID: \"b906e5b3-61d3-41da-9776-795376a73811\") " pod="kube-system/global-pull-secret-syncer-2c9kq"
Sep 09 20:52:55.134201 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:55.133686    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b815b2b-983b-4afe-900d-13dd791e4778-metrics-certs\") pod \"network-metrics-daemon-v74xn\" (UID: \"5b815b2b-983b-4afe-900d-13dd791e4778\") " pod="openshift-multus/network-metrics-daemon-v74xn"
Sep 09 20:52:55.134201 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:55.133730    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krdp6\" (UniqueName: \"kubernetes.io/projected/82d55335-0036-4166-a35c-65adec33932d-kube-api-access-krdp6\") pod \"network-check-target-t8tk8\" (UID: \"82d55335-0036-4166-a35c-65adec33932d\") " pod="openshift-network-diagnostics/network-check-target-t8tk8"
Sep 09 20:52:55.134201 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:52:55.133795    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 20:52:55.134201 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:52:55.133868    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b906e5b3-61d3-41da-9776-795376a73811-original-pull-secret podName:b906e5b3-61d3-41da-9776-795376a73811 nodeName:}" failed. No retries permitted until 2026-09-09 20:53:11.133847587 +0000 UTC m=+34.247533423 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b906e5b3-61d3-41da-9776-795376a73811-original-pull-secret") pod "global-pull-secret-syncer-2c9kq" (UID: "b906e5b3-61d3-41da-9776-795376a73811") : object "kube-system"/"original-pull-secret" not registered
Sep 09 20:52:55.134201 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:52:55.133888    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 20:52:55.134201 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:52:55.133910    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 20:52:55.134201 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:52:55.133921    2576 projected.go:196] Error preparing data for projected volume kube-api-access-krdp6 for pod openshift-network-diagnostics/network-check-target-t8tk8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:52:55.134201 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:52:55.133965    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/82d55335-0036-4166-a35c-65adec33932d-kube-api-access-krdp6 podName:82d55335-0036-4166-a35c-65adec33932d nodeName:}" failed. No retries permitted until 2026-09-09 20:53:11.133949674 +0000 UTC m=+34.247635513 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-krdp6" (UniqueName: "kubernetes.io/projected/82d55335-0036-4166-a35c-65adec33932d-kube-api-access-krdp6") pod "network-check-target-t8tk8" (UID: "82d55335-0036-4166-a35c-65adec33932d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:52:55.134201 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:52:55.134030    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:52:55.134201 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:52:55.134084    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b815b2b-983b-4afe-900d-13dd791e4778-metrics-certs podName:5b815b2b-983b-4afe-900d-13dd791e4778 nodeName:}" failed. No retries permitted until 2026-09-09 20:53:11.134053566 +0000 UTC m=+34.247739428 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5b815b2b-983b-4afe-900d-13dd791e4778-metrics-certs") pod "network-metrics-daemon-v74xn" (UID: "5b815b2b-983b-4afe-900d-13dd791e4778") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:52:55.404124 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:55.403587    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2c9kq"
Sep 09 20:52:55.404124 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:52:55.403721    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2c9kq" podUID="b906e5b3-61d3-41da-9776-795376a73811"
Sep 09 20:52:56.403676 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:56.403632    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t8tk8"
Sep 09 20:52:56.403676 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:56.403664    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v74xn"
Sep 09 20:52:56.404172 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:52:56.403760    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t8tk8" podUID="82d55335-0036-4166-a35c-65adec33932d"
Sep 09 20:52:56.404172 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:52:56.403907    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v74xn" podUID="5b815b2b-983b-4afe-900d-13dd791e4778"
Sep 09 20:52:57.404319 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:57.404290    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2c9kq"
Sep 09 20:52:57.404789 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:52:57.404402    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2c9kq" podUID="b906e5b3-61d3-41da-9776-795376a73811"
Sep 09 20:52:58.403636 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:58.403602    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t8tk8"
Sep 09 20:52:58.403818 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:58.403611    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v74xn"
Sep 09 20:52:58.403818 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:52:58.403716    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t8tk8" podUID="82d55335-0036-4166-a35c-65adec33932d"
Sep 09 20:52:58.403818 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:52:58.403806    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v74xn" podUID="5b815b2b-983b-4afe-900d-13dd791e4778"
Sep 09 20:52:59.403124 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:52:59.403094    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2c9kq"
Sep 09 20:52:59.403616 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:52:59.403231    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2c9kq" podUID="b906e5b3-61d3-41da-9776-795376a73811"
Sep 09 20:53:00.403279 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:00.403080    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t8tk8"
Sep 09 20:53:00.404129 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:00.403124    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v74xn"
Sep 09 20:53:00.404129 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:53:00.403361    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t8tk8" podUID="82d55335-0036-4166-a35c-65adec33932d"
Sep 09 20:53:00.404129 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:53:00.403485    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v74xn" podUID="5b815b2b-983b-4afe-900d-13dd791e4778"
Sep 09 20:53:00.583272 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:00.583227    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-6mr4s" event={"ID":"af099159-41cf-43d9-8ffb-6e5fb9aeea23","Type":"ContainerStarted","Data":"43afc812e82b184368a190d888b989861f3085d482afc48c2cf9097420302ed5"}
Sep 09 20:53:00.584740 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:00.584715    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5vpsh" event={"ID":"b3a7af8e-62e0-43d9-8845-a71f6b10c551","Type":"ContainerStarted","Data":"ed0c0ee86537cec718b4f356f301875aad4bb2b2e1c7ab8ce2355041f2728038"}
Sep 09 20:53:00.586049 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:00.586028    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-dcv48" event={"ID":"e706fb03-415d-407d-b105-bd06474f6882","Type":"ContainerStarted","Data":"0c4a7bc0c9c2f4442c63bafa68514983b35409e1f6bb39c42489ccd46bd4389f"}
Sep 09 20:53:00.587450 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:00.587430    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-5v6vg" event={"ID":"f30a6570-2579-4271-ba77-d75fd2fd8390","Type":"ContainerStarted","Data":"7001cbc68181ceaec786d97a9f7fda8524f93203fa3f1cfa86082eabaa3b2e4d"}
Sep 09 20:53:00.588848 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:00.588830    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-5vzc7" event={"ID":"855262cd-a32b-4283-9482-cd25b26b0433","Type":"ContainerStarted","Data":"50a83055fb4c1360cab3614d38e00902607e61cd016097e2bf4ded4583abc0f8"}
Sep 09 20:53:00.590086 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:00.590067    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-wvw9h" event={"ID":"09f40168-2bb2-4eff-8a3f-7ef28a149eed","Type":"ContainerStarted","Data":"2aecd4f91eb8b8b6b4932623f7547fae751372a35040aaa721f5943f8a83c0a2"}
Sep 09 20:53:00.591409 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:00.591389    2576 generic.go:356] "Generic (PLEG): container finished" podID="3ede0a95-3907-4d9c-aa45-acf822104e28" containerID="c25ab9bc71186ba09990ce0608bef1ce1b3c8b9fe3d6910b0820d3f53ff0e2ca" exitCode=0
Sep 09 20:53:00.591487 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:00.591445    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d8z7g" event={"ID":"3ede0a95-3907-4d9c-aa45-acf822104e28","Type":"ContainerDied","Data":"c25ab9bc71186ba09990ce0608bef1ce1b3c8b9fe3d6910b0820d3f53ff0e2ca"}
Sep 09 20:53:00.594580 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:00.594560    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tkbs9" event={"ID":"f0db116c-804d-4ce3-b4f9-135288ad7cea","Type":"ContainerStarted","Data":"5c6b5b9ebce5ae23cbb1965d1668bd5d263f5c995088546344bed096aa7cbc85"}
Sep 09 20:53:00.594660 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:00.594622    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tkbs9" event={"ID":"f0db116c-804d-4ce3-b4f9-135288ad7cea","Type":"ContainerStarted","Data":"9fa83c4fa4a7542484a9a317729c72f8de4ee7c14dd78632e26e7e1e3c12f356"}
Sep 09 20:53:00.594660 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:00.594653    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tkbs9" event={"ID":"f0db116c-804d-4ce3-b4f9-135288ad7cea","Type":"ContainerStarted","Data":"1f0d5024c7208fe4a2e3c7e30174429c2a517bf7a37903638570026441706379"}
Sep 09 20:53:00.594731 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:00.594681    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tkbs9" event={"ID":"f0db116c-804d-4ce3-b4f9-135288ad7cea","Type":"ContainerStarted","Data":"ca52eb6afee19867d51e91755fce5b8c957e00a5f188157b8d591e964d698bfa"}
Sep 09 20:53:00.594731 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:00.594708    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tkbs9" event={"ID":"f0db116c-804d-4ce3-b4f9-135288ad7cea","Type":"ContainerStarted","Data":"61cb256c8d43ba2cc235197eae34f80919270fa5bff6cd3c49be660938af7e7d"}
Sep 09 20:53:00.594790 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:00.594737    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tkbs9" event={"ID":"f0db116c-804d-4ce3-b4f9-135288ad7cea","Type":"ContainerStarted","Data":"493197ed459c76571f897997f411a2fcb23c3d6f9b8386e95fbb62cc63ebde88"}
Sep 09 20:53:00.601740 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:00.601696    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-72.ec2.internal" podStartSLOduration=21.601681841 podStartE2EDuration="21.601681841s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:52:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:52:42.564337914 +0000 UTC m=+5.678023776" watchObservedRunningTime="2026-09-09 20:53:00.601681841 +0000 UTC m=+23.715367701"
Sep 09 20:53:00.602081 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:00.602059    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-6mr4s" podStartSLOduration=3.7372780519999997 podStartE2EDuration="22.602052138s" totalImagesPullingTime="18.864774086s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:52:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:53:00.601636085 +0000 UTC m=+23.715321945" watchObservedRunningTime="2026-09-09 20:53:00.602052138 +0000 UTC m=+23.715738038"
Sep 09 20:53:00.617983 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:00.617804    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-wvw9h" podStartSLOduration=3.773494903 podStartE2EDuration="22.617795415s" totalImagesPullingTime="18.844300512s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:52:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:53:00.617563524 +0000 UTC m=+23.731249382" watchObservedRunningTime="2026-09-09 20:53:00.617795415 +0000 UTC m=+23.731481274"
Sep 09 20:53:00.671787 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:00.671729    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-5v6vg" podStartSLOduration=7.562246544 podStartE2EDuration="22.671713305s" totalImagesPullingTime="15.109466761s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:52:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:53:00.647388685 +0000 UTC m=+23.761074543" watchObservedRunningTime="2026-09-09 20:53:00.671713305 +0000 UTC m=+23.785399163"
Sep 09 20:53:00.672325 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:00.672282    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-5vzc7" podStartSLOduration=3.7780061480000002 podStartE2EDuration="22.672270018s" totalImagesPullingTime="18.89426387s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:52:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:53:00.672171875 +0000 UTC m=+23.785857733" watchObservedRunningTime="2026-09-09 20:53:00.672270018 +0000 UTC m=+23.785955877"
Sep 09 20:53:00.689322 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:00.689276    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-dcv48" podStartSLOduration=3.845899947 podStartE2EDuration="22.689256513s" totalImagesPullingTime="18.843356566s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:52:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:53:00.689163903 +0000 UTC m=+23.802849762" watchObservedRunningTime="2026-09-09 20:53:00.689256513 +0000 UTC m=+23.802942373"
Sep 09 20:53:00.757689 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:00.757654    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-5v6vg"
Sep 09 20:53:00.758487 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:00.758338    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-5v6vg"
Sep 09 20:53:01.138895 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:01.138873    2576 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 09 20:53:01.374203 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:01.374063    2576 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-09T20:53:01.138890102Z","UUID":"dddc39f8-26b8-4256-8e87-481327667bdd","Handler":null,"Name":"","Endpoint":""}
Sep 09 20:53:01.375714 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:01.375692    2576 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 09 20:53:01.375835 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:01.375729    2576 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 09 20:53:01.403894 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:01.403869    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2c9kq"
Sep 09 20:53:01.404422 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:53:01.404002    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2c9kq" podUID="b906e5b3-61d3-41da-9776-795376a73811"
Sep 09 20:53:01.598988 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:01.598953    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5vpsh" event={"ID":"b3a7af8e-62e0-43d9-8845-a71f6b10c551","Type":"ContainerStarted","Data":"c695f9f80e5d153e11cbb1cf01742e342696e142450e127f666f1ebcdc1123f4"}
Sep 09 20:53:01.600789 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:01.600761    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-x9txm" event={"ID":"cc3e4e7d-c7ef-462c-963a-bbcb4931a486","Type":"ContainerStarted","Data":"a09d0edfa29e54d67fb6ae3ea4e050f8f895a6c6871972630b8b5417a58ff052"}
Sep 09 20:53:01.601446 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:01.601420    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-5v6vg"
Sep 09 20:53:01.601897 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:01.601880    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-5v6vg"
Sep 09 20:53:01.634873 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:01.634783    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-x9txm" podStartSLOduration=4.797366816 podStartE2EDuration="23.634770354s" totalImagesPullingTime="18.837403538s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:52:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:53:01.617472947 +0000 UTC m=+24.731158807" watchObservedRunningTime="2026-09-09 20:53:01.634770354 +0000 UTC m=+24.748456212"
Sep 09 20:53:02.403087 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:02.403061    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t8tk8"
Sep 09 20:53:02.403259 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:53:02.403175    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t8tk8" podUID="82d55335-0036-4166-a35c-65adec33932d"
Sep 09 20:53:02.403552 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:02.403528    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v74xn"
Sep 09 20:53:02.403677 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:53:02.403612    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v74xn" podUID="5b815b2b-983b-4afe-900d-13dd791e4778"
Sep 09 20:53:02.607454 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:02.607412    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tkbs9" event={"ID":"f0db116c-804d-4ce3-b4f9-135288ad7cea","Type":"ContainerStarted","Data":"1a5f90c852aae9fc85945a6a8bf38af0ea078509a463aa093449ef8e31e8a8c5"}
Sep 09 20:53:03.404165 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:03.404128    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2c9kq"
Sep 09 20:53:03.404357 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:53:03.404263    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2c9kq" podUID="b906e5b3-61d3-41da-9776-795376a73811"
Sep 09 20:53:03.612202 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:03.612161    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5vpsh" event={"ID":"b3a7af8e-62e0-43d9-8845-a71f6b10c551","Type":"ContainerStarted","Data":"ceb2218d2b470993ea243329f9629f6df271b4d256e6abd61f04e42d78a565a5"}
Sep 09 20:53:03.639573 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:03.639517    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5vpsh" podStartSLOduration=3.891875318 podStartE2EDuration="25.639497408s" totalImagesPullingTime="21.74762209s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:52:38 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:53:03.638806737 +0000 UTC m=+26.752492605" watchObservedRunningTime="2026-09-09 20:53:03.639497408 +0000 UTC m=+26.753183268"
Sep 09 20:53:04.403648 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:04.403613    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t8tk8"
Sep 09 20:53:04.403841 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:53:04.403726    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t8tk8" podUID="82d55335-0036-4166-a35c-65adec33932d"
Sep 09 20:53:04.403841 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:04.403793    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v74xn"
Sep 09 20:53:04.403940 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:53:04.403861    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v74xn" podUID="5b815b2b-983b-4afe-900d-13dd791e4778"
Sep 09 20:53:05.403703 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:05.403621    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2c9kq"
Sep 09 20:53:05.404360 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:53:05.403742    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2c9kq" podUID="b906e5b3-61d3-41da-9776-795376a73811"
Sep 09 20:53:05.618292 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:05.618263    2576 generic.go:356] "Generic (PLEG): container finished" podID="3ede0a95-3907-4d9c-aa45-acf822104e28" containerID="bb79efe72e363d6fe4dd6c58bdc7071f7f946c92da568ad6f555101bdf8a2e2c" exitCode=0
Sep 09 20:53:05.618529 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:05.618341    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d8z7g" event={"ID":"3ede0a95-3907-4d9c-aa45-acf822104e28","Type":"ContainerDied","Data":"bb79efe72e363d6fe4dd6c58bdc7071f7f946c92da568ad6f555101bdf8a2e2c"}
Sep 09 20:53:05.622145 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:05.622123    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tkbs9" event={"ID":"f0db116c-804d-4ce3-b4f9-135288ad7cea","Type":"ContainerStarted","Data":"90b908aef258dd6be0f82540e1b5434b70c51d5e6d19be3263791af64e26380a"}
Sep 09 20:53:05.622497 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:05.622343    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-tkbs9"
Sep 09 20:53:05.622497 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:05.622450    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-tkbs9"
Sep 09 20:53:05.622629 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:05.622501    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-tkbs9"
Sep 09 20:53:05.637875 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:05.637848    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tkbs9"
Sep 09 20:53:05.638019 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:05.638006    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tkbs9"
Sep 09 20:53:06.403604 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:06.403577    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v74xn"
Sep 09 20:53:06.403834 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:53:06.403704    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v74xn" podUID="5b815b2b-983b-4afe-900d-13dd791e4778"
Sep 09 20:53:06.403834 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:06.403755    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t8tk8"
Sep 09 20:53:06.404273 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:53:06.403836    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t8tk8" podUID="82d55335-0036-4166-a35c-65adec33932d"
Sep 09 20:53:06.873895 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:06.873420    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-tkbs9" podStartSLOduration=9.972254191 podStartE2EDuration="28.873394529s" totalImagesPullingTime="18.901140338s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:52:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:53:05.690999286 +0000 UTC m=+28.804685144" watchObservedRunningTime="2026-09-09 20:53:06.873394529 +0000 UTC m=+29.987080383"
Sep 09 20:53:06.874688 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:06.874665    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-2c9kq"]
Sep 09 20:53:06.874815 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:06.874784    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2c9kq"
Sep 09 20:53:06.874920 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:53:06.874898    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2c9kq" podUID="b906e5b3-61d3-41da-9776-795376a73811"
Sep 09 20:53:06.879643 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:06.879619    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-v74xn"]
Sep 09 20:53:06.879798 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:06.879750    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v74xn"
Sep 09 20:53:06.879879 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:53:06.879853    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v74xn" podUID="5b815b2b-983b-4afe-900d-13dd791e4778"
Sep 09 20:53:06.901382 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:06.901348    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-t8tk8"]
Sep 09 20:53:06.901504 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:06.901463    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t8tk8"
Sep 09 20:53:06.901568 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:53:06.901549    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t8tk8" podUID="82d55335-0036-4166-a35c-65adec33932d"
Sep 09 20:53:07.629225 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:07.629028    2576 generic.go:356] "Generic (PLEG): container finished" podID="3ede0a95-3907-4d9c-aa45-acf822104e28" containerID="d11056add32d778586c8fcd53a1f63ff4cec7e7e815bed7e0cbf3a11a0dd63da" exitCode=0
Sep 09 20:53:07.629628 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:07.629116    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d8z7g" event={"ID":"3ede0a95-3907-4d9c-aa45-acf822104e28","Type":"ContainerDied","Data":"d11056add32d778586c8fcd53a1f63ff4cec7e7e815bed7e0cbf3a11a0dd63da"}
Sep 09 20:53:08.403643 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:08.403604    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t8tk8"
Sep 09 20:53:08.403874 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:53:08.403729    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t8tk8" podUID="82d55335-0036-4166-a35c-65adec33932d"
Sep 09 20:53:08.403874 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:08.403850    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v74xn"
Sep 09 20:53:08.403874 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:08.403862    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2c9kq"
Sep 09 20:53:08.404029 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:53:08.403948    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v74xn" podUID="5b815b2b-983b-4afe-900d-13dd791e4778"
Sep 09 20:53:08.404029 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:53:08.404023    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2c9kq" podUID="b906e5b3-61d3-41da-9776-795376a73811"
Sep 09 20:53:09.635977 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:09.635890    2576 generic.go:356] "Generic (PLEG): container finished" podID="3ede0a95-3907-4d9c-aa45-acf822104e28" containerID="03473eae7fab72069d668606183a9c69e04ac31ac14acb43f12d56dc0a8187de" exitCode=0
Sep 09 20:53:09.635977 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:09.635945    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d8z7g" event={"ID":"3ede0a95-3907-4d9c-aa45-acf822104e28","Type":"ContainerDied","Data":"03473eae7fab72069d668606183a9c69e04ac31ac14acb43f12d56dc0a8187de"}
Sep 09 20:53:10.403067 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:10.403030    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2c9kq"
Sep 09 20:53:10.403254 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:10.403029    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v74xn"
Sep 09 20:53:10.403322 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:10.403042    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t8tk8"
Sep 09 20:53:10.403396 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:53:10.403351    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t8tk8" podUID="82d55335-0036-4166-a35c-65adec33932d"
Sep 09 20:53:10.403460 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:53:10.403154    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2c9kq" podUID="b906e5b3-61d3-41da-9776-795376a73811"
Sep 09 20:53:10.403514 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:53:10.403461    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v74xn" podUID="5b815b2b-983b-4afe-900d-13dd791e4778"
Sep 09 20:53:11.153382 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:11.153328    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b906e5b3-61d3-41da-9776-795376a73811-original-pull-secret\") pod \"global-pull-secret-syncer-2c9kq\" (UID: \"b906e5b3-61d3-41da-9776-795376a73811\") " pod="kube-system/global-pull-secret-syncer-2c9kq"
Sep 09 20:53:11.153382 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:11.153389    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b815b2b-983b-4afe-900d-13dd791e4778-metrics-certs\") pod \"network-metrics-daemon-v74xn\" (UID: \"5b815b2b-983b-4afe-900d-13dd791e4778\") " pod="openshift-multus/network-metrics-daemon-v74xn"
Sep 09 20:53:11.153848 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:11.153426    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krdp6\" (UniqueName: \"kubernetes.io/projected/82d55335-0036-4166-a35c-65adec33932d-kube-api-access-krdp6\") pod \"network-check-target-t8tk8\" (UID: \"82d55335-0036-4166-a35c-65adec33932d\") " pod="openshift-network-diagnostics/network-check-target-t8tk8"
Sep 09 20:53:11.153848 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:53:11.153513    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 20:53:11.153848 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:53:11.153542    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:53:11.153848 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:53:11.153550    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 20:53:11.153848 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:53:11.153567    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 20:53:11.153848 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:53:11.153579    2576 projected.go:196] Error preparing data for projected volume kube-api-access-krdp6 for pod openshift-network-diagnostics/network-check-target-t8tk8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:53:11.153848 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:53:11.153594    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b906e5b3-61d3-41da-9776-795376a73811-original-pull-secret podName:b906e5b3-61d3-41da-9776-795376a73811 nodeName:}" failed. No retries permitted until 2026-09-09 20:53:43.153575014 +0000 UTC m=+66.267260851 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b906e5b3-61d3-41da-9776-795376a73811-original-pull-secret") pod "global-pull-secret-syncer-2c9kq" (UID: "b906e5b3-61d3-41da-9776-795376a73811") : object "kube-system"/"original-pull-secret" not registered
Sep 09 20:53:11.153848 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:53:11.153613    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b815b2b-983b-4afe-900d-13dd791e4778-metrics-certs podName:5b815b2b-983b-4afe-900d-13dd791e4778 nodeName:}" failed. No retries permitted until 2026-09-09 20:53:43.153605522 +0000 UTC m=+66.267291359 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5b815b2b-983b-4afe-900d-13dd791e4778-metrics-certs") pod "network-metrics-daemon-v74xn" (UID: "5b815b2b-983b-4afe-900d-13dd791e4778") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:53:11.153848 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:53:11.153641    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/82d55335-0036-4166-a35c-65adec33932d-kube-api-access-krdp6 podName:82d55335-0036-4166-a35c-65adec33932d nodeName:}" failed. No retries permitted until 2026-09-09 20:53:43.153625796 +0000 UTC m=+66.267311639 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-krdp6" (UniqueName: "kubernetes.io/projected/82d55335-0036-4166-a35c-65adec33932d-kube-api-access-krdp6") pod "network-check-target-t8tk8" (UID: "82d55335-0036-4166-a35c-65adec33932d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:53:12.403684 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:12.403653    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v74xn"
Sep 09 20:53:12.403684 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:12.403682    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t8tk8"
Sep 09 20:53:12.404249 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:12.403670    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2c9kq"
Sep 09 20:53:12.404249 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:53:12.403777    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v74xn" podUID="5b815b2b-983b-4afe-900d-13dd791e4778"
Sep 09 20:53:12.404249 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:53:12.403868    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2c9kq" podUID="b906e5b3-61d3-41da-9776-795376a73811"
Sep 09 20:53:12.404249 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:53:12.403960    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t8tk8" podUID="82d55335-0036-4166-a35c-65adec33932d"
Sep 09 20:53:13.205728 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:13.205692    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-72.ec2.internal" event="NodeReady"
Sep 09 20:53:13.205889 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:13.205850    2576 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 09 20:53:13.249740 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:13.249712    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-cb585848c-289xj"]
Sep 09 20:53:13.254311 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:13.254291    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-cb585848c-289xj"
Sep 09 20:53:13.257790 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:13.257250    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 09 20:53:13.257790 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:13.257580    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 09 20:53:13.257790 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:13.257790    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 09 20:53:13.258007 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:13.257800    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-xb6jr\""
Sep 09 20:53:13.267138 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:13.267121    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 09 20:53:13.275066 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:13.275047    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-cb585848c-289xj"]
Sep 09 20:53:13.275160 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:13.275096    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-968c2"]
Sep 09 20:53:13.281269 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:13.280618    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-hrrp2"]
Sep 09 20:53:13.281269 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:13.280738    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-968c2"
Sep 09 20:53:13.283766 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:13.283719    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-nr885\""
Sep 09 20:53:13.283766 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:13.283719    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 09 20:53:13.283766 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:13.283748    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 09 20:53:13.284006 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:13.283915    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-hrrp2"
Sep 09 20:53:13.287593 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:13.287573    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 09 20:53:13.287686 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:13.287661    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 09 20:53:13.287748 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:13.287704    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 09 20:53:13.287903 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:13.287888    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-h7nzq\""
Sep 09 20:53:13.291260 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:13.291241    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-hrrp2"]
Sep 09 20:53:13.293316 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:13.293299    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-968c2"]
Sep 09 20:53:13.369142 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:13.369114    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0989b0f6-8115-4152-95ae-1f52fdb2012a-installation-pull-secrets\") pod \"image-registry-cb585848c-289xj\" (UID: \"0989b0f6-8115-4152-95ae-1f52fdb2012a\") " pod="openshift-image-registry/image-registry-cb585848c-289xj"
Sep 09 20:53:13.369306 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:13.369180    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0989b0f6-8115-4152-95ae-1f52fdb2012a-ca-trust-extracted\") pod \"image-registry-cb585848c-289xj\" (UID: \"0989b0f6-8115-4152-95ae-1f52fdb2012a\") " pod="openshift-image-registry/image-registry-cb585848c-289xj"
Sep 09 20:53:13.369306 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:13.369215    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0989b0f6-8115-4152-95ae-1f52fdb2012a-registry-tls\") pod \"image-registry-cb585848c-289xj\" (UID: \"0989b0f6-8115-4152-95ae-1f52fdb2012a\") " pod="openshift-image-registry/image-registry-cb585848c-289xj"
Sep 09 20:53:13.369306 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:13.369238    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f69e7fb4-fa7d-444e-9bfa-d173783106c2-config-volume\") pod \"dns-default-968c2\" (UID: \"f69e7fb4-fa7d-444e-9bfa-d173783106c2\") " pod="openshift-dns/dns-default-968c2"
Sep 09 20:53:13.369306 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:13.369261    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6sh2m\" (UniqueName: \"kubernetes.io/projected/f69e7fb4-fa7d-444e-9bfa-d173783106c2-kube-api-access-6sh2m\") pod \"dns-default-968c2\" (UID: \"f69e7fb4-fa7d-444e-9bfa-d173783106c2\") " pod="openshift-dns/dns-default-968c2"
Sep 09 20:53:13.369306 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:13.369290    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0989b0f6-8115-4152-95ae-1f52fdb2012a-registry-certificates\") pod \"image-registry-cb585848c-289xj\" (UID: \"0989b0f6-8115-4152-95ae-1f52fdb2012a\") " pod="openshift-image-registry/image-registry-cb585848c-289xj"
Sep 09 20:53:13.369587 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:13.369315    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4ff8d077-2ba3-4e24-997c-5f0a263c5bb0-cert\") pod \"ingress-canary-hrrp2\" (UID: \"4ff8d077-2ba3-4e24-997c-5f0a263c5bb0\") " pod="openshift-ingress-canary/ingress-canary-hrrp2"
Sep 09 20:53:13.369587 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:13.369394    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkjng\" (UniqueName: \"kubernetes.io/projected/0989b0f6-8115-4152-95ae-1f52fdb2012a-kube-api-access-tkjng\") pod \"image-registry-cb585848c-289xj\" (UID: \"0989b0f6-8115-4152-95ae-1f52fdb2012a\") " pod="openshift-image-registry/image-registry-cb585848c-289xj"
Sep 09 20:53:13.369587 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:13.369456    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bcvt\" (UniqueName: \"kubernetes.io/projected/4ff8d077-2ba3-4e24-997c-5f0a263c5bb0-kube-api-access-8bcvt\") pod \"ingress-canary-hrrp2\" (UID: \"4ff8d077-2ba3-4e24-997c-5f0a263c5bb0\") " pod="openshift-ingress-canary/ingress-canary-hrrp2"
Sep 09 20:53:13.369587 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:13.369492    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0989b0f6-8115-4152-95ae-1f52fdb2012a-image-registry-private-configuration\") pod \"image-registry-cb585848c-289xj\" (UID: \"0989b0f6-8115-4152-95ae-1f52fdb2012a\") " pod="openshift-image-registry/image-registry-cb585848c-289xj"
Sep 09 20:53:13.369587 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:13.369518    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0989b0f6-8115-4152-95ae-1f52fdb2012a-trusted-ca\") pod \"image-registry-cb585848c-289xj\" (UID: \"0989b0f6-8115-4152-95ae-1f52fdb2012a\") " pod="openshift-image-registry/image-registry-cb585848c-289xj"
Sep 09 20:53:13.369778 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:13.369586    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0989b0f6-8115-4152-95ae-1f52fdb2012a-bound-sa-token\") pod \"image-registry-cb585848c-289xj\" (UID: \"0989b0f6-8115-4152-95ae-1f52fdb2012a\") " pod="openshift-image-registry/image-registry-cb585848c-289xj"
Sep 09 20:53:13.369778 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:13.369633    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f69e7fb4-fa7d-444e-9bfa-d173783106c2-metrics-tls\") pod \"dns-default-968c2\" (UID: \"f69e7fb4-fa7d-444e-9bfa-d173783106c2\") " pod="openshift-dns/dns-default-968c2"
Sep 09 20:53:13.369778 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:13.369659    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f69e7fb4-fa7d-444e-9bfa-d173783106c2-tmp-dir\") pod \"dns-default-968c2\" (UID: \"f69e7fb4-fa7d-444e-9bfa-d173783106c2\") " pod="openshift-dns/dns-default-968c2"
Sep 09 20:53:13.470643 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:13.470616    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0989b0f6-8115-4152-95ae-1f52fdb2012a-ca-trust-extracted\") pod \"image-registry-cb585848c-289xj\" (UID: \"0989b0f6-8115-4152-95ae-1f52fdb2012a\") " pod="openshift-image-registry/image-registry-cb585848c-289xj"
Sep 09 20:53:13.471117 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:13.470651    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0989b0f6-8115-4152-95ae-1f52fdb2012a-registry-tls\") pod \"image-registry-cb585848c-289xj\" (UID: \"0989b0f6-8115-4152-95ae-1f52fdb2012a\") " pod="openshift-image-registry/image-registry-cb585848c-289xj"
Sep 09 20:53:13.471117 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:13.470675    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f69e7fb4-fa7d-444e-9bfa-d173783106c2-config-volume\") pod \"dns-default-968c2\" (UID: \"f69e7fb4-fa7d-444e-9bfa-d173783106c2\") " pod="openshift-dns/dns-default-968c2"
Sep 09 20:53:13.471117 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:13.470698    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sh2m\" (UniqueName: \"kubernetes.io/projected/f69e7fb4-fa7d-444e-9bfa-d173783106c2-kube-api-access-6sh2m\") pod \"dns-default-968c2\" (UID: \"f69e7fb4-fa7d-444e-9bfa-d173783106c2\") " pod="openshift-dns/dns-default-968c2"
Sep 09 20:53:13.471117 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:13.470724    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0989b0f6-8115-4152-95ae-1f52fdb2012a-registry-certificates\") pod \"image-registry-cb585848c-289xj\" (UID: \"0989b0f6-8115-4152-95ae-1f52fdb2012a\") " pod="openshift-image-registry/image-registry-cb585848c-289xj"
Sep 09 20:53:13.471117 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:13.470752    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4ff8d077-2ba3-4e24-997c-5f0a263c5bb0-cert\") pod \"ingress-canary-hrrp2\" (UID: \"4ff8d077-2ba3-4e24-997c-5f0a263c5bb0\") " pod="openshift-ingress-canary/ingress-canary-hrrp2"
Sep 09 20:53:13.471117 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:13.470773    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkjng\" (UniqueName: \"kubernetes.io/projected/0989b0f6-8115-4152-95ae-1f52fdb2012a-kube-api-access-tkjng\") pod \"image-registry-cb585848c-289xj\" (UID: \"0989b0f6-8115-4152-95ae-1f52fdb2012a\") " pod="openshift-image-registry/image-registry-cb585848c-289xj"
Sep 09 20:53:13.471117 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:13.470803    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bcvt\" (UniqueName: \"kubernetes.io/projected/4ff8d077-2ba3-4e24-997c-5f0a263c5bb0-kube-api-access-8bcvt\") pod \"ingress-canary-hrrp2\" (UID: \"4ff8d077-2ba3-4e24-997c-5f0a263c5bb0\") " pod="openshift-ingress-canary/ingress-canary-hrrp2"
Sep 09 20:53:13.471117 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:13.470833    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0989b0f6-8115-4152-95ae-1f52fdb2012a-image-registry-private-configuration\") pod \"image-registry-cb585848c-289xj\" (UID: \"0989b0f6-8115-4152-95ae-1f52fdb2012a\") " pod="openshift-image-registry/image-registry-cb585848c-289xj"
Sep 09 20:53:13.471117 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:53:13.470840    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 20:53:13.471117 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:53:13.470861    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-cb585848c-289xj: secret "image-registry-tls" not found
Sep 09 20:53:13.471117 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:53:13.470936    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0989b0f6-8115-4152-95ae-1f52fdb2012a-registry-tls podName:0989b0f6-8115-4152-95ae-1f52fdb2012a nodeName:}" failed. No retries permitted until 2026-09-09 20:53:13.970912404 +0000 UTC m=+37.084598241 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0989b0f6-8115-4152-95ae-1f52fdb2012a-registry-tls") pod "image-registry-cb585848c-289xj" (UID: "0989b0f6-8115-4152-95ae-1f52fdb2012a") : secret "image-registry-tls" not found
Sep 09 20:53:13.471684 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:13.470857    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0989b0f6-8115-4152-95ae-1f52fdb2012a-trusted-ca\") pod \"image-registry-cb585848c-289xj\" (UID: \"0989b0f6-8115-4152-95ae-1f52fdb2012a\") " pod="openshift-image-registry/image-registry-cb585848c-289xj"
Sep 09 20:53:13.471684 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:13.471196    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0989b0f6-8115-4152-95ae-1f52fdb2012a-bound-sa-token\") pod \"image-registry-cb585848c-289xj\" (UID: \"0989b0f6-8115-4152-95ae-1f52fdb2012a\") " pod="openshift-image-registry/image-registry-cb585848c-289xj"
Sep 09 20:53:13.471684 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:13.471255    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f69e7fb4-fa7d-444e-9bfa-d173783106c2-metrics-tls\") pod \"dns-default-968c2\" (UID: \"f69e7fb4-fa7d-444e-9bfa-d173783106c2\") " pod="openshift-dns/dns-default-968c2"
Sep 09 20:53:13.471684 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:13.471278    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f69e7fb4-fa7d-444e-9bfa-d173783106c2-tmp-dir\") pod \"dns-default-968c2\" (UID: \"f69e7fb4-fa7d-444e-9bfa-d173783106c2\") " pod="openshift-dns/dns-default-968c2"
Sep 09 20:53:13.471684 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:13.471307    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0989b0f6-8115-4152-95ae-1f52fdb2012a-ca-trust-extracted\") pod \"image-registry-cb585848c-289xj\" (UID: \"0989b0f6-8115-4152-95ae-1f52fdb2012a\") " pod="openshift-image-registry/image-registry-cb585848c-289xj"
Sep 09 20:53:13.471684 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:53:13.471418    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 20:53:13.471684 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:53:13.471467    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f69e7fb4-fa7d-444e-9bfa-d173783106c2-metrics-tls podName:f69e7fb4-fa7d-444e-9bfa-d173783106c2 nodeName:}" failed. No retries permitted until 2026-09-09 20:53:13.971450908 +0000 UTC m=+37.085136746 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f69e7fb4-fa7d-444e-9bfa-d173783106c2-metrics-tls") pod "dns-default-968c2" (UID: "f69e7fb4-fa7d-444e-9bfa-d173783106c2") : secret "dns-default-metrics-tls" not found
Sep 09 20:53:13.471684 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:13.471465    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f69e7fb4-fa7d-444e-9bfa-d173783106c2-config-volume\") pod \"dns-default-968c2\" (UID: \"f69e7fb4-fa7d-444e-9bfa-d173783106c2\") " pod="openshift-dns/dns-default-968c2"
Sep 09 20:53:13.471684 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:13.471515    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0989b0f6-8115-4152-95ae-1f52fdb2012a-installation-pull-secrets\") pod \"image-registry-cb585848c-289xj\" (UID: \"0989b0f6-8115-4152-95ae-1f52fdb2012a\") " pod="openshift-image-registry/image-registry-cb585848c-289xj"
Sep 09 20:53:13.471684 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:13.471620    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f69e7fb4-fa7d-444e-9bfa-d173783106c2-tmp-dir\") pod \"dns-default-968c2\" (UID: \"f69e7fb4-fa7d-444e-9bfa-d173783106c2\") " pod="openshift-dns/dns-default-968c2"
Sep 09 20:53:13.472004 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:13.471848    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0989b0f6-8115-4152-95ae-1f52fdb2012a-trusted-ca\") pod \"image-registry-cb585848c-289xj\" (UID: \"0989b0f6-8115-4152-95ae-1f52fdb2012a\") " pod="openshift-image-registry/image-registry-cb585848c-289xj"
Sep 09 20:53:13.472004 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:13.471933    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0989b0f6-8115-4152-95ae-1f52fdb2012a-registry-certificates\") pod \"image-registry-cb585848c-289xj\" (UID: \"0989b0f6-8115-4152-95ae-1f52fdb2012a\") " pod="openshift-image-registry/image-registry-cb585848c-289xj"
Sep 09 20:53:13.476277 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:13.476252    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0989b0f6-8115-4152-95ae-1f52fdb2012a-installation-pull-secrets\") pod \"image-registry-cb585848c-289xj\" (UID: \"0989b0f6-8115-4152-95ae-1f52fdb2012a\") " pod="openshift-image-registry/image-registry-cb585848c-289xj"
Sep 09 20:53:13.476277 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:13.476266    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4ff8d077-2ba3-4e24-997c-5f0a263c5bb0-cert\") pod \"ingress-canary-hrrp2\" (UID: \"4ff8d077-2ba3-4e24-997c-5f0a263c5bb0\") " pod="openshift-ingress-canary/ingress-canary-hrrp2"
Sep 09 20:53:13.476438 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:13.476308    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0989b0f6-8115-4152-95ae-1f52fdb2012a-image-registry-private-configuration\") pod \"image-registry-cb585848c-289xj\" (UID: \"0989b0f6-8115-4152-95ae-1f52fdb2012a\") " pod="openshift-image-registry/image-registry-cb585848c-289xj"
Sep 09 20:53:13.480671 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:13.480646    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sh2m\" (UniqueName: \"kubernetes.io/projected/f69e7fb4-fa7d-444e-9bfa-d173783106c2-kube-api-access-6sh2m\") pod \"dns-default-968c2\" (UID: \"f69e7fb4-fa7d-444e-9bfa-d173783106c2\") " pod="openshift-dns/dns-default-968c2"
Sep 09 20:53:13.481195 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:13.481167    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bcvt\" (UniqueName: \"kubernetes.io/projected/4ff8d077-2ba3-4e24-997c-5f0a263c5bb0-kube-api-access-8bcvt\") pod \"ingress-canary-hrrp2\" (UID: \"4ff8d077-2ba3-4e24-997c-5f0a263c5bb0\") " pod="openshift-ingress-canary/ingress-canary-hrrp2"
Sep 09 20:53:13.481265 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:13.481246    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0989b0f6-8115-4152-95ae-1f52fdb2012a-bound-sa-token\") pod \"image-registry-cb585848c-289xj\" (UID: \"0989b0f6-8115-4152-95ae-1f52fdb2012a\") " pod="openshift-image-registry/image-registry-cb585848c-289xj"
Sep 09 20:53:13.481313 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:13.481263    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkjng\" (UniqueName: \"kubernetes.io/projected/0989b0f6-8115-4152-95ae-1f52fdb2012a-kube-api-access-tkjng\") pod \"image-registry-cb585848c-289xj\" (UID: \"0989b0f6-8115-4152-95ae-1f52fdb2012a\") " pod="openshift-image-registry/image-registry-cb585848c-289xj"
Sep 09 20:53:13.503082 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:13.503057    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:53:13.597865 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:13.597838    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-hrrp2"
Sep 09 20:53:13.766756 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:13.766720    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:53:13.880685 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:13.880659    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-hrrp2"]
Sep 09 20:53:13.894503 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:53:13.894477    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ff8d077_2ba3_4e24_997c_5f0a263c5bb0.slice/crio-89a995d63d8020264fef384edb97ecd5cbe87f3689e5fe4ccc194749a697a1ca WatchSource:0}: Error finding container 89a995d63d8020264fef384edb97ecd5cbe87f3689e5fe4ccc194749a697a1ca: Status 404 returned error can't find the container with id 89a995d63d8020264fef384edb97ecd5cbe87f3689e5fe4ccc194749a697a1ca
Sep 09 20:53:13.974800 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:13.974769    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f69e7fb4-fa7d-444e-9bfa-d173783106c2-metrics-tls\") pod \"dns-default-968c2\" (UID: \"f69e7fb4-fa7d-444e-9bfa-d173783106c2\") " pod="openshift-dns/dns-default-968c2"
Sep 09 20:53:13.974927 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:13.974841    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0989b0f6-8115-4152-95ae-1f52fdb2012a-registry-tls\") pod \"image-registry-cb585848c-289xj\" (UID: \"0989b0f6-8115-4152-95ae-1f52fdb2012a\") " pod="openshift-image-registry/image-registry-cb585848c-289xj"
Sep 09 20:53:13.974927 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:53:13.974905    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 20:53:13.974991 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:53:13.974951    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 20:53:13.974991 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:53:13.974962    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-cb585848c-289xj: secret "image-registry-tls" not found
Sep 09 20:53:13.974991 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:53:13.974972    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f69e7fb4-fa7d-444e-9bfa-d173783106c2-metrics-tls podName:f69e7fb4-fa7d-444e-9bfa-d173783106c2 nodeName:}" failed. No retries permitted until 2026-09-09 20:53:14.974955528 +0000 UTC m=+38.088641365 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f69e7fb4-fa7d-444e-9bfa-d173783106c2-metrics-tls") pod "dns-default-968c2" (UID: "f69e7fb4-fa7d-444e-9bfa-d173783106c2") : secret "dns-default-metrics-tls" not found
Sep 09 20:53:13.975089 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:53:13.975000    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0989b0f6-8115-4152-95ae-1f52fdb2012a-registry-tls podName:0989b0f6-8115-4152-95ae-1f52fdb2012a nodeName:}" failed. No retries permitted until 2026-09-09 20:53:14.974988888 +0000 UTC m=+38.088674725 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0989b0f6-8115-4152-95ae-1f52fdb2012a-registry-tls") pod "image-registry-cb585848c-289xj" (UID: "0989b0f6-8115-4152-95ae-1f52fdb2012a") : secret "image-registry-tls" not found
Sep 09 20:53:14.199317 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:14.199228    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:53:14.263097 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:14.263066    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:53:14.403881 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:14.403847    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v74xn"
Sep 09 20:53:14.404044 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:14.403848    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2c9kq"
Sep 09 20:53:14.404107 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:14.403857    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t8tk8"
Sep 09 20:53:14.407554 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:14.407537    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 09 20:53:14.408873 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:14.408853    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-j2slq\""
Sep 09 20:53:14.408983 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:14.408944    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 09 20:53:14.408983 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:14.408957    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 09 20:53:14.409163 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:14.408998    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-njm8p\""
Sep 09 20:53:14.409163 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:14.409056    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 09 20:53:14.650293 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:14.650258    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-hrrp2" event={"ID":"4ff8d077-2ba3-4e24-997c-5f0a263c5bb0","Type":"ContainerStarted","Data":"89a995d63d8020264fef384edb97ecd5cbe87f3689e5fe4ccc194749a697a1ca"}
Sep 09 20:53:14.653635 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:14.653602    2576 generic.go:356] "Generic (PLEG): container finished" podID="3ede0a95-3907-4d9c-aa45-acf822104e28" containerID="2e9818164645f5cc6ddb2b2a350c77d2838f6c796b47e97687fb71aaecf20c44" exitCode=0
Sep 09 20:53:14.653749 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:14.653659    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d8z7g" event={"ID":"3ede0a95-3907-4d9c-aa45-acf822104e28","Type":"ContainerDied","Data":"2e9818164645f5cc6ddb2b2a350c77d2838f6c796b47e97687fb71aaecf20c44"}
Sep 09 20:53:14.983340 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:14.983248    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f69e7fb4-fa7d-444e-9bfa-d173783106c2-metrics-tls\") pod \"dns-default-968c2\" (UID: \"f69e7fb4-fa7d-444e-9bfa-d173783106c2\") " pod="openshift-dns/dns-default-968c2"
Sep 09 20:53:14.983340 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:14.983332    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0989b0f6-8115-4152-95ae-1f52fdb2012a-registry-tls\") pod \"image-registry-cb585848c-289xj\" (UID: \"0989b0f6-8115-4152-95ae-1f52fdb2012a\") " pod="openshift-image-registry/image-registry-cb585848c-289xj"
Sep 09 20:53:14.983555 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:53:14.983484    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 20:53:14.983627 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:53:14.983496    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 20:53:14.983697 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:53:14.983632    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-cb585848c-289xj: secret "image-registry-tls" not found
Sep 09 20:53:14.983697 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:53:14.983648    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f69e7fb4-fa7d-444e-9bfa-d173783106c2-metrics-tls podName:f69e7fb4-fa7d-444e-9bfa-d173783106c2 nodeName:}" failed. No retries permitted until 2026-09-09 20:53:16.983625135 +0000 UTC m=+40.097310974 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f69e7fb4-fa7d-444e-9bfa-d173783106c2-metrics-tls") pod "dns-default-968c2" (UID: "f69e7fb4-fa7d-444e-9bfa-d173783106c2") : secret "dns-default-metrics-tls" not found
Sep 09 20:53:14.983697 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:53:14.983680    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0989b0f6-8115-4152-95ae-1f52fdb2012a-registry-tls podName:0989b0f6-8115-4152-95ae-1f52fdb2012a nodeName:}" failed. No retries permitted until 2026-09-09 20:53:16.983665365 +0000 UTC m=+40.097351201 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0989b0f6-8115-4152-95ae-1f52fdb2012a-registry-tls") pod "image-registry-cb585848c-289xj" (UID: "0989b0f6-8115-4152-95ae-1f52fdb2012a") : secret "image-registry-tls" not found
Sep 09 20:53:15.660330 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:15.660295    2576 generic.go:356] "Generic (PLEG): container finished" podID="3ede0a95-3907-4d9c-aa45-acf822104e28" containerID="40452bfc6abc2dc9782d49203ae8618c9495ef981e63971d480021e533b67009" exitCode=0
Sep 09 20:53:15.660759 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:15.660382    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d8z7g" event={"ID":"3ede0a95-3907-4d9c-aa45-acf822104e28","Type":"ContainerDied","Data":"40452bfc6abc2dc9782d49203ae8618c9495ef981e63971d480021e533b67009"}
Sep 09 20:53:16.518249 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:16.518068    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:53:16.634419 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:16.634386    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:53:16.665399 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:16.665353    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d8z7g" event={"ID":"3ede0a95-3907-4d9c-aa45-acf822104e28","Type":"ContainerStarted","Data":"e312b10a479eb46f1f2177e61f8e7f1a5b311f2ddee8127f6f1f9cc500633f13"}
Sep 09 20:53:16.693823 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:16.693783    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-d8z7g" podStartSLOduration=8.514516637 podStartE2EDuration="38.693768276s" totalImagesPullingTime="30.179251639s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:52:38 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:53:16.692030492 +0000 UTC m=+39.805716351" watchObservedRunningTime="2026-09-09 20:53:16.693768276 +0000 UTC m=+39.807454156"
Sep 09 20:53:16.855122 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:16.855034    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:53:16.941046 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:16.940988    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:53:17.000698 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:17.000672    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0989b0f6-8115-4152-95ae-1f52fdb2012a-registry-tls\") pod \"image-registry-cb585848c-289xj\" (UID: \"0989b0f6-8115-4152-95ae-1f52fdb2012a\") " pod="openshift-image-registry/image-registry-cb585848c-289xj"
Sep 09 20:53:17.000816 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:17.000735    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f69e7fb4-fa7d-444e-9bfa-d173783106c2-metrics-tls\") pod \"dns-default-968c2\" (UID: \"f69e7fb4-fa7d-444e-9bfa-d173783106c2\") " pod="openshift-dns/dns-default-968c2"
Sep 09 20:53:17.000873 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:53:17.000821    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 20:53:17.000873 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:53:17.000837    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-cb585848c-289xj: secret "image-registry-tls" not found
Sep 09 20:53:17.000873 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:53:17.000867    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 20:53:17.000961 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:53:17.000888    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0989b0f6-8115-4152-95ae-1f52fdb2012a-registry-tls podName:0989b0f6-8115-4152-95ae-1f52fdb2012a nodeName:}" failed. No retries permitted until 2026-09-09 20:53:21.000873735 +0000 UTC m=+44.114559607 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0989b0f6-8115-4152-95ae-1f52fdb2012a-registry-tls") pod "image-registry-cb585848c-289xj" (UID: "0989b0f6-8115-4152-95ae-1f52fdb2012a") : secret "image-registry-tls" not found
Sep 09 20:53:17.000961 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:53:17.000908    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f69e7fb4-fa7d-444e-9bfa-d173783106c2-metrics-tls podName:f69e7fb4-fa7d-444e-9bfa-d173783106c2 nodeName:}" failed. No retries permitted until 2026-09-09 20:53:21.000896543 +0000 UTC m=+44.114582399 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f69e7fb4-fa7d-444e-9bfa-d173783106c2-metrics-tls") pod "dns-default-968c2" (UID: "f69e7fb4-fa7d-444e-9bfa-d173783106c2") : secret "dns-default-metrics-tls" not found
Sep 09 20:53:17.042021 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:17.041995    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:53:17.669142 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:17.669109    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-hrrp2" event={"ID":"4ff8d077-2ba3-4e24-997c-5f0a263c5bb0","Type":"ContainerStarted","Data":"d63349db4ae30639935019c1c134a813b0f577bd11379ebd6fbda3f9986a1de8"}
Sep 09 20:53:17.692200 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:17.692150    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-hrrp2" podStartSLOduration=1.986646954 podStartE2EDuration="4.692135329s" totalImagesPullingTime="2.705488375s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:53:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:53:17.691156226 +0000 UTC m=+40.804842085" watchObservedRunningTime="2026-09-09 20:53:17.692135329 +0000 UTC m=+40.805821187"
Sep 09 20:53:18.399258 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:18.399215    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:53:18.499108 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:18.499071    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:53:21.029620 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:21.029584    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f69e7fb4-fa7d-444e-9bfa-d173783106c2-metrics-tls\") pod \"dns-default-968c2\" (UID: \"f69e7fb4-fa7d-444e-9bfa-d173783106c2\") " pod="openshift-dns/dns-default-968c2"
Sep 09 20:53:21.030038 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:21.029647    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0989b0f6-8115-4152-95ae-1f52fdb2012a-registry-tls\") pod \"image-registry-cb585848c-289xj\" (UID: \"0989b0f6-8115-4152-95ae-1f52fdb2012a\") " pod="openshift-image-registry/image-registry-cb585848c-289xj"
Sep 09 20:53:21.030038 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:53:21.029736    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 20:53:21.030038 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:53:21.029737    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 20:53:21.030038 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:53:21.029796    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f69e7fb4-fa7d-444e-9bfa-d173783106c2-metrics-tls podName:f69e7fb4-fa7d-444e-9bfa-d173783106c2 nodeName:}" failed. No retries permitted until 2026-09-09 20:53:29.02978181 +0000 UTC m=+52.143467647 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f69e7fb4-fa7d-444e-9bfa-d173783106c2-metrics-tls") pod "dns-default-968c2" (UID: "f69e7fb4-fa7d-444e-9bfa-d173783106c2") : secret "dns-default-metrics-tls" not found
Sep 09 20:53:21.030038 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:53:21.029748    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-cb585848c-289xj: secret "image-registry-tls" not found
Sep 09 20:53:21.030038 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:53:21.029862    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0989b0f6-8115-4152-95ae-1f52fdb2012a-registry-tls podName:0989b0f6-8115-4152-95ae-1f52fdb2012a nodeName:}" failed. No retries permitted until 2026-09-09 20:53:29.029851302 +0000 UTC m=+52.143537139 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0989b0f6-8115-4152-95ae-1f52fdb2012a-registry-tls") pod "image-registry-cb585848c-289xj" (UID: "0989b0f6-8115-4152-95ae-1f52fdb2012a") : secret "image-registry-tls" not found
Sep 09 20:53:21.072013 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:21.071981    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:53:21.157784 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:21.157752    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:53:21.225619 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:21.225582    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:53:24.981602 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:24.981486    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:53:25.114482 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:25.114452    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:53:25.202526 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:25.202492    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:53:26.068154 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:26.068116    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:53:26.158120 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:26.158089    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:53:27.951736 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:27.951701    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:53:29.090045 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:29.090008    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0989b0f6-8115-4152-95ae-1f52fdb2012a-registry-tls\") pod \"image-registry-cb585848c-289xj\" (UID: \"0989b0f6-8115-4152-95ae-1f52fdb2012a\") " pod="openshift-image-registry/image-registry-cb585848c-289xj"
Sep 09 20:53:29.090429 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:29.090069    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f69e7fb4-fa7d-444e-9bfa-d173783106c2-metrics-tls\") pod \"dns-default-968c2\" (UID: \"f69e7fb4-fa7d-444e-9bfa-d173783106c2\") " pod="openshift-dns/dns-default-968c2"
Sep 09 20:53:29.090429 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:53:29.090159    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 20:53:29.090429 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:53:29.090164    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 20:53:29.090429 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:53:29.090180    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-cb585848c-289xj: secret "image-registry-tls" not found
Sep 09 20:53:29.090429 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:53:29.090212    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f69e7fb4-fa7d-444e-9bfa-d173783106c2-metrics-tls podName:f69e7fb4-fa7d-444e-9bfa-d173783106c2 nodeName:}" failed. No retries permitted until 2026-09-09 20:53:45.090198499 +0000 UTC m=+68.203884336 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f69e7fb4-fa7d-444e-9bfa-d173783106c2-metrics-tls") pod "dns-default-968c2" (UID: "f69e7fb4-fa7d-444e-9bfa-d173783106c2") : secret "dns-default-metrics-tls" not found
Sep 09 20:53:29.090429 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:53:29.090233    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0989b0f6-8115-4152-95ae-1f52fdb2012a-registry-tls podName:0989b0f6-8115-4152-95ae-1f52fdb2012a nodeName:}" failed. No retries permitted until 2026-09-09 20:53:45.090220137 +0000 UTC m=+68.203905999 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0989b0f6-8115-4152-95ae-1f52fdb2012a-registry-tls") pod "image-registry-cb585848c-289xj" (UID: "0989b0f6-8115-4152-95ae-1f52fdb2012a") : secret "image-registry-tls" not found
Sep 09 20:53:37.650040 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:37.650009    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tkbs9"
Sep 09 20:53:37.833420 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:37.833383    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:53:39.662508 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:39.662465    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:53:41.283617 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:41.283545    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:53:41.359837 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:41.359808    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:53:42.408798 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:42.408764    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:53:43.184442 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:43.184357    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krdp6\" (UniqueName: \"kubernetes.io/projected/82d55335-0036-4166-a35c-65adec33932d-kube-api-access-krdp6\") pod \"network-check-target-t8tk8\" (UID: \"82d55335-0036-4166-a35c-65adec33932d\") " pod="openshift-network-diagnostics/network-check-target-t8tk8"
Sep 09 20:53:43.184647 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:43.184514    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b906e5b3-61d3-41da-9776-795376a73811-original-pull-secret\") pod \"global-pull-secret-syncer-2c9kq\" (UID: \"b906e5b3-61d3-41da-9776-795376a73811\") " pod="kube-system/global-pull-secret-syncer-2c9kq"
Sep 09 20:53:43.184647 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:43.184553    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b815b2b-983b-4afe-900d-13dd791e4778-metrics-certs\") pod \"network-metrics-daemon-v74xn\" (UID: \"5b815b2b-983b-4afe-900d-13dd791e4778\") " pod="openshift-multus/network-metrics-daemon-v74xn"
Sep 09 20:53:43.193831 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:43.193809    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 09 20:53:43.193898 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:43.193811    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 09 20:53:43.194720 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:43.194701    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 09 20:53:43.198028 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:43.198012    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b906e5b3-61d3-41da-9776-795376a73811-original-pull-secret\") pod \"global-pull-secret-syncer-2c9kq\" (UID: \"b906e5b3-61d3-41da-9776-795376a73811\") " pod="kube-system/global-pull-secret-syncer-2c9kq"
Sep 09 20:53:43.204503 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:43.204486    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 09 20:53:43.204753 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:53:43.204738    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 09 20:53:43.204821 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:53:43.204803    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b815b2b-983b-4afe-900d-13dd791e4778-metrics-certs podName:5b815b2b-983b-4afe-900d-13dd791e4778 nodeName:}" failed. No retries permitted until 2026-09-09 20:54:47.204789661 +0000 UTC m=+130.318475498 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5b815b2b-983b-4afe-900d-13dd791e4778-metrics-certs") pod "network-metrics-daemon-v74xn" (UID: "5b815b2b-983b-4afe-900d-13dd791e4778") : secret "metrics-daemon-secret" not found
Sep 09 20:53:43.206765 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:43.206750    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-krdp6\" (UniqueName: \"kubernetes.io/projected/82d55335-0036-4166-a35c-65adec33932d-kube-api-access-krdp6\") pod \"network-check-target-t8tk8\" (UID: \"82d55335-0036-4166-a35c-65adec33932d\") " pod="openshift-network-diagnostics/network-check-target-t8tk8"
Sep 09 20:53:43.219865 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:43.219845    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2c9kq"
Sep 09 20:53:43.226550 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:43.226531    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-njm8p\""
Sep 09 20:53:43.233615 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:43.233598    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t8tk8"
Sep 09 20:53:43.502250 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:43.502121    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-2c9kq"]
Sep 09 20:53:43.505817 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:53:43.505791    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb906e5b3_61d3_41da_9776_795376a73811.slice/crio-fa379323c9f4b4e0c09cbad9acb890e0376d10e2d8385e5d6234dfa3c1e41b8d WatchSource:0}: Error finding container fa379323c9f4b4e0c09cbad9acb890e0376d10e2d8385e5d6234dfa3c1e41b8d: Status 404 returned error can't find the container with id fa379323c9f4b4e0c09cbad9acb890e0376d10e2d8385e5d6234dfa3c1e41b8d
Sep 09 20:53:43.523444 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:43.523419    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-t8tk8"]
Sep 09 20:53:43.526673 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:53:43.526653    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82d55335_0036_4166_a35c_65adec33932d.slice/crio-9848e36e2bed96486b6a9920fd056ba4e84ca888d58de753744dbf88fc80329c WatchSource:0}: Error finding container 9848e36e2bed96486b6a9920fd056ba4e84ca888d58de753744dbf88fc80329c: Status 404 returned error can't find the container with id 9848e36e2bed96486b6a9920fd056ba4e84ca888d58de753744dbf88fc80329c
Sep 09 20:53:43.729540 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:43.729504    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-2c9kq" event={"ID":"b906e5b3-61d3-41da-9776-795376a73811","Type":"ContainerStarted","Data":"fa379323c9f4b4e0c09cbad9acb890e0376d10e2d8385e5d6234dfa3c1e41b8d"}
Sep 09 20:53:43.730726 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:43.730696    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-t8tk8" event={"ID":"82d55335-0036-4166-a35c-65adec33932d","Type":"ContainerStarted","Data":"9848e36e2bed96486b6a9920fd056ba4e84ca888d58de753744dbf88fc80329c"}
Sep 09 20:53:45.097158 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:45.096874    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f69e7fb4-fa7d-444e-9bfa-d173783106c2-metrics-tls\") pod \"dns-default-968c2\" (UID: \"f69e7fb4-fa7d-444e-9bfa-d173783106c2\") " pod="openshift-dns/dns-default-968c2"
Sep 09 20:53:45.097158 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:45.096954    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0989b0f6-8115-4152-95ae-1f52fdb2012a-registry-tls\") pod \"image-registry-cb585848c-289xj\" (UID: \"0989b0f6-8115-4152-95ae-1f52fdb2012a\") " pod="openshift-image-registry/image-registry-cb585848c-289xj"
Sep 09 20:53:45.097158 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:53:45.097012    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 20:53:45.097158 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:53:45.097051    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 20:53:45.097158 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:53:45.097065    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-cb585848c-289xj: secret "image-registry-tls" not found
Sep 09 20:53:45.097158 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:53:45.097103    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f69e7fb4-fa7d-444e-9bfa-d173783106c2-metrics-tls podName:f69e7fb4-fa7d-444e-9bfa-d173783106c2 nodeName:}" failed. No retries permitted until 2026-09-09 20:54:17.097080219 +0000 UTC m=+100.210766072 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f69e7fb4-fa7d-444e-9bfa-d173783106c2-metrics-tls") pod "dns-default-968c2" (UID: "f69e7fb4-fa7d-444e-9bfa-d173783106c2") : secret "dns-default-metrics-tls" not found
Sep 09 20:53:45.097158 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:53:45.097131    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0989b0f6-8115-4152-95ae-1f52fdb2012a-registry-tls podName:0989b0f6-8115-4152-95ae-1f52fdb2012a nodeName:}" failed. No retries permitted until 2026-09-09 20:54:17.097114036 +0000 UTC m=+100.210799879 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0989b0f6-8115-4152-95ae-1f52fdb2012a-registry-tls") pod "image-registry-cb585848c-289xj" (UID: "0989b0f6-8115-4152-95ae-1f52fdb2012a") : secret "image-registry-tls" not found
Sep 09 20:53:48.744854 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:48.744823    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-t8tk8" event={"ID":"82d55335-0036-4166-a35c-65adec33932d","Type":"ContainerStarted","Data":"5d78cd8203107d79a807ea7275f909abed0288173034deb465cfa102714d2795"}
Sep 09 20:53:48.745304 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:48.744877    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-t8tk8"
Sep 09 20:53:48.746282 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:48.746262    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-2c9kq" event={"ID":"b906e5b3-61d3-41da-9776-795376a73811","Type":"ContainerStarted","Data":"ae2d61ba3b6aa7333d622292d1c1162159d7f0bc1e74b8da827702e88d4ddb1c"}
Sep 09 20:53:48.792006 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:48.791965    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-2c9kq" podStartSLOduration=66.424232382 podStartE2EDuration="1m10.791952184s" totalImagesPullingTime="4.367719802s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:52:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:53:48.791618563 +0000 UTC m=+71.905304424" watchObservedRunningTime="2026-09-09 20:53:48.791952184 +0000 UTC m=+71.905638037"
Sep 09 20:53:48.792127 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:53:48.792089    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-t8tk8" podStartSLOduration=66.448989478 podStartE2EDuration="1m10.792082046s" totalImagesPullingTime="4.343092568s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:52:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:53:48.769059902 +0000 UTC m=+71.882745763" watchObservedRunningTime="2026-09-09 20:53:48.792082046 +0000 UTC m=+71.905767905"
Sep 09 20:54:04.122223 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:04.122087    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:04.226660 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:04.226617    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:04.302339 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:04.302310    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:08.991779 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:08.991746    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:09.091286 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:09.091242    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:12.566309 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:12.566271    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:12.569580 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:12.569560    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-l852l"]
Sep 09 20:54:12.573624 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:12.573610    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-l852l"
Sep 09 20:54:12.576240 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:12.576217    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 09 20:54:12.576339 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:12.576248    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 09 20:54:12.576339 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:12.576218    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 09 20:54:12.577331 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:12.577317    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-ksvhl\""
Sep 09 20:54:12.577453 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:12.577321    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 09 20:54:12.584697 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:12.584680    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-l852l"]
Sep 09 20:54:12.628293 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:12.628263    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:12.688562 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:12.688535    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c6e98f74-7546-4383-89e2-e8ea2c8197e3-crio-socket\") pod \"insights-runtime-extractor-l852l\" (UID: \"c6e98f74-7546-4383-89e2-e8ea2c8197e3\") " pod="openshift-insights/insights-runtime-extractor-l852l"
Sep 09 20:54:12.688669 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:12.688568    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c6e98f74-7546-4383-89e2-e8ea2c8197e3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l852l\" (UID: \"c6e98f74-7546-4383-89e2-e8ea2c8197e3\") " pod="openshift-insights/insights-runtime-extractor-l852l"
Sep 09 20:54:12.688669 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:12.688648    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c6e98f74-7546-4383-89e2-e8ea2c8197e3-data-volume\") pod \"insights-runtime-extractor-l852l\" (UID: \"c6e98f74-7546-4383-89e2-e8ea2c8197e3\") " pod="openshift-insights/insights-runtime-extractor-l852l"
Sep 09 20:54:12.688787 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:12.688677    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c6e98f74-7546-4383-89e2-e8ea2c8197e3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l852l\" (UID: \"c6e98f74-7546-4383-89e2-e8ea2c8197e3\") " pod="openshift-insights/insights-runtime-extractor-l852l"
Sep 09 20:54:12.688787 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:12.688751    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqrw8\" (UniqueName: \"kubernetes.io/projected/c6e98f74-7546-4383-89e2-e8ea2c8197e3-kube-api-access-pqrw8\") pod \"insights-runtime-extractor-l852l\" (UID: \"c6e98f74-7546-4383-89e2-e8ea2c8197e3\") " pod="openshift-insights/insights-runtime-extractor-l852l"
Sep 09 20:54:12.789277 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:12.789231    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c6e98f74-7546-4383-89e2-e8ea2c8197e3-data-volume\") pod \"insights-runtime-extractor-l852l\" (UID: \"c6e98f74-7546-4383-89e2-e8ea2c8197e3\") " pod="openshift-insights/insights-runtime-extractor-l852l"
Sep 09 20:54:12.789277 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:12.789289    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c6e98f74-7546-4383-89e2-e8ea2c8197e3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l852l\" (UID: \"c6e98f74-7546-4383-89e2-e8ea2c8197e3\") " pod="openshift-insights/insights-runtime-extractor-l852l"
Sep 09 20:54:12.789498 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:54:12.789414    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 20:54:12.789498 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:12.789471    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqrw8\" (UniqueName: \"kubernetes.io/projected/c6e98f74-7546-4383-89e2-e8ea2c8197e3-kube-api-access-pqrw8\") pod \"insights-runtime-extractor-l852l\" (UID: \"c6e98f74-7546-4383-89e2-e8ea2c8197e3\") " pod="openshift-insights/insights-runtime-extractor-l852l"
Sep 09 20:54:12.789498 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:54:12.789476    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6e98f74-7546-4383-89e2-e8ea2c8197e3-insights-runtime-extractor-tls podName:c6e98f74-7546-4383-89e2-e8ea2c8197e3 nodeName:}" failed. No retries permitted until 2026-09-09 20:54:13.289459817 +0000 UTC m=+96.403145657 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c6e98f74-7546-4383-89e2-e8ea2c8197e3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-l852l" (UID: "c6e98f74-7546-4383-89e2-e8ea2c8197e3") : secret "insights-runtime-extractor-tls" not found
Sep 09 20:54:12.789659 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:12.789538    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c6e98f74-7546-4383-89e2-e8ea2c8197e3-crio-socket\") pod \"insights-runtime-extractor-l852l\" (UID: \"c6e98f74-7546-4383-89e2-e8ea2c8197e3\") " pod="openshift-insights/insights-runtime-extractor-l852l"
Sep 09 20:54:12.789659 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:12.789568    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c6e98f74-7546-4383-89e2-e8ea2c8197e3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l852l\" (UID: \"c6e98f74-7546-4383-89e2-e8ea2c8197e3\") " pod="openshift-insights/insights-runtime-extractor-l852l"
Sep 09 20:54:12.789659 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:54:12.789656    2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 20:54:12.789774 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:12.789671    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c6e98f74-7546-4383-89e2-e8ea2c8197e3-crio-socket\") pod \"insights-runtime-extractor-l852l\" (UID: \"c6e98f74-7546-4383-89e2-e8ea2c8197e3\") " pod="openshift-insights/insights-runtime-extractor-l852l"
Sep 09 20:54:12.789774 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:12.789654    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c6e98f74-7546-4383-89e2-e8ea2c8197e3-data-volume\") pod \"insights-runtime-extractor-l852l\" (UID: \"c6e98f74-7546-4383-89e2-e8ea2c8197e3\") " pod="openshift-insights/insights-runtime-extractor-l852l"
Sep 09 20:54:12.789774 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:54:12.789714    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c6e98f74-7546-4383-89e2-e8ea2c8197e3-kube-rbac-proxy-cm podName:c6e98f74-7546-4383-89e2-e8ea2c8197e3 nodeName:}" failed. No retries permitted until 2026-09-09 20:54:13.289700725 +0000 UTC m=+96.403386561 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c6e98f74-7546-4383-89e2-e8ea2c8197e3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-l852l" (UID: "c6e98f74-7546-4383-89e2-e8ea2c8197e3") : configmap "kube-rbac-proxy" not found
Sep 09 20:54:12.802945 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:12.802920    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqrw8\" (UniqueName: \"kubernetes.io/projected/c6e98f74-7546-4383-89e2-e8ea2c8197e3-kube-api-access-pqrw8\") pod \"insights-runtime-extractor-l852l\" (UID: \"c6e98f74-7546-4383-89e2-e8ea2c8197e3\") " pod="openshift-insights/insights-runtime-extractor-l852l"
Sep 09 20:54:13.295079 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:13.295045    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c6e98f74-7546-4383-89e2-e8ea2c8197e3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l852l\" (UID: \"c6e98f74-7546-4383-89e2-e8ea2c8197e3\") " pod="openshift-insights/insights-runtime-extractor-l852l"
Sep 09 20:54:13.295243 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:13.295104    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c6e98f74-7546-4383-89e2-e8ea2c8197e3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l852l\" (UID: \"c6e98f74-7546-4383-89e2-e8ea2c8197e3\") " pod="openshift-insights/insights-runtime-extractor-l852l"
Sep 09 20:54:13.295243 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:54:13.295173    2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 20:54:13.295243 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:54:13.295186    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 20:54:13.295243 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:54:13.295232    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c6e98f74-7546-4383-89e2-e8ea2c8197e3-kube-rbac-proxy-cm podName:c6e98f74-7546-4383-89e2-e8ea2c8197e3 nodeName:}" failed. No retries permitted until 2026-09-09 20:54:14.295219277 +0000 UTC m=+97.408905113 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c6e98f74-7546-4383-89e2-e8ea2c8197e3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-l852l" (UID: "c6e98f74-7546-4383-89e2-e8ea2c8197e3") : configmap "kube-rbac-proxy" not found
Sep 09 20:54:13.295243 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:54:13.295246    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6e98f74-7546-4383-89e2-e8ea2c8197e3-insights-runtime-extractor-tls podName:c6e98f74-7546-4383-89e2-e8ea2c8197e3 nodeName:}" failed. No retries permitted until 2026-09-09 20:54:14.29524009 +0000 UTC m=+97.408925927 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c6e98f74-7546-4383-89e2-e8ea2c8197e3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-l852l" (UID: "c6e98f74-7546-4383-89e2-e8ea2c8197e3") : secret "insights-runtime-extractor-tls" not found
Sep 09 20:54:14.302288 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:14.302249    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c6e98f74-7546-4383-89e2-e8ea2c8197e3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l852l\" (UID: \"c6e98f74-7546-4383-89e2-e8ea2c8197e3\") " pod="openshift-insights/insights-runtime-extractor-l852l"
Sep 09 20:54:14.302759 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:14.302331    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c6e98f74-7546-4383-89e2-e8ea2c8197e3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l852l\" (UID: \"c6e98f74-7546-4383-89e2-e8ea2c8197e3\") " pod="openshift-insights/insights-runtime-extractor-l852l"
Sep 09 20:54:14.302759 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:54:14.302425    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 20:54:14.302759 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:54:14.302435    2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 20:54:14.302759 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:54:14.302511    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6e98f74-7546-4383-89e2-e8ea2c8197e3-insights-runtime-extractor-tls podName:c6e98f74-7546-4383-89e2-e8ea2c8197e3 nodeName:}" failed. No retries permitted until 2026-09-09 20:54:16.302495273 +0000 UTC m=+99.416181113 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c6e98f74-7546-4383-89e2-e8ea2c8197e3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-l852l" (UID: "c6e98f74-7546-4383-89e2-e8ea2c8197e3") : secret "insights-runtime-extractor-tls" not found
Sep 09 20:54:14.302759 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:54:14.302527    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c6e98f74-7546-4383-89e2-e8ea2c8197e3-kube-rbac-proxy-cm podName:c6e98f74-7546-4383-89e2-e8ea2c8197e3 nodeName:}" failed. No retries permitted until 2026-09-09 20:54:16.302520304 +0000 UTC m=+99.416206141 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c6e98f74-7546-4383-89e2-e8ea2c8197e3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-l852l" (UID: "c6e98f74-7546-4383-89e2-e8ea2c8197e3") : configmap "kube-rbac-proxy" not found
Sep 09 20:54:14.691014 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:14.690986    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-wvw9h_09f40168-2bb2-4eff-8a3f-7ef28a149eed/dns-node-resolver/0.log"
Sep 09 20:54:15.091083 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:15.091059    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-dcv48_e706fb03-415d-407d-b105-bd06474f6882/node-ca/0.log"
Sep 09 20:54:16.092932 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:16.092897    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-hrrp2_4ff8d077-2ba3-4e24-997c-5f0a263c5bb0/serve-healthcheck-canary/0.log"
Sep 09 20:54:16.316458 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:16.316411    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c6e98f74-7546-4383-89e2-e8ea2c8197e3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l852l\" (UID: \"c6e98f74-7546-4383-89e2-e8ea2c8197e3\") " pod="openshift-insights/insights-runtime-extractor-l852l"
Sep 09 20:54:16.316643 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:16.316496    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c6e98f74-7546-4383-89e2-e8ea2c8197e3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l852l\" (UID: \"c6e98f74-7546-4383-89e2-e8ea2c8197e3\") " pod="openshift-insights/insights-runtime-extractor-l852l"
Sep 09 20:54:16.316643 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:54:16.316567    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 20:54:16.316643 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:54:16.316574    2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 20:54:16.316643 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:54:16.316631    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c6e98f74-7546-4383-89e2-e8ea2c8197e3-kube-rbac-proxy-cm podName:c6e98f74-7546-4383-89e2-e8ea2c8197e3 nodeName:}" failed. No retries permitted until 2026-09-09 20:54:20.316617675 +0000 UTC m=+103.430303512 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c6e98f74-7546-4383-89e2-e8ea2c8197e3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-l852l" (UID: "c6e98f74-7546-4383-89e2-e8ea2c8197e3") : configmap "kube-rbac-proxy" not found
Sep 09 20:54:16.316643 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:54:16.316643    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6e98f74-7546-4383-89e2-e8ea2c8197e3-insights-runtime-extractor-tls podName:c6e98f74-7546-4383-89e2-e8ea2c8197e3 nodeName:}" failed. No retries permitted until 2026-09-09 20:54:20.316637783 +0000 UTC m=+103.430323620 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c6e98f74-7546-4383-89e2-e8ea2c8197e3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-l852l" (UID: "c6e98f74-7546-4383-89e2-e8ea2c8197e3") : secret "insights-runtime-extractor-tls" not found
Sep 09 20:54:17.123918 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:17.123891    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0989b0f6-8115-4152-95ae-1f52fdb2012a-registry-tls\") pod \"image-registry-cb585848c-289xj\" (UID: \"0989b0f6-8115-4152-95ae-1f52fdb2012a\") " pod="openshift-image-registry/image-registry-cb585848c-289xj"
Sep 09 20:54:17.124277 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:17.123949    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f69e7fb4-fa7d-444e-9bfa-d173783106c2-metrics-tls\") pod \"dns-default-968c2\" (UID: \"f69e7fb4-fa7d-444e-9bfa-d173783106c2\") " pod="openshift-dns/dns-default-968c2"
Sep 09 20:54:17.124277 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:54:17.124047    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 20:54:17.124277 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:54:17.124051    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 20:54:17.124277 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:54:17.124071    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-cb585848c-289xj: secret "image-registry-tls" not found
Sep 09 20:54:17.124277 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:54:17.124097    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f69e7fb4-fa7d-444e-9bfa-d173783106c2-metrics-tls podName:f69e7fb4-fa7d-444e-9bfa-d173783106c2 nodeName:}" failed. No retries permitted until 2026-09-09 20:55:21.124084422 +0000 UTC m=+164.237770259 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f69e7fb4-fa7d-444e-9bfa-d173783106c2-metrics-tls") pod "dns-default-968c2" (UID: "f69e7fb4-fa7d-444e-9bfa-d173783106c2") : secret "dns-default-metrics-tls" not found
Sep 09 20:54:17.124277 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:54:17.124115    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0989b0f6-8115-4152-95ae-1f52fdb2012a-registry-tls podName:0989b0f6-8115-4152-95ae-1f52fdb2012a nodeName:}" failed. No retries permitted until 2026-09-09 20:55:21.124102667 +0000 UTC m=+164.237788504 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0989b0f6-8115-4152-95ae-1f52fdb2012a-registry-tls") pod "image-registry-cb585848c-289xj" (UID: "0989b0f6-8115-4152-95ae-1f52fdb2012a") : secret "image-registry-tls" not found
Sep 09 20:54:18.861584 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:18.861548    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-l9q9f"]
Sep 09 20:54:18.876239 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:18.876220    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-l9q9f"
Sep 09 20:54:18.876879 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:18.876857    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-l9q9f"]
Sep 09 20:54:18.882747 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:18.882722    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Sep 09 20:54:18.882875 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:18.882778    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Sep 09 20:54:18.882875 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:18.882809    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Sep 09 20:54:18.882875 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:18.882822    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-rbwf6\""
Sep 09 20:54:18.883031 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:18.882815    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Sep 09 20:54:18.937844 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:18.937817    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8dd00baa-1d00-498a-a56a-bc7f90c53317-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-l9q9f\" (UID: \"8dd00baa-1d00-498a-a56a-bc7f90c53317\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-l9q9f"
Sep 09 20:54:18.937958 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:18.937877    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8dd00baa-1d00-498a-a56a-bc7f90c53317-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-l9q9f\" (UID: \"8dd00baa-1d00-498a-a56a-bc7f90c53317\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-l9q9f"
Sep 09 20:54:18.937958 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:18.937919    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xq5js\" (UniqueName: \"kubernetes.io/projected/8dd00baa-1d00-498a-a56a-bc7f90c53317-kube-api-access-xq5js\") pod \"kube-storage-version-migrator-operator-8d7dd4675-l9q9f\" (UID: \"8dd00baa-1d00-498a-a56a-bc7f90c53317\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-l9q9f"
Sep 09 20:54:19.038355 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:19.038326    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8dd00baa-1d00-498a-a56a-bc7f90c53317-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-l9q9f\" (UID: \"8dd00baa-1d00-498a-a56a-bc7f90c53317\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-l9q9f"
Sep 09 20:54:19.038522 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:19.038359    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xq5js\" (UniqueName: \"kubernetes.io/projected/8dd00baa-1d00-498a-a56a-bc7f90c53317-kube-api-access-xq5js\") pod \"kube-storage-version-migrator-operator-8d7dd4675-l9q9f\" (UID: \"8dd00baa-1d00-498a-a56a-bc7f90c53317\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-l9q9f"
Sep 09 20:54:19.038522 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:19.038462    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8dd00baa-1d00-498a-a56a-bc7f90c53317-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-l9q9f\" (UID: \"8dd00baa-1d00-498a-a56a-bc7f90c53317\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-l9q9f"
Sep 09 20:54:19.038890 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:19.038862    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8dd00baa-1d00-498a-a56a-bc7f90c53317-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-l9q9f\" (UID: \"8dd00baa-1d00-498a-a56a-bc7f90c53317\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-l9q9f"
Sep 09 20:54:19.040725 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:19.040709    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8dd00baa-1d00-498a-a56a-bc7f90c53317-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-l9q9f\" (UID: \"8dd00baa-1d00-498a-a56a-bc7f90c53317\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-l9q9f"
Sep 09 20:54:19.048526 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:19.048505    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xq5js\" (UniqueName: \"kubernetes.io/projected/8dd00baa-1d00-498a-a56a-bc7f90c53317-kube-api-access-xq5js\") pod \"kube-storage-version-migrator-operator-8d7dd4675-l9q9f\" (UID: \"8dd00baa-1d00-498a-a56a-bc7f90c53317\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-l9q9f"
Sep 09 20:54:19.185999 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:19.185919    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-l9q9f"
Sep 09 20:54:19.477004 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:19.476929    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-l9q9f"]
Sep 09 20:54:19.491562 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:54:19.491536    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8dd00baa_1d00_498a_a56a_bc7f90c53317.slice/crio-d1aadbf82f9b58efebaa93763c573a584dd438cf419229f87b8745e3c11a82ea WatchSource:0}: Error finding container d1aadbf82f9b58efebaa93763c573a584dd438cf419229f87b8745e3c11a82ea: Status 404 returned error can't find the container with id d1aadbf82f9b58efebaa93763c573a584dd438cf419229f87b8745e3c11a82ea
Sep 09 20:54:19.750878 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:19.750803    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-t8tk8"
Sep 09 20:54:19.821065 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:19.821031    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-l9q9f" event={"ID":"8dd00baa-1d00-498a-a56a-bc7f90c53317","Type":"ContainerStarted","Data":"d1aadbf82f9b58efebaa93763c573a584dd438cf419229f87b8745e3c11a82ea"}
Sep 09 20:54:20.350209 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:20.350172    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c6e98f74-7546-4383-89e2-e8ea2c8197e3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l852l\" (UID: \"c6e98f74-7546-4383-89e2-e8ea2c8197e3\") " pod="openshift-insights/insights-runtime-extractor-l852l"
Sep 09 20:54:20.350669 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:20.350290    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c6e98f74-7546-4383-89e2-e8ea2c8197e3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l852l\" (UID: \"c6e98f74-7546-4383-89e2-e8ea2c8197e3\") " pod="openshift-insights/insights-runtime-extractor-l852l"
Sep 09 20:54:20.350669 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:54:20.350315    2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 20:54:20.350669 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:54:20.350413    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c6e98f74-7546-4383-89e2-e8ea2c8197e3-kube-rbac-proxy-cm podName:c6e98f74-7546-4383-89e2-e8ea2c8197e3 nodeName:}" failed. No retries permitted until 2026-09-09 20:54:28.350391955 +0000 UTC m=+111.464077797 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c6e98f74-7546-4383-89e2-e8ea2c8197e3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-l852l" (UID: "c6e98f74-7546-4383-89e2-e8ea2c8197e3") : configmap "kube-rbac-proxy" not found
Sep 09 20:54:20.350669 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:54:20.350445    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 20:54:20.350669 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:54:20.350524    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6e98f74-7546-4383-89e2-e8ea2c8197e3-insights-runtime-extractor-tls podName:c6e98f74-7546-4383-89e2-e8ea2c8197e3 nodeName:}" failed. No retries permitted until 2026-09-09 20:54:28.35050586 +0000 UTC m=+111.464191703 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c6e98f74-7546-4383-89e2-e8ea2c8197e3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-l852l" (UID: "c6e98f74-7546-4383-89e2-e8ea2c8197e3") : secret "insights-runtime-extractor-tls" not found
Sep 09 20:54:21.161193 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:21.161154    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:21.258520 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:21.258491    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:21.827070 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:21.827037    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-l9q9f" event={"ID":"8dd00baa-1d00-498a-a56a-bc7f90c53317","Type":"ContainerStarted","Data":"eeb87b29c07e2b1fe6cd0b2a7a6c6dc07dcccb13c422b393c48d18c476b78d45"}
Sep 09 20:54:22.806430 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:22.806390    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:22.899942 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:22.899905    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:23.262579 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:23.262546    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:23.362597 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:23.362562    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:23.546484 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:23.546390    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:23.643275 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:23.643249    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:23.806634 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:23.806544    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-l9q9f" podStartSLOduration=4.044104211 podStartE2EDuration="5.806526295s" totalImagesPullingTime="1.762422084s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:54:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:54:21.848436949 +0000 UTC m=+104.962122806" watchObservedRunningTime="2026-09-09 20:54:23.806526295 +0000 UTC m=+106.920212154"
Sep 09 20:54:23.807071 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:23.807057    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-5567m"]
Sep 09 20:54:23.809124 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:23.809107    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-5567m"
Sep 09 20:54:23.816602 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:23.816571    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-sbw4q\""
Sep 09 20:54:23.816602 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:23.816591    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 09 20:54:23.817443 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:23.817429    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 09 20:54:23.839644 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:23.839622    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-5567m"]
Sep 09 20:54:23.868226 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:23.868189    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:23.877213 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:23.877191    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptt7j\" (UniqueName: \"kubernetes.io/projected/b24b3cc1-dff7-4f2a-a5d4-5c7d65e6c8c3-kube-api-access-ptt7j\") pod \"migrator-fdbb8fdff-5567m\" (UID: \"b24b3cc1-dff7-4f2a-a5d4-5c7d65e6c8c3\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-5567m"
Sep 09 20:54:23.945163 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:23.945132    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:23.978149 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:23.978125    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptt7j\" (UniqueName: \"kubernetes.io/projected/b24b3cc1-dff7-4f2a-a5d4-5c7d65e6c8c3-kube-api-access-ptt7j\") pod \"migrator-fdbb8fdff-5567m\" (UID: \"b24b3cc1-dff7-4f2a-a5d4-5c7d65e6c8c3\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-5567m"
Sep 09 20:54:23.992652 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:23.992632    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptt7j\" (UniqueName: \"kubernetes.io/projected/b24b3cc1-dff7-4f2a-a5d4-5c7d65e6c8c3-kube-api-access-ptt7j\") pod \"migrator-fdbb8fdff-5567m\" (UID: \"b24b3cc1-dff7-4f2a-a5d4-5c7d65e6c8c3\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-5567m"
Sep 09 20:54:24.117276 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:24.117237    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-5567m"
Sep 09 20:54:24.395914 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:24.395831    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-5567m"]
Sep 09 20:54:24.398254 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:54:24.398229    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb24b3cc1_dff7_4f2a_a5d4_5c7d65e6c8c3.slice/crio-7685acea72bb3870ff82990e023a6fe4e86aab96bd9ee537e6fac1053e6930ff WatchSource:0}: Error finding container 7685acea72bb3870ff82990e023a6fe4e86aab96bd9ee537e6fac1053e6930ff: Status 404 returned error can't find the container with id 7685acea72bb3870ff82990e023a6fe4e86aab96bd9ee537e6fac1053e6930ff
Sep 09 20:54:24.835728 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:24.835644    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-5567m" event={"ID":"b24b3cc1-dff7-4f2a-a5d4-5c7d65e6c8c3","Type":"ContainerStarted","Data":"7685acea72bb3870ff82990e023a6fe4e86aab96bd9ee537e6fac1053e6930ff"}
Sep 09 20:54:25.201642 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:25.201609    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:25.803548 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:25.803444    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:25.840093 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:25.840063    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-5567m" event={"ID":"b24b3cc1-dff7-4f2a-a5d4-5c7d65e6c8c3","Type":"ContainerStarted","Data":"6045345c66989a0564f24e2033d914ba2079b4a226987fed0c7110dc72f8d4f1"}
Sep 09 20:54:25.840211 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:25.840115    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-5567m" event={"ID":"b24b3cc1-dff7-4f2a-a5d4-5c7d65e6c8c3","Type":"ContainerStarted","Data":"2820154343bbda79cb308c98b9120cc419b8d7ab346d357b1843eb97e16b438c"}
Sep 09 20:54:25.866513 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:25.866470    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-5567m" podStartSLOduration=1.740965545 podStartE2EDuration="2.866456293s" totalImagesPullingTime="1.125490748s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:54:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:54:25.865901333 +0000 UTC m=+108.979587194" watchObservedRunningTime="2026-09-09 20:54:25.866456293 +0000 UTC m=+108.980142149"
Sep 09 20:54:26.732635 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:26.732600    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:28.412269 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:28.412235    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c6e98f74-7546-4383-89e2-e8ea2c8197e3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l852l\" (UID: \"c6e98f74-7546-4383-89e2-e8ea2c8197e3\") " pod="openshift-insights/insights-runtime-extractor-l852l"
Sep 09 20:54:28.412749 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:28.412301    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c6e98f74-7546-4383-89e2-e8ea2c8197e3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l852l\" (UID: \"c6e98f74-7546-4383-89e2-e8ea2c8197e3\") " pod="openshift-insights/insights-runtime-extractor-l852l"
Sep 09 20:54:28.412749 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:54:28.412419    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 20:54:28.412749 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:54:28.412500    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6e98f74-7546-4383-89e2-e8ea2c8197e3-insights-runtime-extractor-tls podName:c6e98f74-7546-4383-89e2-e8ea2c8197e3 nodeName:}" failed. No retries permitted until 2026-09-09 20:54:44.41248044 +0000 UTC m=+127.526166297 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c6e98f74-7546-4383-89e2-e8ea2c8197e3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-l852l" (UID: "c6e98f74-7546-4383-89e2-e8ea2c8197e3") : secret "insights-runtime-extractor-tls" not found
Sep 09 20:54:28.412909 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:28.412760    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c6e98f74-7546-4383-89e2-e8ea2c8197e3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l852l\" (UID: \"c6e98f74-7546-4383-89e2-e8ea2c8197e3\") " pod="openshift-insights/insights-runtime-extractor-l852l"
Sep 09 20:54:29.200489 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:29.200455    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:29.306036 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:29.306000    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:29.829132 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:29.829096    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:29.936050 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:29.936011    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:30.060073 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:30.060040    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:30.170181 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:30.170147    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:30.326158 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:30.326127    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:30.700712 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:30.700673    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:31.494089 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:31.494051    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:31.881591 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:31.881565    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:32.097766 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:32.097729    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:32.290848 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:32.290769    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:32.682939 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:32.682906    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:33.078888 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:33.078808    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:33.153838 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:33.153806    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:33.684186 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:33.684155    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:33.773134 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:33.773100    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:34.098231 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:34.098195    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:34.486014 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:34.485925    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:34.686135 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:34.686101    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:35.088726 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:35.088691    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:35.489738 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:35.489649    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:35.597847 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:35.597821    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:36.702623 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:36.702591    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:36.767611 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:36.767580    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:36.886422 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:36.886395    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:37.090791 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:37.090764    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:37.563441 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:37.563335    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:37.688570 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:37.688537    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:38.279241 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:38.279201    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:38.480461 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:38.480427    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:38.905340 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:38.905309    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:39.096537 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:39.096506    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:39.485085 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:39.485042    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:39.881273 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:39.881222    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:40.076611 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:40.076579    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:43.142376 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:43.142327    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:43.696839 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:43.696806    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:43.797734 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:43.797702    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:43.898839 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:43.898800    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:44.029311 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:44.029226    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:44.177569 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:44.177537    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:44.244074 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:44.244044    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:44.365034 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:44.365005    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:44.428579 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:44.428546    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c6e98f74-7546-4383-89e2-e8ea2c8197e3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l852l\" (UID: \"c6e98f74-7546-4383-89e2-e8ea2c8197e3\") " pod="openshift-insights/insights-runtime-extractor-l852l"
Sep 09 20:54:44.431050 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:44.431025    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c6e98f74-7546-4383-89e2-e8ea2c8197e3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l852l\" (UID: \"c6e98f74-7546-4383-89e2-e8ea2c8197e3\") " pod="openshift-insights/insights-runtime-extractor-l852l"
Sep 09 20:54:44.433100 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:44.433078    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:44.684530 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:44.684454    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-ksvhl\""
Sep 09 20:54:44.692530 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:44.692512    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-l852l"
Sep 09 20:54:44.773248 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:44.773216    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:44.960729 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:44.960699    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-l852l"]
Sep 09 20:54:44.963238 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:54:44.963212    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6e98f74_7546_4383_89e2_e8ea2c8197e3.slice/crio-de7cd93ec7c9ad5f0a777b3761eecbcc9f1066b93e4d7b33467c5440b25995d1 WatchSource:0}: Error finding container de7cd93ec7c9ad5f0a777b3761eecbcc9f1066b93e4d7b33467c5440b25995d1: Status 404 returned error can't find the container with id de7cd93ec7c9ad5f0a777b3761eecbcc9f1066b93e4d7b33467c5440b25995d1
Sep 09 20:54:45.537379 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:45.537344    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:45.637760 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:45.637726    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:45.779881 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:45.779839    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:45.897614 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:45.897587    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-l852l" event={"ID":"c6e98f74-7546-4383-89e2-e8ea2c8197e3","Type":"ContainerStarted","Data":"63aad805dd854267ac63726e3192203469fe1ff8c1f2e2b0a6d62a548c5a0f4c"}
Sep 09 20:54:45.897705 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:45.897643    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-l852l" event={"ID":"c6e98f74-7546-4383-89e2-e8ea2c8197e3","Type":"ContainerStarted","Data":"0af0c7ef9124d28da1840c231a3c8c2820176d436779cba0241fcbdc82746183"}
Sep 09 20:54:45.897705 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:45.897664    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-l852l" event={"ID":"c6e98f74-7546-4383-89e2-e8ea2c8197e3","Type":"ContainerStarted","Data":"de7cd93ec7c9ad5f0a777b3761eecbcc9f1066b93e4d7b33467c5440b25995d1"}
Sep 09 20:54:46.301263 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:46.301232    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:47.002180 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:47.002152    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:47.249505 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:47.249467    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b815b2b-983b-4afe-900d-13dd791e4778-metrics-certs\") pod \"network-metrics-daemon-v74xn\" (UID: \"5b815b2b-983b-4afe-900d-13dd791e4778\") " pod="openshift-multus/network-metrics-daemon-v74xn"
Sep 09 20:54:47.252247 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:47.252188    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b815b2b-983b-4afe-900d-13dd791e4778-metrics-certs\") pod \"network-metrics-daemon-v74xn\" (UID: \"5b815b2b-983b-4afe-900d-13dd791e4778\") " pod="openshift-multus/network-metrics-daemon-v74xn"
Sep 09 20:54:47.418472 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:47.418443    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-j2slq\""
Sep 09 20:54:47.423605 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:47.423591    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v74xn"
Sep 09 20:54:47.632155 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:47.632122    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:47.690907 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:47.690880    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-v74xn"]
Sep 09 20:54:47.693885 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:54:47.693863    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b815b2b_983b_4afe_900d_13dd791e4778.slice/crio-e4cc6cca6504cd6d41cb9b84631a725c55356c481b5f8ab55259b41b37f7e48b WatchSource:0}: Error finding container e4cc6cca6504cd6d41cb9b84631a725c55356c481b5f8ab55259b41b37f7e48b: Status 404 returned error can't find the container with id e4cc6cca6504cd6d41cb9b84631a725c55356c481b5f8ab55259b41b37f7e48b
Sep 09 20:54:47.904570 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:47.904476    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-v74xn" event={"ID":"5b815b2b-983b-4afe-900d-13dd791e4778","Type":"ContainerStarted","Data":"e4cc6cca6504cd6d41cb9b84631a725c55356c481b5f8ab55259b41b37f7e48b"}
Sep 09 20:54:47.906531 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:47.906509    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-l852l" event={"ID":"c6e98f74-7546-4383-89e2-e8ea2c8197e3","Type":"ContainerStarted","Data":"58e02607d05a9a08bdc381970086b2a20c6d7bda4f416ae56b7c14bc8e620b7a"}
Sep 09 20:54:47.927866 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:47.927827    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-l852l" podStartSLOduration=33.699207705 podStartE2EDuration="35.927812609s" totalImagesPullingTime="2.228604904s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:54:12 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:54:47.926727615 +0000 UTC m=+131.040413475" watchObservedRunningTime="2026-09-09 20:54:47.927812609 +0000 UTC m=+131.041498458"
Sep 09 20:54:48.737453 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:48.737422    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6c679bc786-7pg86"]
Sep 09 20:54:48.739642 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:48.739625    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6c679bc786-7pg86"
Sep 09 20:54:48.743554 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:48.743536    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 09 20:54:48.745016 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:48.744997    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 09 20:54:48.745508 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:48.745489    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 09 20:54:48.745590 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:48.745514    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 09 20:54:48.771093 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:48.771055    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6c679bc786-7pg86"]
Sep 09 20:54:48.863206 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:48.863125    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1035f3ba-839d-4a8c-85bd-f386b821b53b-tmp\") pod \"klusterlet-addon-workmgr-6c679bc786-7pg86\" (UID: \"1035f3ba-839d-4a8c-85bd-f386b821b53b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6c679bc786-7pg86"
Sep 09 20:54:48.863206 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:48.863163    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/1035f3ba-839d-4a8c-85bd-f386b821b53b-klusterlet-config\") pod \"klusterlet-addon-workmgr-6c679bc786-7pg86\" (UID: \"1035f3ba-839d-4a8c-85bd-f386b821b53b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6c679bc786-7pg86"
Sep 09 20:54:48.863411 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:48.863212    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fthfx\" (UniqueName: \"kubernetes.io/projected/1035f3ba-839d-4a8c-85bd-f386b821b53b-kube-api-access-fthfx\") pod \"klusterlet-addon-workmgr-6c679bc786-7pg86\" (UID: \"1035f3ba-839d-4a8c-85bd-f386b821b53b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6c679bc786-7pg86"
Sep 09 20:54:48.878884 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:48.877832    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-w6wnq"]
Sep 09 20:54:48.880532 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:48.879940    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-685456655f-w6wnq"
Sep 09 20:54:48.889692 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:48.889644    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-pjwvh\""
Sep 09 20:54:48.902892 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:48.902737    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-dvfw5"]
Sep 09 20:54:48.904504 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:48.904489    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-dvfw5"
Sep 09 20:54:48.911053 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:48.911032    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-v74xn" event={"ID":"5b815b2b-983b-4afe-900d-13dd791e4778","Type":"ContainerStarted","Data":"d0ea383da6ece4e91f57fc74e09701cdf3e9eea442328bed2e5858cf398281ed"}
Sep 09 20:54:48.911152 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:48.911079    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-v74xn" event={"ID":"5b815b2b-983b-4afe-900d-13dd791e4778","Type":"ContainerStarted","Data":"de063540c207c1c5cd003b6ba6c624f85f8fc13d79c06e836ffedfb36249afcd"}
Sep 09 20:54:48.913964 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:48.913949    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-5xrmz\""
Sep 09 20:54:48.917733 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:48.917720    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 09 20:54:48.940684 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:48.940655    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:48.962064 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:48.962046    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-dvfw5"]
Sep 09 20:54:48.963497 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:48.963476    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fthfx\" (UniqueName: \"kubernetes.io/projected/1035f3ba-839d-4a8c-85bd-f386b821b53b-kube-api-access-fthfx\") pod \"klusterlet-addon-workmgr-6c679bc786-7pg86\" (UID: \"1035f3ba-839d-4a8c-85bd-f386b821b53b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6c679bc786-7pg86"
Sep 09 20:54:48.963579 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:48.963527    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1035f3ba-839d-4a8c-85bd-f386b821b53b-tmp\") pod \"klusterlet-addon-workmgr-6c679bc786-7pg86\" (UID: \"1035f3ba-839d-4a8c-85bd-f386b821b53b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6c679bc786-7pg86"
Sep 09 20:54:48.963579 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:48.963551    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/1035f3ba-839d-4a8c-85bd-f386b821b53b-klusterlet-config\") pod \"klusterlet-addon-workmgr-6c679bc786-7pg86\" (UID: \"1035f3ba-839d-4a8c-85bd-f386b821b53b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6c679bc786-7pg86"
Sep 09 20:54:48.963979 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:48.963962    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1035f3ba-839d-4a8c-85bd-f386b821b53b-tmp\") pod \"klusterlet-addon-workmgr-6c679bc786-7pg86\" (UID: \"1035f3ba-839d-4a8c-85bd-f386b821b53b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6c679bc786-7pg86"
Sep 09 20:54:48.966245 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:48.966220    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/1035f3ba-839d-4a8c-85bd-f386b821b53b-klusterlet-config\") pod \"klusterlet-addon-workmgr-6c679bc786-7pg86\" (UID: \"1035f3ba-839d-4a8c-85bd-f386b821b53b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6c679bc786-7pg86"
Sep 09 20:54:48.971135 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:48.971112    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-85d98c998f-wqqtc"]
Sep 09 20:54:48.973142 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:48.973126    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-85d98c998f-wqqtc"
Sep 09 20:54:48.976100 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:48.976083    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 09 20:54:48.976978 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:48.976962    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-ltjxq\""
Sep 09 20:54:48.977478 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:48.977464    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 09 20:54:48.982344 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:48.982324    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fthfx\" (UniqueName: \"kubernetes.io/projected/1035f3ba-839d-4a8c-85bd-f386b821b53b-kube-api-access-fthfx\") pod \"klusterlet-addon-workmgr-6c679bc786-7pg86\" (UID: \"1035f3ba-839d-4a8c-85bd-f386b821b53b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6c679bc786-7pg86"
Sep 09 20:54:49.005231 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:49.005196    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-85d98c998f-wqqtc"]
Sep 09 20:54:49.006419 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:49.006400    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-w6wnq"]
Sep 09 20:54:49.027892 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:49.027849    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-v74xn" podStartSLOduration=130.126048637 podStartE2EDuration="2m11.027836514s" totalImagesPullingTime="901.787877ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:52:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:54:49.027493119 +0000 UTC m=+132.141178988" watchObservedRunningTime="2026-09-09 20:54:49.027836514 +0000 UTC m=+132.141522373"
Sep 09 20:54:49.050707 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:49.050689    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6c679bc786-7pg86"
Sep 09 20:54:49.064067 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:49.064043    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6sp4\" (UniqueName: \"kubernetes.io/projected/17c3256b-b4ea-4ec0-b9f8-9d753beb353e-kube-api-access-c6sp4\") pod \"network-check-source-685456655f-w6wnq\" (UID: \"17c3256b-b4ea-4ec0-b9f8-9d753beb353e\") " pod="openshift-network-diagnostics/network-check-source-685456655f-w6wnq"
Sep 09 20:54:49.064312 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:49.064082    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/ae5f845e-636d-4484-9c71-bae4cc3c2d2a-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-dvfw5\" (UID: \"ae5f845e-636d-4484-9c71-bae4cc3c2d2a\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-dvfw5"
Sep 09 20:54:49.064558 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:49.064536    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:49.165020 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:49.164948    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/ae5f845e-636d-4484-9c71-bae4cc3c2d2a-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-dvfw5\" (UID: \"ae5f845e-636d-4484-9c71-bae4cc3c2d2a\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-dvfw5"
Sep 09 20:54:49.165020 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:49.164990    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zksrk\" (UniqueName: \"kubernetes.io/projected/ca8ff022-fc16-478e-8dad-d819bf9331f8-kube-api-access-zksrk\") pod \"downloads-85d98c998f-wqqtc\" (UID: \"ca8ff022-fc16-478e-8dad-d819bf9331f8\") " pod="openshift-console/downloads-85d98c998f-wqqtc"
Sep 09 20:54:49.165173 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:49.165116    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6sp4\" (UniqueName: \"kubernetes.io/projected/17c3256b-b4ea-4ec0-b9f8-9d753beb353e-kube-api-access-c6sp4\") pod \"network-check-source-685456655f-w6wnq\" (UID: \"17c3256b-b4ea-4ec0-b9f8-9d753beb353e\") " pod="openshift-network-diagnostics/network-check-source-685456655f-w6wnq"
Sep 09 20:54:49.167411 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:49.167392    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/ae5f845e-636d-4484-9c71-bae4cc3c2d2a-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-dvfw5\" (UID: \"ae5f845e-636d-4484-9c71-bae4cc3c2d2a\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-dvfw5"
Sep 09 20:54:49.184414 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:49.184386    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6sp4\" (UniqueName: \"kubernetes.io/projected/17c3256b-b4ea-4ec0-b9f8-9d753beb353e-kube-api-access-c6sp4\") pod \"network-check-source-685456655f-w6wnq\" (UID: \"17c3256b-b4ea-4ec0-b9f8-9d753beb353e\") " pod="openshift-network-diagnostics/network-check-source-685456655f-w6wnq"
Sep 09 20:54:49.201183 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:49.201162    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-685456655f-w6wnq"
Sep 09 20:54:49.213212 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:49.213191    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-dvfw5"
Sep 09 20:54:49.226527 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:49.226500    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:49.265946 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:49.265915    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zksrk\" (UniqueName: \"kubernetes.io/projected/ca8ff022-fc16-478e-8dad-d819bf9331f8-kube-api-access-zksrk\") pod \"downloads-85d98c998f-wqqtc\" (UID: \"ca8ff022-fc16-478e-8dad-d819bf9331f8\") " pod="openshift-console/downloads-85d98c998f-wqqtc"
Sep 09 20:54:49.279613 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:49.279564    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zksrk\" (UniqueName: \"kubernetes.io/projected/ca8ff022-fc16-478e-8dad-d819bf9331f8-kube-api-access-zksrk\") pod \"downloads-85d98c998f-wqqtc\" (UID: \"ca8ff022-fc16-478e-8dad-d819bf9331f8\") " pod="openshift-console/downloads-85d98c998f-wqqtc"
Sep 09 20:54:49.281353 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:49.281309    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-85d98c998f-wqqtc"
Sep 09 20:54:49.345184 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:49.344962    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6c679bc786-7pg86"]
Sep 09 20:54:49.347443 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:54:49.347420    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1035f3ba_839d_4a8c_85bd_f386b821b53b.slice/crio-a2dfbdc2db6ed223e2bfadb8ded6f884fb2bcff5a4089fd4cb9b4394ad6ae51b WatchSource:0}: Error finding container a2dfbdc2db6ed223e2bfadb8ded6f884fb2bcff5a4089fd4cb9b4394ad6ae51b: Status 404 returned error can't find the container with id a2dfbdc2db6ed223e2bfadb8ded6f884fb2bcff5a4089fd4cb9b4394ad6ae51b
Sep 09 20:54:49.350504 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:49.350471    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:49.491788 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:49.491717    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-w6wnq"]
Sep 09 20:54:49.497872 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:54:49.497841    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17c3256b_b4ea_4ec0_b9f8_9d753beb353e.slice/crio-8cdddd9ff53d3acdfd48d02ac031ec9411a75403b74f6b46c8256afb99deac73 WatchSource:0}: Error finding container 8cdddd9ff53d3acdfd48d02ac031ec9411a75403b74f6b46c8256afb99deac73: Status 404 returned error can't find the container with id 8cdddd9ff53d3acdfd48d02ac031ec9411a75403b74f6b46c8256afb99deac73
Sep 09 20:54:49.506790 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:49.506768    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-dvfw5"]
Sep 09 20:54:49.562298 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:49.562273    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-85d98c998f-wqqtc"]
Sep 09 20:54:49.565535 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:54:49.565502    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca8ff022_fc16_478e_8dad_d819bf9331f8.slice/crio-83fad91ffd2158489599fe049bc68e73ff928daf6436bcd9cc19be5ef744cc97 WatchSource:0}: Error finding container 83fad91ffd2158489599fe049bc68e73ff928daf6436bcd9cc19be5ef744cc97: Status 404 returned error can't find the container with id 83fad91ffd2158489599fe049bc68e73ff928daf6436bcd9cc19be5ef744cc97
Sep 09 20:54:49.916340 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:49.916304    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-dvfw5" event={"ID":"ae5f845e-636d-4484-9c71-bae4cc3c2d2a","Type":"ContainerStarted","Data":"f26aabd2895d24c080ad6e37a3364e53a15f5136cc0f29894ff4184653d6da53"}
Sep 09 20:54:49.917794 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:49.917764    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-685456655f-w6wnq" event={"ID":"17c3256b-b4ea-4ec0-b9f8-9d753beb353e","Type":"ContainerStarted","Data":"5d845668e3ed0555ab01efa7fbe5ec581dea8ce54e26d945010a022458dce042"}
Sep 09 20:54:49.917923 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:49.917815    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-685456655f-w6wnq" event={"ID":"17c3256b-b4ea-4ec0-b9f8-9d753beb353e","Type":"ContainerStarted","Data":"8cdddd9ff53d3acdfd48d02ac031ec9411a75403b74f6b46c8256afb99deac73"}
Sep 09 20:54:49.918988 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:49.918964    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6c679bc786-7pg86" event={"ID":"1035f3ba-839d-4a8c-85bd-f386b821b53b","Type":"ContainerStarted","Data":"a2dfbdc2db6ed223e2bfadb8ded6f884fb2bcff5a4089fd4cb9b4394ad6ae51b"}
Sep 09 20:54:49.920163 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:49.920145    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-85d98c998f-wqqtc" event={"ID":"ca8ff022-fc16-478e-8dad-d819bf9331f8","Type":"ContainerStarted","Data":"83fad91ffd2158489599fe049bc68e73ff928daf6436bcd9cc19be5ef744cc97"}
Sep 09 20:54:49.939446 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:49.939407    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-685456655f-w6wnq" podStartSLOduration=1.939393499 podStartE2EDuration="1.939393499s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:54:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:54:49.938401918 +0000 UTC m=+133.052087777" watchObservedRunningTime="2026-09-09 20:54:49.939393499 +0000 UTC m=+133.053079694"
Sep 09 20:54:51.614706 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:51.614618    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:51.713885 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:51.713848    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:51.933525 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:51.933435    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-dvfw5" event={"ID":"ae5f845e-636d-4484-9c71-bae4cc3c2d2a","Type":"ContainerStarted","Data":"2589da257fd7dd8c9f32881ee068b7d4584e9db1f3ed63aeb4fec034a7f67afd"}
Sep 09 20:54:51.933826 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:51.933807    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-dvfw5"
Sep 09 20:54:51.940276 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:51.940251    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-dvfw5"
Sep 09 20:54:51.955441 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:51.954956    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-dvfw5" podStartSLOduration=2.289198883 podStartE2EDuration="3.954936851s" totalImagesPullingTime="1.665737968s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:54:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:54:51.953600153 +0000 UTC m=+135.067286013" watchObservedRunningTime="2026-09-09 20:54:51.954936851 +0000 UTC m=+135.068622705"
Sep 09 20:54:52.337723 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:52.337551    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:52.424054 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:52.424021    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:52.668043 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:52.667963    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:53.520035 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:53.519993    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:53.591219 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:53.591134    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:54.245091 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:54.245020    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7f49d694d7-zg2p7"]
Sep 09 20:54:54.248376 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:54.248338    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f49d694d7-zg2p7"
Sep 09 20:54:54.253803 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:54.253783    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-z9qdk\""
Sep 09 20:54:54.254092 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:54.254074    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 09 20:54:54.255286 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:54.255267    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 09 20:54:54.255424 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:54.255285    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 09 20:54:54.255685 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:54.255624    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 09 20:54:54.256481 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:54.256465    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 09 20:54:54.263387 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:54.263342    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7f49d694d7-zg2p7"]
Sep 09 20:54:54.306265 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:54.306228    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d9bef8f5-e612-4504-a259-c2550b05cea2-oauth-serving-cert\") pod \"console-7f49d694d7-zg2p7\" (UID: \"d9bef8f5-e612-4504-a259-c2550b05cea2\") " pod="openshift-console/console-7f49d694d7-zg2p7"
Sep 09 20:54:54.306437 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:54.306274    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d9bef8f5-e612-4504-a259-c2550b05cea2-console-config\") pod \"console-7f49d694d7-zg2p7\" (UID: \"d9bef8f5-e612-4504-a259-c2550b05cea2\") " pod="openshift-console/console-7f49d694d7-zg2p7"
Sep 09 20:54:54.306437 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:54.306297    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d9bef8f5-e612-4504-a259-c2550b05cea2-service-ca\") pod \"console-7f49d694d7-zg2p7\" (UID: \"d9bef8f5-e612-4504-a259-c2550b05cea2\") " pod="openshift-console/console-7f49d694d7-zg2p7"
Sep 09 20:54:54.306437 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:54.306401    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbj9p\" (UniqueName: \"kubernetes.io/projected/d9bef8f5-e612-4504-a259-c2550b05cea2-kube-api-access-qbj9p\") pod \"console-7f49d694d7-zg2p7\" (UID: \"d9bef8f5-e612-4504-a259-c2550b05cea2\") " pod="openshift-console/console-7f49d694d7-zg2p7"
Sep 09 20:54:54.306599 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:54.306445    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d9bef8f5-e612-4504-a259-c2550b05cea2-console-oauth-config\") pod \"console-7f49d694d7-zg2p7\" (UID: \"d9bef8f5-e612-4504-a259-c2550b05cea2\") " pod="openshift-console/console-7f49d694d7-zg2p7"
Sep 09 20:54:54.306599 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:54.306502    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d9bef8f5-e612-4504-a259-c2550b05cea2-console-serving-cert\") pod \"console-7f49d694d7-zg2p7\" (UID: \"d9bef8f5-e612-4504-a259-c2550b05cea2\") " pod="openshift-console/console-7f49d694d7-zg2p7"
Sep 09 20:54:54.406811 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:54.406772    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d9bef8f5-e612-4504-a259-c2550b05cea2-oauth-serving-cert\") pod \"console-7f49d694d7-zg2p7\" (UID: \"d9bef8f5-e612-4504-a259-c2550b05cea2\") " pod="openshift-console/console-7f49d694d7-zg2p7"
Sep 09 20:54:54.406811 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:54.406815    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d9bef8f5-e612-4504-a259-c2550b05cea2-console-config\") pod \"console-7f49d694d7-zg2p7\" (UID: \"d9bef8f5-e612-4504-a259-c2550b05cea2\") " pod="openshift-console/console-7f49d694d7-zg2p7"
Sep 09 20:54:54.407034 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:54.406832    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d9bef8f5-e612-4504-a259-c2550b05cea2-service-ca\") pod \"console-7f49d694d7-zg2p7\" (UID: \"d9bef8f5-e612-4504-a259-c2550b05cea2\") " pod="openshift-console/console-7f49d694d7-zg2p7"
Sep 09 20:54:54.407034 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:54.406867    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbj9p\" (UniqueName: \"kubernetes.io/projected/d9bef8f5-e612-4504-a259-c2550b05cea2-kube-api-access-qbj9p\") pod \"console-7f49d694d7-zg2p7\" (UID: \"d9bef8f5-e612-4504-a259-c2550b05cea2\") " pod="openshift-console/console-7f49d694d7-zg2p7"
Sep 09 20:54:54.407034 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:54.406902    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d9bef8f5-e612-4504-a259-c2550b05cea2-console-oauth-config\") pod \"console-7f49d694d7-zg2p7\" (UID: \"d9bef8f5-e612-4504-a259-c2550b05cea2\") " pod="openshift-console/console-7f49d694d7-zg2p7"
Sep 09 20:54:54.407034 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:54.406933    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d9bef8f5-e612-4504-a259-c2550b05cea2-console-serving-cert\") pod \"console-7f49d694d7-zg2p7\" (UID: \"d9bef8f5-e612-4504-a259-c2550b05cea2\") " pod="openshift-console/console-7f49d694d7-zg2p7"
Sep 09 20:54:54.407648 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:54.407619    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d9bef8f5-e612-4504-a259-c2550b05cea2-oauth-serving-cert\") pod \"console-7f49d694d7-zg2p7\" (UID: \"d9bef8f5-e612-4504-a259-c2550b05cea2\") " pod="openshift-console/console-7f49d694d7-zg2p7"
Sep 09 20:54:54.407748 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:54.407693    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d9bef8f5-e612-4504-a259-c2550b05cea2-console-config\") pod \"console-7f49d694d7-zg2p7\" (UID: \"d9bef8f5-e612-4504-a259-c2550b05cea2\") " pod="openshift-console/console-7f49d694d7-zg2p7"
Sep 09 20:54:54.407823 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:54.407795    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d9bef8f5-e612-4504-a259-c2550b05cea2-service-ca\") pod \"console-7f49d694d7-zg2p7\" (UID: \"d9bef8f5-e612-4504-a259-c2550b05cea2\") " pod="openshift-console/console-7f49d694d7-zg2p7"
Sep 09 20:54:54.409653 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:54.409618    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d9bef8f5-e612-4504-a259-c2550b05cea2-console-serving-cert\") pod \"console-7f49d694d7-zg2p7\" (UID: \"d9bef8f5-e612-4504-a259-c2550b05cea2\") " pod="openshift-console/console-7f49d694d7-zg2p7"
Sep 09 20:54:54.410179 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:54.410151    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d9bef8f5-e612-4504-a259-c2550b05cea2-console-oauth-config\") pod \"console-7f49d694d7-zg2p7\" (UID: \"d9bef8f5-e612-4504-a259-c2550b05cea2\") " pod="openshift-console/console-7f49d694d7-zg2p7"
Sep 09 20:54:54.417654 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:54.417635    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbj9p\" (UniqueName: \"kubernetes.io/projected/d9bef8f5-e612-4504-a259-c2550b05cea2-kube-api-access-qbj9p\") pod \"console-7f49d694d7-zg2p7\" (UID: \"d9bef8f5-e612-4504-a259-c2550b05cea2\") " pod="openshift-console/console-7f49d694d7-zg2p7"
Sep 09 20:54:54.559050 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:54.558957    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f49d694d7-zg2p7"
Sep 09 20:54:54.705549 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:54.705498    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:54.845144 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:54.845115    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7f49d694d7-zg2p7"]
Sep 09 20:54:54.848576 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:54:54.848544    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9bef8f5_e612_4504_a259_c2550b05cea2.slice/crio-cbce41b52036031d896cdf7a96fc67edf95ad7f04d49254c1f3ad6a3107737d1 WatchSource:0}: Error finding container cbce41b52036031d896cdf7a96fc67edf95ad7f04d49254c1f3ad6a3107737d1: Status 404 returned error can't find the container with id cbce41b52036031d896cdf7a96fc67edf95ad7f04d49254c1f3ad6a3107737d1
Sep 09 20:54:54.946453 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:54.946403    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f49d694d7-zg2p7" event={"ID":"d9bef8f5-e612-4504-a259-c2550b05cea2","Type":"ContainerStarted","Data":"cbce41b52036031d896cdf7a96fc67edf95ad7f04d49254c1f3ad6a3107737d1"}
Sep 09 20:54:54.948034 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:54.948004    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6c679bc786-7pg86" event={"ID":"1035f3ba-839d-4a8c-85bd-f386b821b53b","Type":"ContainerStarted","Data":"3dddd2ee39244a67af596a645a172dca2d0d98b698a4ea343f6d21ba9c16a749"}
Sep 09 20:54:54.948289 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:54.948258    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6c679bc786-7pg86"
Sep 09 20:54:54.950141 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:54.950123    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6c679bc786-7pg86"
Sep 09 20:54:55.003829 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:55.003772    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6c679bc786-7pg86" podStartSLOduration=2.378269462 podStartE2EDuration="7.003752362s" totalImagesPullingTime="4.6254829s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:54:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:54:54.973286387 +0000 UTC m=+138.086972247" watchObservedRunningTime="2026-09-09 20:54:55.003752362 +0000 UTC m=+138.117438223"
Sep 09 20:54:55.626806 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:55.626771    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:55.747220 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:55.747189    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:57.513889 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:57.513846    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:57.729976 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:57.729873    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:57.790260 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:57.790153    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-svqd5"]
Sep 09 20:54:57.793744 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:57.793722    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-svqd5"
Sep 09 20:54:57.798064 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:57.797893    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 09 20:54:57.799061 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:57.798382    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 09 20:54:57.799061 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:57.798360    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 09 20:54:57.799175 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:57.799124    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 09 20:54:57.799599 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:57.799458    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 09 20:54:57.799786 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:57.799761    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 09 20:54:57.799786 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:57.799778    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-tqnv2\""
Sep 09 20:54:57.837651 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:57.836259    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/392d4ac7-d590-4225-bef4-c66590b846cb-node-exporter-tls\") pod \"node-exporter-svqd5\" (UID: \"392d4ac7-d590-4225-bef4-c66590b846cb\") " pod="openshift-monitoring/node-exporter-svqd5"
Sep 09 20:54:57.837651 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:57.836305    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/392d4ac7-d590-4225-bef4-c66590b846cb-sys\") pod \"node-exporter-svqd5\" (UID: \"392d4ac7-d590-4225-bef4-c66590b846cb\") " pod="openshift-monitoring/node-exporter-svqd5"
Sep 09 20:54:57.837651 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:57.836329    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/392d4ac7-d590-4225-bef4-c66590b846cb-root\") pod \"node-exporter-svqd5\" (UID: \"392d4ac7-d590-4225-bef4-c66590b846cb\") " pod="openshift-monitoring/node-exporter-svqd5"
Sep 09 20:54:57.837651 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:57.836387    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/392d4ac7-d590-4225-bef4-c66590b846cb-node-exporter-wtmp\") pod \"node-exporter-svqd5\" (UID: \"392d4ac7-d590-4225-bef4-c66590b846cb\") " pod="openshift-monitoring/node-exporter-svqd5"
Sep 09 20:54:57.837651 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:57.836468    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/392d4ac7-d590-4225-bef4-c66590b846cb-node-exporter-accelerators-collector-config\") pod \"node-exporter-svqd5\" (UID: \"392d4ac7-d590-4225-bef4-c66590b846cb\") " pod="openshift-monitoring/node-exporter-svqd5"
Sep 09 20:54:57.837651 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:57.836502    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/392d4ac7-d590-4225-bef4-c66590b846cb-node-exporter-textfile\") pod \"node-exporter-svqd5\" (UID: \"392d4ac7-d590-4225-bef4-c66590b846cb\") " pod="openshift-monitoring/node-exporter-svqd5"
Sep 09 20:54:57.837651 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:57.836528    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjjcc\" (UniqueName: \"kubernetes.io/projected/392d4ac7-d590-4225-bef4-c66590b846cb-kube-api-access-rjjcc\") pod \"node-exporter-svqd5\" (UID: \"392d4ac7-d590-4225-bef4-c66590b846cb\") " pod="openshift-monitoring/node-exporter-svqd5"
Sep 09 20:54:57.837651 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:57.836581    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/392d4ac7-d590-4225-bef4-c66590b846cb-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-svqd5\" (UID: \"392d4ac7-d590-4225-bef4-c66590b846cb\") " pod="openshift-monitoring/node-exporter-svqd5"
Sep 09 20:54:57.837651 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:57.836604    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/392d4ac7-d590-4225-bef4-c66590b846cb-metrics-client-ca\") pod \"node-exporter-svqd5\" (UID: \"392d4ac7-d590-4225-bef4-c66590b846cb\") " pod="openshift-monitoring/node-exporter-svqd5"
Sep 09 20:54:57.840646 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:57.840616    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:57.926739 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:57.926695    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:57.937723 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:57.937693    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjjcc\" (UniqueName: \"kubernetes.io/projected/392d4ac7-d590-4225-bef4-c66590b846cb-kube-api-access-rjjcc\") pod \"node-exporter-svqd5\" (UID: \"392d4ac7-d590-4225-bef4-c66590b846cb\") " pod="openshift-monitoring/node-exporter-svqd5"
Sep 09 20:54:57.937871 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:57.937754    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/392d4ac7-d590-4225-bef4-c66590b846cb-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-svqd5\" (UID: \"392d4ac7-d590-4225-bef4-c66590b846cb\") " pod="openshift-monitoring/node-exporter-svqd5"
Sep 09 20:54:57.937871 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:57.937782    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/392d4ac7-d590-4225-bef4-c66590b846cb-metrics-client-ca\") pod \"node-exporter-svqd5\" (UID: \"392d4ac7-d590-4225-bef4-c66590b846cb\") " pod="openshift-monitoring/node-exporter-svqd5"
Sep 09 20:54:57.937871 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:57.937801    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/392d4ac7-d590-4225-bef4-c66590b846cb-node-exporter-tls\") pod \"node-exporter-svqd5\" (UID: \"392d4ac7-d590-4225-bef4-c66590b846cb\") " pod="openshift-monitoring/node-exporter-svqd5"
Sep 09 20:54:57.937871 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:57.937818    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/392d4ac7-d590-4225-bef4-c66590b846cb-sys\") pod \"node-exporter-svqd5\" (UID: \"392d4ac7-d590-4225-bef4-c66590b846cb\") " pod="openshift-monitoring/node-exporter-svqd5"
Sep 09 20:54:57.937871 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:57.937833    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/392d4ac7-d590-4225-bef4-c66590b846cb-root\") pod \"node-exporter-svqd5\" (UID: \"392d4ac7-d590-4225-bef4-c66590b846cb\") " pod="openshift-monitoring/node-exporter-svqd5"
Sep 09 20:54:57.937871 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:57.937856    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/392d4ac7-d590-4225-bef4-c66590b846cb-node-exporter-wtmp\") pod \"node-exporter-svqd5\" (UID: \"392d4ac7-d590-4225-bef4-c66590b846cb\") " pod="openshift-monitoring/node-exporter-svqd5"
Sep 09 20:54:57.938180 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:57.937901    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/392d4ac7-d590-4225-bef4-c66590b846cb-node-exporter-accelerators-collector-config\") pod \"node-exporter-svqd5\" (UID: \"392d4ac7-d590-4225-bef4-c66590b846cb\") " pod="openshift-monitoring/node-exporter-svqd5"
Sep 09 20:54:57.938180 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:57.937923    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/392d4ac7-d590-4225-bef4-c66590b846cb-node-exporter-textfile\") pod \"node-exporter-svqd5\" (UID: \"392d4ac7-d590-4225-bef4-c66590b846cb\") " pod="openshift-monitoring/node-exporter-svqd5"
Sep 09 20:54:57.938281 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:57.938202    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/392d4ac7-d590-4225-bef4-c66590b846cb-node-exporter-textfile\") pod \"node-exporter-svqd5\" (UID: \"392d4ac7-d590-4225-bef4-c66590b846cb\") " pod="openshift-monitoring/node-exporter-svqd5"
Sep 09 20:54:57.939178 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:57.938780    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/392d4ac7-d590-4225-bef4-c66590b846cb-sys\") pod \"node-exporter-svqd5\" (UID: \"392d4ac7-d590-4225-bef4-c66590b846cb\") " pod="openshift-monitoring/node-exporter-svqd5"
Sep 09 20:54:57.939178 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:57.938799    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/392d4ac7-d590-4225-bef4-c66590b846cb-root\") pod \"node-exporter-svqd5\" (UID: \"392d4ac7-d590-4225-bef4-c66590b846cb\") " pod="openshift-monitoring/node-exporter-svqd5"
Sep 09 20:54:57.939360 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:57.939275    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/392d4ac7-d590-4225-bef4-c66590b846cb-node-exporter-wtmp\") pod \"node-exporter-svqd5\" (UID: \"392d4ac7-d590-4225-bef4-c66590b846cb\") " pod="openshift-monitoring/node-exporter-svqd5"
Sep 09 20:54:57.939360 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:57.939304    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/392d4ac7-d590-4225-bef4-c66590b846cb-metrics-client-ca\") pod \"node-exporter-svqd5\" (UID: \"392d4ac7-d590-4225-bef4-c66590b846cb\") " pod="openshift-monitoring/node-exporter-svqd5"
Sep 09 20:54:57.939360 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:57.939321    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/392d4ac7-d590-4225-bef4-c66590b846cb-node-exporter-accelerators-collector-config\") pod \"node-exporter-svqd5\" (UID: \"392d4ac7-d590-4225-bef4-c66590b846cb\") " pod="openshift-monitoring/node-exporter-svqd5"
Sep 09 20:54:57.941210 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:57.941185    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/392d4ac7-d590-4225-bef4-c66590b846cb-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-svqd5\" (UID: \"392d4ac7-d590-4225-bef4-c66590b846cb\") " pod="openshift-monitoring/node-exporter-svqd5"
Sep 09 20:54:57.945076 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:57.944681    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/392d4ac7-d590-4225-bef4-c66590b846cb-node-exporter-tls\") pod \"node-exporter-svqd5\" (UID: \"392d4ac7-d590-4225-bef4-c66590b846cb\") " pod="openshift-monitoring/node-exporter-svqd5"
Sep 09 20:54:57.947358 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:57.947316    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjjcc\" (UniqueName: \"kubernetes.io/projected/392d4ac7-d590-4225-bef4-c66590b846cb-kube-api-access-rjjcc\") pod \"node-exporter-svqd5\" (UID: \"392d4ac7-d590-4225-bef4-c66590b846cb\") " pod="openshift-monitoring/node-exporter-svqd5"
Sep 09 20:54:58.084330 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:58.084216    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:58.108063 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:58.108035    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-svqd5"
Sep 09 20:54:58.221266 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:58.221187    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:58.321438 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:58.321325    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:58.506222 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:58.506180    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:58.732439 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:54:58.732411    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod392d4ac7_d590_4225_bef4_c66590b846cb.slice/crio-67900c9608901228a795b60e76b8b63e03db933f39b11422aaa3584d1a17790a WatchSource:0}: Error finding container 67900c9608901228a795b60e76b8b63e03db933f39b11422aaa3584d1a17790a: Status 404 returned error can't find the container with id 67900c9608901228a795b60e76b8b63e03db933f39b11422aaa3584d1a17790a
Sep 09 20:54:58.747643 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:58.747613    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:58.847962 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:58.847846    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:58.855416 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:58.855359    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 20:54:58.858990 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:58.858971    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:54:58.863491 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:58.861923    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 09 20:54:58.863491 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:58.861981    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 09 20:54:58.863491 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:58.862013    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 09 20:54:58.863491 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:58.862167    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 09 20:54:58.863491 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:58.862338    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 09 20:54:58.863491 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:58.862412    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 09 20:54:58.863491 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:58.862342    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 09 20:54:58.863491 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:58.862753    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-f4nfl\""
Sep 09 20:54:58.864865 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:58.864846    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 09 20:54:58.865072 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:58.865042    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 09 20:54:58.873939 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:58.873919    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 20:54:58.916111 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:58.916081    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:58.947335 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:58.947299    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/22c8a115-79a9-4fc5-9cfa-c27836bf9e87-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"22c8a115-79a9-4fc5-9cfa-c27836bf9e87\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:54:58.947522 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:58.947347    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/22c8a115-79a9-4fc5-9cfa-c27836bf9e87-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"22c8a115-79a9-4fc5-9cfa-c27836bf9e87\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:54:58.947522 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:58.947433    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/22c8a115-79a9-4fc5-9cfa-c27836bf9e87-web-config\") pod \"alertmanager-main-0\" (UID: \"22c8a115-79a9-4fc5-9cfa-c27836bf9e87\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:54:58.947522 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:58.947459    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/22c8a115-79a9-4fc5-9cfa-c27836bf9e87-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"22c8a115-79a9-4fc5-9cfa-c27836bf9e87\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:54:58.947668 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:58.947536    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/22c8a115-79a9-4fc5-9cfa-c27836bf9e87-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"22c8a115-79a9-4fc5-9cfa-c27836bf9e87\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:54:58.947668 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:58.947567    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/22c8a115-79a9-4fc5-9cfa-c27836bf9e87-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"22c8a115-79a9-4fc5-9cfa-c27836bf9e87\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:54:58.947668 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:58.947646    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/22c8a115-79a9-4fc5-9cfa-c27836bf9e87-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"22c8a115-79a9-4fc5-9cfa-c27836bf9e87\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:54:58.947668 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:58.947663    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/22c8a115-79a9-4fc5-9cfa-c27836bf9e87-config-out\") pod \"alertmanager-main-0\" (UID: \"22c8a115-79a9-4fc5-9cfa-c27836bf9e87\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:54:58.947863 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:58.947689    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/22c8a115-79a9-4fc5-9cfa-c27836bf9e87-config-volume\") pod \"alertmanager-main-0\" (UID: \"22c8a115-79a9-4fc5-9cfa-c27836bf9e87\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:54:58.947863 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:58.947752    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/22c8a115-79a9-4fc5-9cfa-c27836bf9e87-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"22c8a115-79a9-4fc5-9cfa-c27836bf9e87\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:54:58.947863 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:58.947778    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/22c8a115-79a9-4fc5-9cfa-c27836bf9e87-tls-assets\") pod \"alertmanager-main-0\" (UID: \"22c8a115-79a9-4fc5-9cfa-c27836bf9e87\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:54:58.947863 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:58.947796    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/22c8a115-79a9-4fc5-9cfa-c27836bf9e87-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"22c8a115-79a9-4fc5-9cfa-c27836bf9e87\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:54:58.947863 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:58.947830    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jp2rj\" (UniqueName: \"kubernetes.io/projected/22c8a115-79a9-4fc5-9cfa-c27836bf9e87-kube-api-access-jp2rj\") pod \"alertmanager-main-0\" (UID: \"22c8a115-79a9-4fc5-9cfa-c27836bf9e87\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:54:58.966114 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:58.966074    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-svqd5" event={"ID":"392d4ac7-d590-4225-bef4-c66590b846cb","Type":"ContainerStarted","Data":"67900c9608901228a795b60e76b8b63e03db933f39b11422aaa3584d1a17790a"}
Sep 09 20:54:58.967871 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:58.967842    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f49d694d7-zg2p7" event={"ID":"d9bef8f5-e612-4504-a259-c2550b05cea2","Type":"ContainerStarted","Data":"d90939b8dec1653c60ae20fe3fc47b15c6c9df87448548def8eb221e2aaf0aef"}
Sep 09 20:54:58.992206 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:58.992148    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7f49d694d7-zg2p7" podStartSLOduration=1.058280438 podStartE2EDuration="4.992126787s" totalImagesPullingTime="3.933846349s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:54:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:54:58.990418323 +0000 UTC m=+142.104104184" watchObservedRunningTime="2026-09-09 20:54:58.992126787 +0000 UTC m=+142.105812649"
Sep 09 20:54:59.049101 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:59.049014    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/22c8a115-79a9-4fc5-9cfa-c27836bf9e87-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"22c8a115-79a9-4fc5-9cfa-c27836bf9e87\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:54:59.049251 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:59.049101    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jp2rj\" (UniqueName: \"kubernetes.io/projected/22c8a115-79a9-4fc5-9cfa-c27836bf9e87-kube-api-access-jp2rj\") pod \"alertmanager-main-0\" (UID: \"22c8a115-79a9-4fc5-9cfa-c27836bf9e87\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:54:59.049251 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:59.049214    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/22c8a115-79a9-4fc5-9cfa-c27836bf9e87-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"22c8a115-79a9-4fc5-9cfa-c27836bf9e87\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:54:59.049251 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:59.049241    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/22c8a115-79a9-4fc5-9cfa-c27836bf9e87-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"22c8a115-79a9-4fc5-9cfa-c27836bf9e87\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:54:59.049419 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:59.049273    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/22c8a115-79a9-4fc5-9cfa-c27836bf9e87-web-config\") pod \"alertmanager-main-0\" (UID: \"22c8a115-79a9-4fc5-9cfa-c27836bf9e87\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:54:59.049419 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:59.049298    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/22c8a115-79a9-4fc5-9cfa-c27836bf9e87-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"22c8a115-79a9-4fc5-9cfa-c27836bf9e87\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:54:59.049419 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:59.049348    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/22c8a115-79a9-4fc5-9cfa-c27836bf9e87-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"22c8a115-79a9-4fc5-9cfa-c27836bf9e87\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:54:59.049419 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:59.049397    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/22c8a115-79a9-4fc5-9cfa-c27836bf9e87-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"22c8a115-79a9-4fc5-9cfa-c27836bf9e87\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:54:59.049592 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:59.049431    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/22c8a115-79a9-4fc5-9cfa-c27836bf9e87-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"22c8a115-79a9-4fc5-9cfa-c27836bf9e87\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:54:59.049592 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:59.049448    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/22c8a115-79a9-4fc5-9cfa-c27836bf9e87-config-out\") pod \"alertmanager-main-0\" (UID: \"22c8a115-79a9-4fc5-9cfa-c27836bf9e87\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:54:59.049592 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:59.049470    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/22c8a115-79a9-4fc5-9cfa-c27836bf9e87-config-volume\") pod \"alertmanager-main-0\" (UID: \"22c8a115-79a9-4fc5-9cfa-c27836bf9e87\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:54:59.049592 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:59.049488    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/22c8a115-79a9-4fc5-9cfa-c27836bf9e87-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"22c8a115-79a9-4fc5-9cfa-c27836bf9e87\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:54:59.049592 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:59.049511    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/22c8a115-79a9-4fc5-9cfa-c27836bf9e87-tls-assets\") pod \"alertmanager-main-0\" (UID: \"22c8a115-79a9-4fc5-9cfa-c27836bf9e87\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:54:59.049850 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:59.049825    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/22c8a115-79a9-4fc5-9cfa-c27836bf9e87-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"22c8a115-79a9-4fc5-9cfa-c27836bf9e87\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:54:59.050244 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:59.050219    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/22c8a115-79a9-4fc5-9cfa-c27836bf9e87-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"22c8a115-79a9-4fc5-9cfa-c27836bf9e87\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:54:59.051230 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:59.051201    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/22c8a115-79a9-4fc5-9cfa-c27836bf9e87-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"22c8a115-79a9-4fc5-9cfa-c27836bf9e87\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:54:59.052849 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:59.052569    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/22c8a115-79a9-4fc5-9cfa-c27836bf9e87-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"22c8a115-79a9-4fc5-9cfa-c27836bf9e87\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:54:59.053042 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:59.053010    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/22c8a115-79a9-4fc5-9cfa-c27836bf9e87-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"22c8a115-79a9-4fc5-9cfa-c27836bf9e87\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:54:59.053352 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:59.053331    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/22c8a115-79a9-4fc5-9cfa-c27836bf9e87-tls-assets\") pod \"alertmanager-main-0\" (UID: \"22c8a115-79a9-4fc5-9cfa-c27836bf9e87\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:54:59.053802 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:59.053781    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/22c8a115-79a9-4fc5-9cfa-c27836bf9e87-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"22c8a115-79a9-4fc5-9cfa-c27836bf9e87\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:54:59.054703 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:59.054640    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/22c8a115-79a9-4fc5-9cfa-c27836bf9e87-config-volume\") pod \"alertmanager-main-0\" (UID: \"22c8a115-79a9-4fc5-9cfa-c27836bf9e87\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:54:59.054703 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:59.054650    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/22c8a115-79a9-4fc5-9cfa-c27836bf9e87-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"22c8a115-79a9-4fc5-9cfa-c27836bf9e87\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:54:59.054910 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:59.054878    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/22c8a115-79a9-4fc5-9cfa-c27836bf9e87-web-config\") pod \"alertmanager-main-0\" (UID: \"22c8a115-79a9-4fc5-9cfa-c27836bf9e87\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:54:59.055029 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:59.055011    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/22c8a115-79a9-4fc5-9cfa-c27836bf9e87-config-out\") pod \"alertmanager-main-0\" (UID: \"22c8a115-79a9-4fc5-9cfa-c27836bf9e87\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:54:59.055078 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:59.055045    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/22c8a115-79a9-4fc5-9cfa-c27836bf9e87-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"22c8a115-79a9-4fc5-9cfa-c27836bf9e87\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:54:59.058068 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:59.058047    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jp2rj\" (UniqueName: \"kubernetes.io/projected/22c8a115-79a9-4fc5-9cfa-c27836bf9e87-kube-api-access-jp2rj\") pod \"alertmanager-main-0\" (UID: \"22c8a115-79a9-4fc5-9cfa-c27836bf9e87\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:54:59.171275 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:59.171238    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:54:59.196802 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:59.196767    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:59.301053 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:59.300944    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:59.471554 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:59.471522    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 20:54:59.617703 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:54:59.617667    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22c8a115_79a9_4fc5_9cfa_c27836bf9e87.slice/crio-d6313c7a1debc4be33946a97459ae4aa89c8d7cba35187a4d83682c4a821612b WatchSource:0}: Error finding container d6313c7a1debc4be33946a97459ae4aa89c8d7cba35187a4d83682c4a821612b: Status 404 returned error can't find the container with id d6313c7a1debc4be33946a97459ae4aa89c8d7cba35187a4d83682c4a821612b
Sep 09 20:54:59.974469 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:59.974275    2576 generic.go:356] "Generic (PLEG): container finished" podID="392d4ac7-d590-4225-bef4-c66590b846cb" containerID="5efd713e354a49298cd468c83ff5c0775689ef8a891a45c37692171ec526430f" exitCode=0
Sep 09 20:54:59.974469 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:59.974389    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-svqd5" event={"ID":"392d4ac7-d590-4225-bef4-c66590b846cb","Type":"ContainerDied","Data":"5efd713e354a49298cd468c83ff5c0775689ef8a891a45c37692171ec526430f"}
Sep 09 20:54:59.975792 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:54:59.975763    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"22c8a115-79a9-4fc5-9cfa-c27836bf9e87","Type":"ContainerStarted","Data":"d6313c7a1debc4be33946a97459ae4aa89c8d7cba35187a4d83682c4a821612b"}
Sep 09 20:55:00.067423 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:00.067355    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:55:00.169046 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:00.169003    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:55:00.381249 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:00.381208    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:55:00.456117 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:00.456083    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:55:00.715440 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:00.715342    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:55:00.797658 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:00.797612    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:55:00.981527 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:00.981452    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-svqd5" event={"ID":"392d4ac7-d590-4225-bef4-c66590b846cb","Type":"ContainerStarted","Data":"3c626f27c2971b21025e7a8cd81d0cca11ddcc42875163679a64614a87dbaca1"}
Sep 09 20:55:01.145173 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:01.145140    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:55:01.541013 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:01.540975    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:55:02.534490 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:02.534453    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:55:02.619265 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:02.619222    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:55:03.001533 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:03.001502    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-74dffcb5f7-84lks"]
Sep 09 20:55:03.005137 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:03.005105    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:55:03.005279 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:03.005157    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74dffcb5f7-84lks"
Sep 09 20:55:03.013943 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:03.013919    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 09 20:55:03.015782 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:03.015760    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-74dffcb5f7-84lks"]
Sep 09 20:55:03.085721 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:03.085689    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/55572f58-18d5-4dc3-b3d0-483f96fc9564-trusted-ca-bundle\") pod \"console-74dffcb5f7-84lks\" (UID: \"55572f58-18d5-4dc3-b3d0-483f96fc9564\") " pod="openshift-console/console-74dffcb5f7-84lks"
Sep 09 20:55:03.085914 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:03.085761    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/55572f58-18d5-4dc3-b3d0-483f96fc9564-service-ca\") pod \"console-74dffcb5f7-84lks\" (UID: \"55572f58-18d5-4dc3-b3d0-483f96fc9564\") " pod="openshift-console/console-74dffcb5f7-84lks"
Sep 09 20:55:03.085914 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:03.085792    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/55572f58-18d5-4dc3-b3d0-483f96fc9564-console-config\") pod \"console-74dffcb5f7-84lks\" (UID: \"55572f58-18d5-4dc3-b3d0-483f96fc9564\") " pod="openshift-console/console-74dffcb5f7-84lks"
Sep 09 20:55:03.085914 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:03.085824    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzjg8\" (UniqueName: \"kubernetes.io/projected/55572f58-18d5-4dc3-b3d0-483f96fc9564-kube-api-access-rzjg8\") pod \"console-74dffcb5f7-84lks\" (UID: \"55572f58-18d5-4dc3-b3d0-483f96fc9564\") " pod="openshift-console/console-74dffcb5f7-84lks"
Sep 09 20:55:03.085914 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:03.085900    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/55572f58-18d5-4dc3-b3d0-483f96fc9564-console-oauth-config\") pod \"console-74dffcb5f7-84lks\" (UID: \"55572f58-18d5-4dc3-b3d0-483f96fc9564\") " pod="openshift-console/console-74dffcb5f7-84lks"
Sep 09 20:55:03.086110 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:03.085935    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/55572f58-18d5-4dc3-b3d0-483f96fc9564-oauth-serving-cert\") pod \"console-74dffcb5f7-84lks\" (UID: \"55572f58-18d5-4dc3-b3d0-483f96fc9564\") " pod="openshift-console/console-74dffcb5f7-84lks"
Sep 09 20:55:03.086110 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:03.085970    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/55572f58-18d5-4dc3-b3d0-483f96fc9564-console-serving-cert\") pod \"console-74dffcb5f7-84lks\" (UID: \"55572f58-18d5-4dc3-b3d0-483f96fc9564\") " pod="openshift-console/console-74dffcb5f7-84lks"
Sep 09 20:55:03.186730 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:03.186687    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/55572f58-18d5-4dc3-b3d0-483f96fc9564-service-ca\") pod \"console-74dffcb5f7-84lks\" (UID: \"55572f58-18d5-4dc3-b3d0-483f96fc9564\") " pod="openshift-console/console-74dffcb5f7-84lks"
Sep 09 20:55:03.186923 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:03.186742    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/55572f58-18d5-4dc3-b3d0-483f96fc9564-console-config\") pod \"console-74dffcb5f7-84lks\" (UID: \"55572f58-18d5-4dc3-b3d0-483f96fc9564\") " pod="openshift-console/console-74dffcb5f7-84lks"
Sep 09 20:55:03.186923 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:03.186780    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzjg8\" (UniqueName: \"kubernetes.io/projected/55572f58-18d5-4dc3-b3d0-483f96fc9564-kube-api-access-rzjg8\") pod \"console-74dffcb5f7-84lks\" (UID: \"55572f58-18d5-4dc3-b3d0-483f96fc9564\") " pod="openshift-console/console-74dffcb5f7-84lks"
Sep 09 20:55:03.186923 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:03.186811    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/55572f58-18d5-4dc3-b3d0-483f96fc9564-console-oauth-config\") pod \"console-74dffcb5f7-84lks\" (UID: \"55572f58-18d5-4dc3-b3d0-483f96fc9564\") " pod="openshift-console/console-74dffcb5f7-84lks"
Sep 09 20:55:03.186923 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:03.186837    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/55572f58-18d5-4dc3-b3d0-483f96fc9564-oauth-serving-cert\") pod \"console-74dffcb5f7-84lks\" (UID: \"55572f58-18d5-4dc3-b3d0-483f96fc9564\") " pod="openshift-console/console-74dffcb5f7-84lks"
Sep 09 20:55:03.186923 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:03.186872    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/55572f58-18d5-4dc3-b3d0-483f96fc9564-console-serving-cert\") pod \"console-74dffcb5f7-84lks\" (UID: \"55572f58-18d5-4dc3-b3d0-483f96fc9564\") " pod="openshift-console/console-74dffcb5f7-84lks"
Sep 09 20:55:03.187173 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:03.186959    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/55572f58-18d5-4dc3-b3d0-483f96fc9564-trusted-ca-bundle\") pod \"console-74dffcb5f7-84lks\" (UID: \"55572f58-18d5-4dc3-b3d0-483f96fc9564\") " pod="openshift-console/console-74dffcb5f7-84lks"
Sep 09 20:55:03.187655 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:03.187557    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/55572f58-18d5-4dc3-b3d0-483f96fc9564-service-ca\") pod \"console-74dffcb5f7-84lks\" (UID: \"55572f58-18d5-4dc3-b3d0-483f96fc9564\") " pod="openshift-console/console-74dffcb5f7-84lks"
Sep 09 20:55:03.187655 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:03.187574    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/55572f58-18d5-4dc3-b3d0-483f96fc9564-oauth-serving-cert\") pod \"console-74dffcb5f7-84lks\" (UID: \"55572f58-18d5-4dc3-b3d0-483f96fc9564\") " pod="openshift-console/console-74dffcb5f7-84lks"
Sep 09 20:55:03.187908 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:03.187764    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/55572f58-18d5-4dc3-b3d0-483f96fc9564-console-config\") pod \"console-74dffcb5f7-84lks\" (UID: \"55572f58-18d5-4dc3-b3d0-483f96fc9564\") " pod="openshift-console/console-74dffcb5f7-84lks"
Sep 09 20:55:03.187908 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:03.187833    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/55572f58-18d5-4dc3-b3d0-483f96fc9564-trusted-ca-bundle\") pod \"console-74dffcb5f7-84lks\" (UID: \"55572f58-18d5-4dc3-b3d0-483f96fc9564\") " pod="openshift-console/console-74dffcb5f7-84lks"
Sep 09 20:55:03.189627 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:03.189594    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/55572f58-18d5-4dc3-b3d0-483f96fc9564-console-oauth-config\") pod \"console-74dffcb5f7-84lks\" (UID: \"55572f58-18d5-4dc3-b3d0-483f96fc9564\") " pod="openshift-console/console-74dffcb5f7-84lks"
Sep 09 20:55:03.189864 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:03.189845    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/55572f58-18d5-4dc3-b3d0-483f96fc9564-console-serving-cert\") pod \"console-74dffcb5f7-84lks\" (UID: \"55572f58-18d5-4dc3-b3d0-483f96fc9564\") " pod="openshift-console/console-74dffcb5f7-84lks"
Sep 09 20:55:03.195993 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:03.195962    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzjg8\" (UniqueName: \"kubernetes.io/projected/55572f58-18d5-4dc3-b3d0-483f96fc9564-kube-api-access-rzjg8\") pod \"console-74dffcb5f7-84lks\" (UID: \"55572f58-18d5-4dc3-b3d0-483f96fc9564\") " pod="openshift-console/console-74dffcb5f7-84lks"
Sep 09 20:55:03.310215 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:03.310122    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:55:03.317423 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:03.317387    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74dffcb5f7-84lks"
Sep 09 20:55:03.446335 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:03.446301    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:55:03.904216 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:03.904169    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:55:03.999774 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:03.999738    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:55:04.089355 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:04.089300    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:55:04.559662 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:04.559629    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7f49d694d7-zg2p7"
Sep 09 20:55:04.559891 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:04.559697    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7f49d694d7-zg2p7"
Sep 09 20:55:04.565516 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:04.565483    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7f49d694d7-zg2p7"
Sep 09 20:55:05.001019 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:05.000988    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7f49d694d7-zg2p7"
Sep 09 20:55:07.344081 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:07.344040    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-74dffcb5f7-84lks"]
Sep 09 20:55:07.348785 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:55:07.348754    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55572f58_18d5_4dc3_b3d0_483f96fc9564.slice/crio-8df0df82ecf788eff96e73cc61c0d9c41f08132bbad48bb1e44c368f27def660 WatchSource:0}: Error finding container 8df0df82ecf788eff96e73cc61c0d9c41f08132bbad48bb1e44c368f27def660: Status 404 returned error can't find the container with id 8df0df82ecf788eff96e73cc61c0d9c41f08132bbad48bb1e44c368f27def660
Sep 09 20:55:08.011929 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:08.011883    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-svqd5" event={"ID":"392d4ac7-d590-4225-bef4-c66590b846cb","Type":"ContainerStarted","Data":"81a18b71bfc9a20a8afae7934c2b0e01375b5933fe835895b50301b000a8a5e6"}
Sep 09 20:55:08.014267 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:08.014238    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-85d98c998f-wqqtc" event={"ID":"ca8ff022-fc16-478e-8dad-d819bf9331f8","Type":"ContainerStarted","Data":"d1b07b6e6a59cb80b5e2914d1960c6aa12e228db9aee7af1a55330cc0ed753f2"}
Sep 09 20:55:08.014488 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:08.014445    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-85d98c998f-wqqtc"
Sep 09 20:55:08.017000 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:08.016308    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74dffcb5f7-84lks" event={"ID":"55572f58-18d5-4dc3-b3d0-483f96fc9564","Type":"ContainerStarted","Data":"878367a807ac6565e8e0aa9a332f33df6ec2dff0831169c31baa7e5c8c8635c7"}
Sep 09 20:55:08.017000 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:08.016354    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74dffcb5f7-84lks" event={"ID":"55572f58-18d5-4dc3-b3d0-483f96fc9564","Type":"ContainerStarted","Data":"8df0df82ecf788eff96e73cc61c0d9c41f08132bbad48bb1e44c368f27def660"}
Sep 09 20:55:08.030837 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:08.030815    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-85d98c998f-wqqtc"
Sep 09 20:55:08.067048 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:08.066987    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-85d98c998f-wqqtc" podStartSLOduration=2.455061356 podStartE2EDuration="20.066964597s" totalImagesPullingTime="17.611903241s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:54:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:55:08.065546093 +0000 UTC m=+151.179231953" watchObservedRunningTime="2026-09-09 20:55:08.066964597 +0000 UTC m=+151.180650456"
Sep 09 20:55:08.067416 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:08.067353    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-svqd5" podStartSLOduration=10.058586795 podStartE2EDuration="11.067342209s" totalImagesPullingTime="928.712151ms" totalInitContainerRuntime="80.043263ms" isStatefulPod=true podCreationTimestamp="2026-09-09 20:54:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:55:08.04448349 +0000 UTC m=+151.158169355" watchObservedRunningTime="2026-09-09 20:55:08.067342209 +0000 UTC m=+151.181028069"
Sep 09 20:55:08.092452 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:08.092398    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-74dffcb5f7-84lks" podStartSLOduration=6.092351234 podStartE2EDuration="6.092351234s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:55:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:55:08.0898415 +0000 UTC m=+151.203527363" watchObservedRunningTime="2026-09-09 20:55:08.092351234 +0000 UTC m=+151.206037093"
Sep 09 20:55:09.022316 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:09.022274    2576 generic.go:356] "Generic (PLEG): container finished" podID="22c8a115-79a9-4fc5-9cfa-c27836bf9e87" containerID="7050ee4f1aa21ddb004c36fc5ff9affe48a00fcc5effbf2b605229e5261d8081" exitCode=0
Sep 09 20:55:09.022793 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:09.022419    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"22c8a115-79a9-4fc5-9cfa-c27836bf9e87","Type":"ContainerDied","Data":"7050ee4f1aa21ddb004c36fc5ff9affe48a00fcc5effbf2b605229e5261d8081"}
Sep 09 20:55:09.477528 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:09.477492    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:55:09.578293 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:09.578262    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:55:11.244356 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:11.244332    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-cb585848c-289xj"]
Sep 09 20:55:11.244705 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:55:11.244675    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-cb585848c-289xj" podUID="0989b0f6-8115-4152-95ae-1f52fdb2012a"
Sep 09 20:55:11.355236 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:11.355207    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:55:11.458643 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:11.458612    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:55:11.570018 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:11.569987    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:55:12.041485 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:12.041396    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"22c8a115-79a9-4fc5-9cfa-c27836bf9e87","Type":"ContainerStarted","Data":"cd42518272da483a57eb7077af7cd9725c494414f40b11d87db91f80855e9a0c"}
Sep 09 20:55:12.041485 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:12.041445    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-cb585848c-289xj"
Sep 09 20:55:12.041719 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:12.041485    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"22c8a115-79a9-4fc5-9cfa-c27836bf9e87","Type":"ContainerStarted","Data":"c13fe4c4e6f5bcc236d1c6a1478dbb3af5f7d987f29a4a8af596725cdc00d72d"}
Sep 09 20:55:12.041719 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:12.041525    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"22c8a115-79a9-4fc5-9cfa-c27836bf9e87","Type":"ContainerStarted","Data":"6db272b15daab520126736cba42eb7748920ee4541bc34d75012999487ae9530"}
Sep 09 20:55:12.041719 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:12.041560    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"22c8a115-79a9-4fc5-9cfa-c27836bf9e87","Type":"ContainerStarted","Data":"bdbe6c30211e7cd33d952cd3a974d29d9ac3df0147ae05f4b490a50af5f1debd"}
Sep 09 20:55:12.041719 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:12.041598    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"22c8a115-79a9-4fc5-9cfa-c27836bf9e87","Type":"ContainerStarted","Data":"aeecedf2fa0f95cc85b5cadfecc5cf9f8183849fd478f54a8d130bb4aba6500b"}
Sep 09 20:55:12.065118 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:12.065086    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-cb585848c-289xj"
Sep 09 20:55:12.172113 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:12.172083    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0989b0f6-8115-4152-95ae-1f52fdb2012a-registry-certificates\") pod \"0989b0f6-8115-4152-95ae-1f52fdb2012a\" (UID: \"0989b0f6-8115-4152-95ae-1f52fdb2012a\") "
Sep 09 20:55:12.172265 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:12.172131    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0989b0f6-8115-4152-95ae-1f52fdb2012a-bound-sa-token\") pod \"0989b0f6-8115-4152-95ae-1f52fdb2012a\" (UID: \"0989b0f6-8115-4152-95ae-1f52fdb2012a\") "
Sep 09 20:55:12.172265 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:12.172192    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0989b0f6-8115-4152-95ae-1f52fdb2012a-installation-pull-secrets\") pod \"0989b0f6-8115-4152-95ae-1f52fdb2012a\" (UID: \"0989b0f6-8115-4152-95ae-1f52fdb2012a\") "
Sep 09 20:55:12.172265 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:12.172227    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0989b0f6-8115-4152-95ae-1f52fdb2012a-ca-trust-extracted\") pod \"0989b0f6-8115-4152-95ae-1f52fdb2012a\" (UID: \"0989b0f6-8115-4152-95ae-1f52fdb2012a\") "
Sep 09 20:55:12.172265 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:12.172248    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tkjng\" (UniqueName: \"kubernetes.io/projected/0989b0f6-8115-4152-95ae-1f52fdb2012a-kube-api-access-tkjng\") pod \"0989b0f6-8115-4152-95ae-1f52fdb2012a\" (UID: \"0989b0f6-8115-4152-95ae-1f52fdb2012a\") "
Sep 09 20:55:12.172477 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:12.172289    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0989b0f6-8115-4152-95ae-1f52fdb2012a-image-registry-private-configuration\") pod \"0989b0f6-8115-4152-95ae-1f52fdb2012a\" (UID: \"0989b0f6-8115-4152-95ae-1f52fdb2012a\") "
Sep 09 20:55:12.172477 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:12.172333    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0989b0f6-8115-4152-95ae-1f52fdb2012a-trusted-ca\") pod \"0989b0f6-8115-4152-95ae-1f52fdb2012a\" (UID: \"0989b0f6-8115-4152-95ae-1f52fdb2012a\") "
Sep 09 20:55:12.172575 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:12.172503    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0989b0f6-8115-4152-95ae-1f52fdb2012a-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "0989b0f6-8115-4152-95ae-1f52fdb2012a" (UID: "0989b0f6-8115-4152-95ae-1f52fdb2012a"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:55:12.172575 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:12.172546    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0989b0f6-8115-4152-95ae-1f52fdb2012a-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "0989b0f6-8115-4152-95ae-1f52fdb2012a" (UID: "0989b0f6-8115-4152-95ae-1f52fdb2012a"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:55:12.172693 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:12.172645    2576 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0989b0f6-8115-4152-95ae-1f52fdb2012a-registry-certificates\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 20:55:12.172693 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:12.172663    2576 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0989b0f6-8115-4152-95ae-1f52fdb2012a-ca-trust-extracted\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 20:55:12.172906 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:12.172879    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0989b0f6-8115-4152-95ae-1f52fdb2012a-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "0989b0f6-8115-4152-95ae-1f52fdb2012a" (UID: "0989b0f6-8115-4152-95ae-1f52fdb2012a"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:55:12.175200 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:12.175174    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0989b0f6-8115-4152-95ae-1f52fdb2012a-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "0989b0f6-8115-4152-95ae-1f52fdb2012a" (UID: "0989b0f6-8115-4152-95ae-1f52fdb2012a"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:55:12.175311 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:12.175213    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0989b0f6-8115-4152-95ae-1f52fdb2012a-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "0989b0f6-8115-4152-95ae-1f52fdb2012a" (UID: "0989b0f6-8115-4152-95ae-1f52fdb2012a"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:55:12.175413 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:12.175331    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0989b0f6-8115-4152-95ae-1f52fdb2012a-kube-api-access-tkjng" (OuterVolumeSpecName: "kube-api-access-tkjng") pod "0989b0f6-8115-4152-95ae-1f52fdb2012a" (UID: "0989b0f6-8115-4152-95ae-1f52fdb2012a"). InnerVolumeSpecName "kube-api-access-tkjng". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:55:12.175482 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:12.175461    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0989b0f6-8115-4152-95ae-1f52fdb2012a-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "0989b0f6-8115-4152-95ae-1f52fdb2012a" (UID: "0989b0f6-8115-4152-95ae-1f52fdb2012a"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:55:12.200679 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:12.200652    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:55:12.273484 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:12.273450    2576 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0989b0f6-8115-4152-95ae-1f52fdb2012a-image-registry-private-configuration\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 20:55:12.273484 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:12.273486    2576 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0989b0f6-8115-4152-95ae-1f52fdb2012a-trusted-ca\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 20:55:12.273994 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:12.273501    2576 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0989b0f6-8115-4152-95ae-1f52fdb2012a-bound-sa-token\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 20:55:12.273994 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:12.273514    2576 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0989b0f6-8115-4152-95ae-1f52fdb2012a-installation-pull-secrets\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 20:55:12.273994 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:12.273524    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tkjng\" (UniqueName: \"kubernetes.io/projected/0989b0f6-8115-4152-95ae-1f52fdb2012a-kube-api-access-tkjng\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 20:55:12.285215 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:12.285190    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:55:12.619770 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:12.619728    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:55:12.745006 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:12.744977    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:55:13.045671 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:13.045608    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-cb585848c-289xj"
Sep 09 20:55:13.082714 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:13.082683    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-cb585848c-289xj"]
Sep 09 20:55:13.084996 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:13.084970    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-cb585848c-289xj"]
Sep 09 20:55:13.182582 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:13.182554    2576 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0989b0f6-8115-4152-95ae-1f52fdb2012a-registry-tls\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 20:55:13.318425 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:13.318341    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-74dffcb5f7-84lks"
Sep 09 20:55:13.318850 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:13.318496    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-74dffcb5f7-84lks"
Sep 09 20:55:13.324257 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:13.324231    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-74dffcb5f7-84lks"
Sep 09 20:55:13.409505 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:13.409468    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0989b0f6-8115-4152-95ae-1f52fdb2012a" path="/var/lib/kubelet/pods/0989b0f6-8115-4152-95ae-1f52fdb2012a/volumes"
Sep 09 20:55:14.054853 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:14.054798    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"22c8a115-79a9-4fc5-9cfa-c27836bf9e87","Type":"ContainerStarted","Data":"3de3ac7298aec2be9a0a8b3bba8eae1f78a01a7ff0eb5ff3f619c1dd37b96785"}
Sep 09 20:55:14.059639 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:14.059617    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-74dffcb5f7-84lks"
Sep 09 20:55:14.092129 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:14.092065    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=4.1468094 podStartE2EDuration="16.092048534s" totalImagesPullingTime="11.919511828s" totalInitContainerRuntime="25.727306ms" isStatefulPod=false podCreationTimestamp="2026-09-09 20:54:58 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:55:14.089774498 +0000 UTC m=+157.203460364" watchObservedRunningTime="2026-09-09 20:55:14.092048534 +0000 UTC m=+157.205734394"
Sep 09 20:55:14.147585 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:14.147550    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7f49d694d7-zg2p7"]
Sep 09 20:55:14.436588 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:14.436550    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:55:14.754663 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:14.754580    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:55:14.854139 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:14.854102    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:55:15.650020 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:15.649979    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:55:15.759129 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:15.759074    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:55:16.291375 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:55:16.291323    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-968c2" podUID="f69e7fb4-fa7d-444e-9bfa-d173783106c2"
Sep 09 20:55:17.040411 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:17.040381    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:55:17.066983 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:17.066960    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-968c2"
Sep 09 20:55:17.443537 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:17.443494    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:55:17.545335 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:17.545300    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:55:17.645956 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:17.645919    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:55:21.152674 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:21.152635    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f69e7fb4-fa7d-444e-9bfa-d173783106c2-metrics-tls\") pod \"dns-default-968c2\" (UID: \"f69e7fb4-fa7d-444e-9bfa-d173783106c2\") " pod="openshift-dns/dns-default-968c2"
Sep 09 20:55:21.155110 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:21.155086    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f69e7fb4-fa7d-444e-9bfa-d173783106c2-metrics-tls\") pod \"dns-default-968c2\" (UID: \"f69e7fb4-fa7d-444e-9bfa-d173783106c2\") " pod="openshift-dns/dns-default-968c2"
Sep 09 20:55:21.271317 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:21.271286    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-nr885\""
Sep 09 20:55:21.279203 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:21.279188    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-968c2"
Sep 09 20:55:21.607927 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:21.607904    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-968c2"]
Sep 09 20:55:21.610495 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:55:21.610458    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf69e7fb4_fa7d_444e_9bfa_d173783106c2.slice/crio-5507c9df3c89957b601dd75454d73b3ad49f07b052324a540320bd0f4c029f04 WatchSource:0}: Error finding container 5507c9df3c89957b601dd75454d73b3ad49f07b052324a540320bd0f4c029f04: Status 404 returned error can't find the container with id 5507c9df3c89957b601dd75454d73b3ad49f07b052324a540320bd0f4c029f04
Sep 09 20:55:22.085917 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:22.085825    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-968c2" event={"ID":"f69e7fb4-fa7d-444e-9bfa-d173783106c2","Type":"ContainerStarted","Data":"5507c9df3c89957b601dd75454d73b3ad49f07b052324a540320bd0f4c029f04"}
Sep 09 20:55:23.740080 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:23.740037    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:55:23.839351 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:23.839319    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:55:23.940269 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:23.940234    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:55:24.095278 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:24.095243    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-968c2" event={"ID":"f69e7fb4-fa7d-444e-9bfa-d173783106c2","Type":"ContainerStarted","Data":"54e88a4f69d6cc168ec5d245cead424d254fd60d51e7c2a07b96fc6e4ea9eb20"}
Sep 09 20:55:24.095506 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:24.095303    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-968c2" event={"ID":"f69e7fb4-fa7d-444e-9bfa-d173783106c2","Type":"ContainerStarted","Data":"452fc90370628cf11dec8431260722234ba27fbd1cdc9268d74fd08246641ef7"}
Sep 09 20:55:24.095506 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:24.095332    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-968c2"
Sep 09 20:55:24.114236 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:24.114192    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-968c2" podStartSLOduration=128.997998884 podStartE2EDuration="2m11.114177895s" totalImagesPullingTime="2.116179011s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:53:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:55:24.112041921 +0000 UTC m=+167.225727791" watchObservedRunningTime="2026-09-09 20:55:24.114177895 +0000 UTC m=+167.227863753"
Sep 09 20:55:29.225390 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:29.225299    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:55:29.325690 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:29.325653    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:55:34.101715 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:34.101684    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-968c2"
Sep 09 20:55:37.144115 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:37.144084    2576 generic.go:356] "Generic (PLEG): container finished" podID="8dd00baa-1d00-498a-a56a-bc7f90c53317" containerID="eeb87b29c07e2b1fe6cd0b2a7a6c6dc07dcccb13c422b393c48d18c476b78d45" exitCode=0
Sep 09 20:55:37.144486 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:37.144158    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-l9q9f" event={"ID":"8dd00baa-1d00-498a-a56a-bc7f90c53317","Type":"ContainerDied","Data":"eeb87b29c07e2b1fe6cd0b2a7a6c6dc07dcccb13c422b393c48d18c476b78d45"}
Sep 09 20:55:37.144526 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:37.144509    2576 scope.go:117] "RemoveContainer" containerID="eeb87b29c07e2b1fe6cd0b2a7a6c6dc07dcccb13c422b393c48d18c476b78d45"
Sep 09 20:55:38.150659 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:38.150626    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-l9q9f" event={"ID":"8dd00baa-1d00-498a-a56a-bc7f90c53317","Type":"ContainerStarted","Data":"2a983da24a15b7543731004a02ebb9be75823dfd9151e8db612235e1ed38ccde"}
Sep 09 20:55:38.255101 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:38.255070    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-968c2_f69e7fb4-fa7d-444e-9bfa-d173783106c2/dns/0.log"
Sep 09 20:55:38.260940 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:38.260913    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-968c2_f69e7fb4-fa7d-444e-9bfa-d173783106c2/kube-rbac-proxy/0.log"
Sep 09 20:55:38.761402 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:38.761359    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-wvw9h_09f40168-2bb2-4eff-8a3f-7ef28a149eed/dns-node-resolver/0.log"
Sep 09 20:55:39.175107 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:39.175073    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7f49d694d7-zg2p7" podUID="d9bef8f5-e612-4504-a259-c2550b05cea2" containerName="console" containerID="cri-o://d90939b8dec1653c60ae20fe3fc47b15c6c9df87448548def8eb221e2aaf0aef" gracePeriod=15
Sep 09 20:55:39.440596 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:39.440526    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f49d694d7-zg2p7"
Sep 09 20:55:39.595315 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:39.595284    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qbj9p\" (UniqueName: \"kubernetes.io/projected/d9bef8f5-e612-4504-a259-c2550b05cea2-kube-api-access-qbj9p\") pod \"d9bef8f5-e612-4504-a259-c2550b05cea2\" (UID: \"d9bef8f5-e612-4504-a259-c2550b05cea2\") "
Sep 09 20:55:39.595315 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:39.595331    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d9bef8f5-e612-4504-a259-c2550b05cea2-oauth-serving-cert\") pod \"d9bef8f5-e612-4504-a259-c2550b05cea2\" (UID: \"d9bef8f5-e612-4504-a259-c2550b05cea2\") "
Sep 09 20:55:39.595577 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:39.595394    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d9bef8f5-e612-4504-a259-c2550b05cea2-service-ca\") pod \"d9bef8f5-e612-4504-a259-c2550b05cea2\" (UID: \"d9bef8f5-e612-4504-a259-c2550b05cea2\") "
Sep 09 20:55:39.595577 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:39.595456    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d9bef8f5-e612-4504-a259-c2550b05cea2-console-oauth-config\") pod \"d9bef8f5-e612-4504-a259-c2550b05cea2\" (UID: \"d9bef8f5-e612-4504-a259-c2550b05cea2\") "
Sep 09 20:55:39.595577 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:39.595484    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d9bef8f5-e612-4504-a259-c2550b05cea2-console-serving-cert\") pod \"d9bef8f5-e612-4504-a259-c2550b05cea2\" (UID: \"d9bef8f5-e612-4504-a259-c2550b05cea2\") "
Sep 09 20:55:39.595577 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:39.595524    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d9bef8f5-e612-4504-a259-c2550b05cea2-console-config\") pod \"d9bef8f5-e612-4504-a259-c2550b05cea2\" (UID: \"d9bef8f5-e612-4504-a259-c2550b05cea2\") "
Sep 09 20:55:39.595792 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:39.595760    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9bef8f5-e612-4504-a259-c2550b05cea2-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "d9bef8f5-e612-4504-a259-c2550b05cea2" (UID: "d9bef8f5-e612-4504-a259-c2550b05cea2"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:55:39.595963 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:39.595926    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9bef8f5-e612-4504-a259-c2550b05cea2-service-ca" (OuterVolumeSpecName: "service-ca") pod "d9bef8f5-e612-4504-a259-c2550b05cea2" (UID: "d9bef8f5-e612-4504-a259-c2550b05cea2"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:55:39.595963 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:39.595953    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9bef8f5-e612-4504-a259-c2550b05cea2-console-config" (OuterVolumeSpecName: "console-config") pod "d9bef8f5-e612-4504-a259-c2550b05cea2" (UID: "d9bef8f5-e612-4504-a259-c2550b05cea2"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:55:39.597791 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:39.597763    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9bef8f5-e612-4504-a259-c2550b05cea2-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "d9bef8f5-e612-4504-a259-c2550b05cea2" (UID: "d9bef8f5-e612-4504-a259-c2550b05cea2"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:55:39.597791 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:39.597773    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9bef8f5-e612-4504-a259-c2550b05cea2-kube-api-access-qbj9p" (OuterVolumeSpecName: "kube-api-access-qbj9p") pod "d9bef8f5-e612-4504-a259-c2550b05cea2" (UID: "d9bef8f5-e612-4504-a259-c2550b05cea2"). InnerVolumeSpecName "kube-api-access-qbj9p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:55:39.597791 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:39.597780    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9bef8f5-e612-4504-a259-c2550b05cea2-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "d9bef8f5-e612-4504-a259-c2550b05cea2" (UID: "d9bef8f5-e612-4504-a259-c2550b05cea2"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:55:39.696856 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:39.696784    2576 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d9bef8f5-e612-4504-a259-c2550b05cea2-console-oauth-config\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 20:55:39.696856 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:39.696812    2576 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d9bef8f5-e612-4504-a259-c2550b05cea2-console-serving-cert\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 20:55:39.696856 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:39.696824    2576 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d9bef8f5-e612-4504-a259-c2550b05cea2-console-config\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 20:55:39.696856 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:39.696832    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbj9p\" (UniqueName: \"kubernetes.io/projected/d9bef8f5-e612-4504-a259-c2550b05cea2-kube-api-access-qbj9p\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 20:55:39.696856 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:39.696841    2576 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d9bef8f5-e612-4504-a259-c2550b05cea2-oauth-serving-cert\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 20:55:39.696856 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:39.696849    2576 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d9bef8f5-e612-4504-a259-c2550b05cea2-service-ca\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 20:55:40.159610 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:40.159579    2576 generic.go:356] "Generic (PLEG): container finished" podID="d9bef8f5-e612-4504-a259-c2550b05cea2" containerID="d90939b8dec1653c60ae20fe3fc47b15c6c9df87448548def8eb221e2aaf0aef" exitCode=0
Sep 09 20:55:40.159855 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:40.159631    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f49d694d7-zg2p7"
Sep 09 20:55:40.159855 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:40.159673    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f49d694d7-zg2p7" event={"ID":"d9bef8f5-e612-4504-a259-c2550b05cea2","Type":"ContainerDied","Data":"d90939b8dec1653c60ae20fe3fc47b15c6c9df87448548def8eb221e2aaf0aef"}
Sep 09 20:55:40.159855 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:40.159725    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f49d694d7-zg2p7" event={"ID":"d9bef8f5-e612-4504-a259-c2550b05cea2","Type":"ContainerDied","Data":"cbce41b52036031d896cdf7a96fc67edf95ad7f04d49254c1f3ad6a3107737d1"}
Sep 09 20:55:40.159855 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:40.159754    2576 scope.go:117] "RemoveContainer" containerID="d90939b8dec1653c60ae20fe3fc47b15c6c9df87448548def8eb221e2aaf0aef"
Sep 09 20:55:40.168310 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:40.168293    2576 scope.go:117] "RemoveContainer" containerID="d90939b8dec1653c60ae20fe3fc47b15c6c9df87448548def8eb221e2aaf0aef"
Sep 09 20:55:40.168579 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:55:40.168556    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d90939b8dec1653c60ae20fe3fc47b15c6c9df87448548def8eb221e2aaf0aef\": container with ID starting with d90939b8dec1653c60ae20fe3fc47b15c6c9df87448548def8eb221e2aaf0aef not found: ID does not exist" containerID="d90939b8dec1653c60ae20fe3fc47b15c6c9df87448548def8eb221e2aaf0aef"
Sep 09 20:55:40.168640 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:40.168588    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d90939b8dec1653c60ae20fe3fc47b15c6c9df87448548def8eb221e2aaf0aef"} err="failed to get container status \"d90939b8dec1653c60ae20fe3fc47b15c6c9df87448548def8eb221e2aaf0aef\": rpc error: code = NotFound desc = could not find container \"d90939b8dec1653c60ae20fe3fc47b15c6c9df87448548def8eb221e2aaf0aef\": container with ID starting with d90939b8dec1653c60ae20fe3fc47b15c6c9df87448548def8eb221e2aaf0aef not found: ID does not exist"
Sep 09 20:55:40.180144 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:40.180121    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7f49d694d7-zg2p7"]
Sep 09 20:55:40.184233 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:40.184210    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7f49d694d7-zg2p7"]
Sep 09 20:55:41.407448 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:41.407416    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d9bef8f5-e612-4504-a259-c2550b05cea2" path="/var/lib/kubelet/pods/d9bef8f5-e612-4504-a259-c2550b05cea2/volumes"
Sep 09 20:55:43.266056 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:43.266017    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:55:44.245090 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:44.245043    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:55:44.319190 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:44.319158    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:55:52.293204 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:52.293172    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:55:52.372514 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:55:52.372476    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:56:17.070389 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:17.070333    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:56:17.549008 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:17.548973    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:56:18.113470 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:18.113439    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 20:56:18.113823 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:18.113717    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="22c8a115-79a9-4fc5-9cfa-c27836bf9e87" containerName="alertmanager" containerID="cri-o://aeecedf2fa0f95cc85b5cadfecc5cf9f8183849fd478f54a8d130bb4aba6500b" gracePeriod=120
Sep 09 20:56:18.113823 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:18.113786    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="22c8a115-79a9-4fc5-9cfa-c27836bf9e87" containerName="kube-rbac-proxy-metric" containerID="cri-o://cd42518272da483a57eb7077af7cd9725c494414f40b11d87db91f80855e9a0c" gracePeriod=120
Sep 09 20:56:18.113927 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:18.113809    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="22c8a115-79a9-4fc5-9cfa-c27836bf9e87" containerName="kube-rbac-proxy-web" containerID="cri-o://6db272b15daab520126736cba42eb7748920ee4541bc34d75012999487ae9530" gracePeriod=120
Sep 09 20:56:18.113927 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:18.113828    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="22c8a115-79a9-4fc5-9cfa-c27836bf9e87" containerName="kube-rbac-proxy" containerID="cri-o://c13fe4c4e6f5bcc236d1c6a1478dbb3af5f7d987f29a4a8af596725cdc00d72d" gracePeriod=120
Sep 09 20:56:18.113927 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:18.113867    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="22c8a115-79a9-4fc5-9cfa-c27836bf9e87" containerName="prom-label-proxy" containerID="cri-o://3de3ac7298aec2be9a0a8b3bba8eae1f78a01a7ff0eb5ff3f619c1dd37b96785" gracePeriod=120
Sep 09 20:56:18.114073 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:18.113905    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="22c8a115-79a9-4fc5-9cfa-c27836bf9e87" containerName="config-reloader" containerID="cri-o://bdbe6c30211e7cd33d952cd3a974d29d9ac3df0147ae05f4b490a50af5f1debd" gracePeriod=120
Sep 09 20:56:18.311421 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:18.311388    2576 generic.go:356] "Generic (PLEG): container finished" podID="22c8a115-79a9-4fc5-9cfa-c27836bf9e87" containerID="3de3ac7298aec2be9a0a8b3bba8eae1f78a01a7ff0eb5ff3f619c1dd37b96785" exitCode=0
Sep 09 20:56:18.311421 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:18.311416    2576 generic.go:356] "Generic (PLEG): container finished" podID="22c8a115-79a9-4fc5-9cfa-c27836bf9e87" containerID="c13fe4c4e6f5bcc236d1c6a1478dbb3af5f7d987f29a4a8af596725cdc00d72d" exitCode=0
Sep 09 20:56:18.311421 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:18.311425    2576 generic.go:356] "Generic (PLEG): container finished" podID="22c8a115-79a9-4fc5-9cfa-c27836bf9e87" containerID="bdbe6c30211e7cd33d952cd3a974d29d9ac3df0147ae05f4b490a50af5f1debd" exitCode=0
Sep 09 20:56:18.311421 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:18.311431    2576 generic.go:356] "Generic (PLEG): container finished" podID="22c8a115-79a9-4fc5-9cfa-c27836bf9e87" containerID="aeecedf2fa0f95cc85b5cadfecc5cf9f8183849fd478f54a8d130bb4aba6500b" exitCode=0
Sep 09 20:56:18.311660 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:18.311435    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"22c8a115-79a9-4fc5-9cfa-c27836bf9e87","Type":"ContainerDied","Data":"3de3ac7298aec2be9a0a8b3bba8eae1f78a01a7ff0eb5ff3f619c1dd37b96785"}
Sep 09 20:56:18.311660 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:18.311494    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"22c8a115-79a9-4fc5-9cfa-c27836bf9e87","Type":"ContainerDied","Data":"c13fe4c4e6f5bcc236d1c6a1478dbb3af5f7d987f29a4a8af596725cdc00d72d"}
Sep 09 20:56:18.311660 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:18.311521    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"22c8a115-79a9-4fc5-9cfa-c27836bf9e87","Type":"ContainerDied","Data":"bdbe6c30211e7cd33d952cd3a974d29d9ac3df0147ae05f4b490a50af5f1debd"}
Sep 09 20:56:18.311660 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:18.311545    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"22c8a115-79a9-4fc5-9cfa-c27836bf9e87","Type":"ContainerDied","Data":"aeecedf2fa0f95cc85b5cadfecc5cf9f8183849fd478f54a8d130bb4aba6500b"}
Sep 09 20:56:19.318722 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:19.318693    2576 generic.go:356] "Generic (PLEG): container finished" podID="22c8a115-79a9-4fc5-9cfa-c27836bf9e87" containerID="cd42518272da483a57eb7077af7cd9725c494414f40b11d87db91f80855e9a0c" exitCode=0
Sep 09 20:56:19.318722 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:19.318718    2576 generic.go:356] "Generic (PLEG): container finished" podID="22c8a115-79a9-4fc5-9cfa-c27836bf9e87" containerID="6db272b15daab520126736cba42eb7748920ee4541bc34d75012999487ae9530" exitCode=0
Sep 09 20:56:19.319113 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:19.318746    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"22c8a115-79a9-4fc5-9cfa-c27836bf9e87","Type":"ContainerDied","Data":"cd42518272da483a57eb7077af7cd9725c494414f40b11d87db91f80855e9a0c"}
Sep 09 20:56:19.319113 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:19.318818    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"22c8a115-79a9-4fc5-9cfa-c27836bf9e87","Type":"ContainerDied","Data":"6db272b15daab520126736cba42eb7748920ee4541bc34d75012999487ae9530"}
Sep 09 20:56:19.346215 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:19.346194    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:56:19.380408 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:19.380355    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/22c8a115-79a9-4fc5-9cfa-c27836bf9e87-secret-alertmanager-kube-rbac-proxy-web\") pod \"22c8a115-79a9-4fc5-9cfa-c27836bf9e87\" (UID: \"22c8a115-79a9-4fc5-9cfa-c27836bf9e87\") "
Sep 09 20:56:19.380408 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:19.380409    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/22c8a115-79a9-4fc5-9cfa-c27836bf9e87-secret-alertmanager-kube-rbac-proxy\") pod \"22c8a115-79a9-4fc5-9cfa-c27836bf9e87\" (UID: \"22c8a115-79a9-4fc5-9cfa-c27836bf9e87\") "
Sep 09 20:56:19.380554 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:19.380430    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/22c8a115-79a9-4fc5-9cfa-c27836bf9e87-web-config\") pod \"22c8a115-79a9-4fc5-9cfa-c27836bf9e87\" (UID: \"22c8a115-79a9-4fc5-9cfa-c27836bf9e87\") "
Sep 09 20:56:19.380554 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:19.380446    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/22c8a115-79a9-4fc5-9cfa-c27836bf9e87-config-out\") pod \"22c8a115-79a9-4fc5-9cfa-c27836bf9e87\" (UID: \"22c8a115-79a9-4fc5-9cfa-c27836bf9e87\") "
Sep 09 20:56:19.380554 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:19.380462    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/22c8a115-79a9-4fc5-9cfa-c27836bf9e87-secret-alertmanager-kube-rbac-proxy-metric\") pod \"22c8a115-79a9-4fc5-9cfa-c27836bf9e87\" (UID: \"22c8a115-79a9-4fc5-9cfa-c27836bf9e87\") "
Sep 09 20:56:19.380554 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:19.380518    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/22c8a115-79a9-4fc5-9cfa-c27836bf9e87-alertmanager-main-db\") pod \"22c8a115-79a9-4fc5-9cfa-c27836bf9e87\" (UID: \"22c8a115-79a9-4fc5-9cfa-c27836bf9e87\") "
Sep 09 20:56:19.380554 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:19.380543    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/22c8a115-79a9-4fc5-9cfa-c27836bf9e87-config-volume\") pod \"22c8a115-79a9-4fc5-9cfa-c27836bf9e87\" (UID: \"22c8a115-79a9-4fc5-9cfa-c27836bf9e87\") "
Sep 09 20:56:19.380797 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:19.380593    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/22c8a115-79a9-4fc5-9cfa-c27836bf9e87-metrics-client-ca\") pod \"22c8a115-79a9-4fc5-9cfa-c27836bf9e87\" (UID: \"22c8a115-79a9-4fc5-9cfa-c27836bf9e87\") "
Sep 09 20:56:19.380797 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:19.380637    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/22c8a115-79a9-4fc5-9cfa-c27836bf9e87-alertmanager-trusted-ca-bundle\") pod \"22c8a115-79a9-4fc5-9cfa-c27836bf9e87\" (UID: \"22c8a115-79a9-4fc5-9cfa-c27836bf9e87\") "
Sep 09 20:56:19.380797 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:19.380666    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/22c8a115-79a9-4fc5-9cfa-c27836bf9e87-tls-assets\") pod \"22c8a115-79a9-4fc5-9cfa-c27836bf9e87\" (UID: \"22c8a115-79a9-4fc5-9cfa-c27836bf9e87\") "
Sep 09 20:56:19.380797 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:19.380695    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/22c8a115-79a9-4fc5-9cfa-c27836bf9e87-cluster-tls-config\") pod \"22c8a115-79a9-4fc5-9cfa-c27836bf9e87\" (UID: \"22c8a115-79a9-4fc5-9cfa-c27836bf9e87\") "
Sep 09 20:56:19.380797 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:19.380753    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/22c8a115-79a9-4fc5-9cfa-c27836bf9e87-secret-alertmanager-main-tls\") pod \"22c8a115-79a9-4fc5-9cfa-c27836bf9e87\" (UID: \"22c8a115-79a9-4fc5-9cfa-c27836bf9e87\") "
Sep 09 20:56:19.380797 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:19.380775    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jp2rj\" (UniqueName: \"kubernetes.io/projected/22c8a115-79a9-4fc5-9cfa-c27836bf9e87-kube-api-access-jp2rj\") pod \"22c8a115-79a9-4fc5-9cfa-c27836bf9e87\" (UID: \"22c8a115-79a9-4fc5-9cfa-c27836bf9e87\") "
Sep 09 20:56:19.381092 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:19.380891    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22c8a115-79a9-4fc5-9cfa-c27836bf9e87-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "22c8a115-79a9-4fc5-9cfa-c27836bf9e87" (UID: "22c8a115-79a9-4fc5-9cfa-c27836bf9e87"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:56:19.381687 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:19.381167    2576 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/22c8a115-79a9-4fc5-9cfa-c27836bf9e87-alertmanager-main-db\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 20:56:19.382038 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:19.382010    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22c8a115-79a9-4fc5-9cfa-c27836bf9e87-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "22c8a115-79a9-4fc5-9cfa-c27836bf9e87" (UID: "22c8a115-79a9-4fc5-9cfa-c27836bf9e87"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:56:19.383404 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:19.383355    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22c8a115-79a9-4fc5-9cfa-c27836bf9e87-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "22c8a115-79a9-4fc5-9cfa-c27836bf9e87" (UID: "22c8a115-79a9-4fc5-9cfa-c27836bf9e87"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:56:19.383708 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:19.383585    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22c8a115-79a9-4fc5-9cfa-c27836bf9e87-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "22c8a115-79a9-4fc5-9cfa-c27836bf9e87" (UID: "22c8a115-79a9-4fc5-9cfa-c27836bf9e87"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:56:19.383708 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:19.383589    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22c8a115-79a9-4fc5-9cfa-c27836bf9e87-config-out" (OuterVolumeSpecName: "config-out") pod "22c8a115-79a9-4fc5-9cfa-c27836bf9e87" (UID: "22c8a115-79a9-4fc5-9cfa-c27836bf9e87"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:56:19.383708 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:19.383681    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22c8a115-79a9-4fc5-9cfa-c27836bf9e87-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "22c8a115-79a9-4fc5-9cfa-c27836bf9e87" (UID: "22c8a115-79a9-4fc5-9cfa-c27836bf9e87"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:56:19.384109 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:19.384081    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22c8a115-79a9-4fc5-9cfa-c27836bf9e87-kube-api-access-jp2rj" (OuterVolumeSpecName: "kube-api-access-jp2rj") pod "22c8a115-79a9-4fc5-9cfa-c27836bf9e87" (UID: "22c8a115-79a9-4fc5-9cfa-c27836bf9e87"). InnerVolumeSpecName "kube-api-access-jp2rj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:56:19.385401 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:19.385283    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22c8a115-79a9-4fc5-9cfa-c27836bf9e87-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "22c8a115-79a9-4fc5-9cfa-c27836bf9e87" (UID: "22c8a115-79a9-4fc5-9cfa-c27836bf9e87"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:56:19.386439 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:19.386407    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22c8a115-79a9-4fc5-9cfa-c27836bf9e87-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "22c8a115-79a9-4fc5-9cfa-c27836bf9e87" (UID: "22c8a115-79a9-4fc5-9cfa-c27836bf9e87"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:56:19.386579 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:19.386458    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22c8a115-79a9-4fc5-9cfa-c27836bf9e87-config-volume" (OuterVolumeSpecName: "config-volume") pod "22c8a115-79a9-4fc5-9cfa-c27836bf9e87" (UID: "22c8a115-79a9-4fc5-9cfa-c27836bf9e87"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:56:19.386705 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:19.386682    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22c8a115-79a9-4fc5-9cfa-c27836bf9e87-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "22c8a115-79a9-4fc5-9cfa-c27836bf9e87" (UID: "22c8a115-79a9-4fc5-9cfa-c27836bf9e87"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:56:19.389793 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:19.389766    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22c8a115-79a9-4fc5-9cfa-c27836bf9e87-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "22c8a115-79a9-4fc5-9cfa-c27836bf9e87" (UID: "22c8a115-79a9-4fc5-9cfa-c27836bf9e87"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:56:19.395750 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:19.395730    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22c8a115-79a9-4fc5-9cfa-c27836bf9e87-web-config" (OuterVolumeSpecName: "web-config") pod "22c8a115-79a9-4fc5-9cfa-c27836bf9e87" (UID: "22c8a115-79a9-4fc5-9cfa-c27836bf9e87"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:56:19.482403 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:19.482308    2576 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/22c8a115-79a9-4fc5-9cfa-c27836bf9e87-secret-alertmanager-main-tls\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 20:56:19.482403 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:19.482336    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jp2rj\" (UniqueName: \"kubernetes.io/projected/22c8a115-79a9-4fc5-9cfa-c27836bf9e87-kube-api-access-jp2rj\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 20:56:19.482403 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:19.482345    2576 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/22c8a115-79a9-4fc5-9cfa-c27836bf9e87-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 20:56:19.482403 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:19.482354    2576 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/22c8a115-79a9-4fc5-9cfa-c27836bf9e87-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 20:56:19.482403 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:19.482383    2576 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/22c8a115-79a9-4fc5-9cfa-c27836bf9e87-web-config\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 20:56:19.482403 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:19.482392    2576 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/22c8a115-79a9-4fc5-9cfa-c27836bf9e87-config-out\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 20:56:19.482403 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:19.482400    2576 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/22c8a115-79a9-4fc5-9cfa-c27836bf9e87-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 20:56:19.482403 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:19.482411    2576 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/22c8a115-79a9-4fc5-9cfa-c27836bf9e87-config-volume\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 20:56:19.482741 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:19.482420    2576 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/22c8a115-79a9-4fc5-9cfa-c27836bf9e87-metrics-client-ca\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 20:56:19.482741 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:19.482428    2576 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/22c8a115-79a9-4fc5-9cfa-c27836bf9e87-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 20:56:19.482741 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:19.482437    2576 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/22c8a115-79a9-4fc5-9cfa-c27836bf9e87-tls-assets\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 20:56:19.482741 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:19.482445    2576 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/22c8a115-79a9-4fc5-9cfa-c27836bf9e87-cluster-tls-config\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 20:56:20.330065 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:20.330027    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"22c8a115-79a9-4fc5-9cfa-c27836bf9e87","Type":"ContainerDied","Data":"d6313c7a1debc4be33946a97459ae4aa89c8d7cba35187a4d83682c4a821612b"}
Sep 09 20:56:20.330440 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:20.330079    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:56:20.330440 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:20.330121    2576 scope.go:117] "RemoveContainer" containerID="3de3ac7298aec2be9a0a8b3bba8eae1f78a01a7ff0eb5ff3f619c1dd37b96785"
Sep 09 20:56:20.337426 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:20.337408    2576 scope.go:117] "RemoveContainer" containerID="cd42518272da483a57eb7077af7cd9725c494414f40b11d87db91f80855e9a0c"
Sep 09 20:56:20.345823 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:20.345804    2576 scope.go:117] "RemoveContainer" containerID="c13fe4c4e6f5bcc236d1c6a1478dbb3af5f7d987f29a4a8af596725cdc00d72d"
Sep 09 20:56:20.352142 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:20.352121    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 20:56:20.353086 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:20.353070    2576 scope.go:117] "RemoveContainer" containerID="6db272b15daab520126736cba42eb7748920ee4541bc34d75012999487ae9530"
Sep 09 20:56:20.360590 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:20.360566    2576 scope.go:117] "RemoveContainer" containerID="bdbe6c30211e7cd33d952cd3a974d29d9ac3df0147ae05f4b490a50af5f1debd"
Sep 09 20:56:20.360834 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:20.360812    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 20:56:20.367499 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:20.367482    2576 scope.go:117] "RemoveContainer" containerID="aeecedf2fa0f95cc85b5cadfecc5cf9f8183849fd478f54a8d130bb4aba6500b"
Sep 09 20:56:20.374171 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:20.374157    2576 scope.go:117] "RemoveContainer" containerID="7050ee4f1aa21ddb004c36fc5ff9affe48a00fcc5effbf2b605229e5261d8081"
Sep 09 20:56:20.400270 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:20.400248    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 20:56:20.400505 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:20.400492    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22c8a115-79a9-4fc5-9cfa-c27836bf9e87" containerName="kube-rbac-proxy"
Sep 09 20:56:20.400563 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:20.400507    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="22c8a115-79a9-4fc5-9cfa-c27836bf9e87" containerName="kube-rbac-proxy"
Sep 09 20:56:20.400563 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:20.400516    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22c8a115-79a9-4fc5-9cfa-c27836bf9e87" containerName="kube-rbac-proxy-metric"
Sep 09 20:56:20.400563 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:20.400522    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="22c8a115-79a9-4fc5-9cfa-c27836bf9e87" containerName="kube-rbac-proxy-metric"
Sep 09 20:56:20.400563 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:20.400533    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22c8a115-79a9-4fc5-9cfa-c27836bf9e87" containerName="kube-rbac-proxy-web"
Sep 09 20:56:20.400563 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:20.400538    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="22c8a115-79a9-4fc5-9cfa-c27836bf9e87" containerName="kube-rbac-proxy-web"
Sep 09 20:56:20.400563 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:20.400547    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22c8a115-79a9-4fc5-9cfa-c27836bf9e87" containerName="prom-label-proxy"
Sep 09 20:56:20.400563 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:20.400553    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="22c8a115-79a9-4fc5-9cfa-c27836bf9e87" containerName="prom-label-proxy"
Sep 09 20:56:20.400563 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:20.400561    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9bef8f5-e612-4504-a259-c2550b05cea2" containerName="console"
Sep 09 20:56:20.400563 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:20.400566    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9bef8f5-e612-4504-a259-c2550b05cea2" containerName="console"
Sep 09 20:56:20.400813 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:20.400577    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22c8a115-79a9-4fc5-9cfa-c27836bf9e87" containerName="config-reloader"
Sep 09 20:56:20.400813 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:20.400582    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="22c8a115-79a9-4fc5-9cfa-c27836bf9e87" containerName="config-reloader"
Sep 09 20:56:20.400813 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:20.400588    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22c8a115-79a9-4fc5-9cfa-c27836bf9e87" containerName="init-config-reloader"
Sep 09 20:56:20.400813 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:20.400593    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="22c8a115-79a9-4fc5-9cfa-c27836bf9e87" containerName="init-config-reloader"
Sep 09 20:56:20.400813 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:20.400599    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22c8a115-79a9-4fc5-9cfa-c27836bf9e87" containerName="alertmanager"
Sep 09 20:56:20.400813 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:20.400603    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="22c8a115-79a9-4fc5-9cfa-c27836bf9e87" containerName="alertmanager"
Sep 09 20:56:20.400813 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:20.400639    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="22c8a115-79a9-4fc5-9cfa-c27836bf9e87" containerName="alertmanager"
Sep 09 20:56:20.400813 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:20.400646    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="22c8a115-79a9-4fc5-9cfa-c27836bf9e87" containerName="kube-rbac-proxy"
Sep 09 20:56:20.400813 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:20.400653    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="22c8a115-79a9-4fc5-9cfa-c27836bf9e87" containerName="prom-label-proxy"
Sep 09 20:56:20.400813 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:20.400657    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9bef8f5-e612-4504-a259-c2550b05cea2" containerName="console"
Sep 09 20:56:20.400813 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:20.400662    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="22c8a115-79a9-4fc5-9cfa-c27836bf9e87" containerName="kube-rbac-proxy-metric"
Sep 09 20:56:20.400813 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:20.400667    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="22c8a115-79a9-4fc5-9cfa-c27836bf9e87" containerName="config-reloader"
Sep 09 20:56:20.400813 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:20.400671    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="22c8a115-79a9-4fc5-9cfa-c27836bf9e87" containerName="kube-rbac-proxy-web"
Sep 09 20:56:20.403817 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:20.403801    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:56:20.406446 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:20.406419    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 09 20:56:20.406446 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:20.406419    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-f4nfl\""
Sep 09 20:56:20.406591 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:20.406487    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 09 20:56:20.406822 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:20.406804    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 09 20:56:20.406860 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:20.406815    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 09 20:56:20.406907 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:20.406843    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 09 20:56:20.406944 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:20.406926    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 09 20:56:20.407176 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:20.407162    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 09 20:56:20.407610 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:20.407594    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 09 20:56:20.412104 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:20.412087    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 09 20:56:20.420566 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:20.420545    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 20:56:20.489161 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:20.489081    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9d8fce6c-74ee-4766-8f8c-c1d4a05337de-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"9d8fce6c-74ee-4766-8f8c-c1d4a05337de\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:56:20.489161 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:20.489118    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9d8fce6c-74ee-4766-8f8c-c1d4a05337de-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"9d8fce6c-74ee-4766-8f8c-c1d4a05337de\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:56:20.489161 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:20.489136    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d8fce6c-74ee-4766-8f8c-c1d4a05337de-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"9d8fce6c-74ee-4766-8f8c-c1d4a05337de\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:56:20.489348 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:20.489190    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9d8fce6c-74ee-4766-8f8c-c1d4a05337de-config-out\") pod \"alertmanager-main-0\" (UID: \"9d8fce6c-74ee-4766-8f8c-c1d4a05337de\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:56:20.489348 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:20.489235    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9d8fce6c-74ee-4766-8f8c-c1d4a05337de-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"9d8fce6c-74ee-4766-8f8c-c1d4a05337de\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:56:20.489348 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:20.489263    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9d8fce6c-74ee-4766-8f8c-c1d4a05337de-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"9d8fce6c-74ee-4766-8f8c-c1d4a05337de\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:56:20.489348 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:20.489292    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9d8fce6c-74ee-4766-8f8c-c1d4a05337de-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"9d8fce6c-74ee-4766-8f8c-c1d4a05337de\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:56:20.489348 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:20.489330    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9d8fce6c-74ee-4766-8f8c-c1d4a05337de-web-config\") pod \"alertmanager-main-0\" (UID: \"9d8fce6c-74ee-4766-8f8c-c1d4a05337de\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:56:20.489348 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:20.489345    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5d5l7\" (UniqueName: \"kubernetes.io/projected/9d8fce6c-74ee-4766-8f8c-c1d4a05337de-kube-api-access-5d5l7\") pod \"alertmanager-main-0\" (UID: \"9d8fce6c-74ee-4766-8f8c-c1d4a05337de\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:56:20.489551 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:20.489391    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9d8fce6c-74ee-4766-8f8c-c1d4a05337de-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"9d8fce6c-74ee-4766-8f8c-c1d4a05337de\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:56:20.489551 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:20.489416    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9d8fce6c-74ee-4766-8f8c-c1d4a05337de-config-volume\") pod \"alertmanager-main-0\" (UID: \"9d8fce6c-74ee-4766-8f8c-c1d4a05337de\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:56:20.489551 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:20.489440    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9d8fce6c-74ee-4766-8f8c-c1d4a05337de-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"9d8fce6c-74ee-4766-8f8c-c1d4a05337de\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:56:20.489551 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:20.489463    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9d8fce6c-74ee-4766-8f8c-c1d4a05337de-tls-assets\") pod \"alertmanager-main-0\" (UID: \"9d8fce6c-74ee-4766-8f8c-c1d4a05337de\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:56:20.590359 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:20.590330    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9d8fce6c-74ee-4766-8f8c-c1d4a05337de-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"9d8fce6c-74ee-4766-8f8c-c1d4a05337de\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:56:20.590482 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:20.590390    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9d8fce6c-74ee-4766-8f8c-c1d4a05337de-web-config\") pod \"alertmanager-main-0\" (UID: \"9d8fce6c-74ee-4766-8f8c-c1d4a05337de\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:56:20.590482 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:20.590412    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5d5l7\" (UniqueName: \"kubernetes.io/projected/9d8fce6c-74ee-4766-8f8c-c1d4a05337de-kube-api-access-5d5l7\") pod \"alertmanager-main-0\" (UID: \"9d8fce6c-74ee-4766-8f8c-c1d4a05337de\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:56:20.590482 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:20.590433    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9d8fce6c-74ee-4766-8f8c-c1d4a05337de-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"9d8fce6c-74ee-4766-8f8c-c1d4a05337de\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:56:20.590482 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:20.590459    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9d8fce6c-74ee-4766-8f8c-c1d4a05337de-config-volume\") pod \"alertmanager-main-0\" (UID: \"9d8fce6c-74ee-4766-8f8c-c1d4a05337de\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:56:20.590666 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:20.590492    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9d8fce6c-74ee-4766-8f8c-c1d4a05337de-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"9d8fce6c-74ee-4766-8f8c-c1d4a05337de\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:56:20.590666 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:20.590518    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9d8fce6c-74ee-4766-8f8c-c1d4a05337de-tls-assets\") pod \"alertmanager-main-0\" (UID: \"9d8fce6c-74ee-4766-8f8c-c1d4a05337de\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:56:20.590666 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:20.590562    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9d8fce6c-74ee-4766-8f8c-c1d4a05337de-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"9d8fce6c-74ee-4766-8f8c-c1d4a05337de\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:56:20.590666 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:20.590594    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9d8fce6c-74ee-4766-8f8c-c1d4a05337de-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"9d8fce6c-74ee-4766-8f8c-c1d4a05337de\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:56:20.590666 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:20.590614    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d8fce6c-74ee-4766-8f8c-c1d4a05337de-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"9d8fce6c-74ee-4766-8f8c-c1d4a05337de\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:56:20.590666 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:20.590638    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9d8fce6c-74ee-4766-8f8c-c1d4a05337de-config-out\") pod \"alertmanager-main-0\" (UID: \"9d8fce6c-74ee-4766-8f8c-c1d4a05337de\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:56:20.590666 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:20.590663    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9d8fce6c-74ee-4766-8f8c-c1d4a05337de-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"9d8fce6c-74ee-4766-8f8c-c1d4a05337de\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:56:20.591002 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:20.590692    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9d8fce6c-74ee-4766-8f8c-c1d4a05337de-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"9d8fce6c-74ee-4766-8f8c-c1d4a05337de\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:56:20.591797 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:20.591483    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9d8fce6c-74ee-4766-8f8c-c1d4a05337de-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"9d8fce6c-74ee-4766-8f8c-c1d4a05337de\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:56:20.591797 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:20.591743    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d8fce6c-74ee-4766-8f8c-c1d4a05337de-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"9d8fce6c-74ee-4766-8f8c-c1d4a05337de\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:56:20.593935 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:20.593745    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9d8fce6c-74ee-4766-8f8c-c1d4a05337de-tls-assets\") pod \"alertmanager-main-0\" (UID: \"9d8fce6c-74ee-4766-8f8c-c1d4a05337de\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:56:20.593935 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:20.593749    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9d8fce6c-74ee-4766-8f8c-c1d4a05337de-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"9d8fce6c-74ee-4766-8f8c-c1d4a05337de\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:56:20.593935 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:20.593859    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9d8fce6c-74ee-4766-8f8c-c1d4a05337de-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"9d8fce6c-74ee-4766-8f8c-c1d4a05337de\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:56:20.593935 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:20.593898    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9d8fce6c-74ee-4766-8f8c-c1d4a05337de-web-config\") pod \"alertmanager-main-0\" (UID: \"9d8fce6c-74ee-4766-8f8c-c1d4a05337de\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:56:20.594156 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:20.594040    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9d8fce6c-74ee-4766-8f8c-c1d4a05337de-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"9d8fce6c-74ee-4766-8f8c-c1d4a05337de\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:56:20.594156 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:20.594087    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9d8fce6c-74ee-4766-8f8c-c1d4a05337de-config-out\") pod \"alertmanager-main-0\" (UID: \"9d8fce6c-74ee-4766-8f8c-c1d4a05337de\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:56:20.594419 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:20.594402    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9d8fce6c-74ee-4766-8f8c-c1d4a05337de-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"9d8fce6c-74ee-4766-8f8c-c1d4a05337de\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:56:20.594494 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:20.594406    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9d8fce6c-74ee-4766-8f8c-c1d4a05337de-config-volume\") pod \"alertmanager-main-0\" (UID: \"9d8fce6c-74ee-4766-8f8c-c1d4a05337de\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:56:20.594531 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:20.594503    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9d8fce6c-74ee-4766-8f8c-c1d4a05337de-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"9d8fce6c-74ee-4766-8f8c-c1d4a05337de\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:56:20.595811 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:20.595794    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9d8fce6c-74ee-4766-8f8c-c1d4a05337de-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"9d8fce6c-74ee-4766-8f8c-c1d4a05337de\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:56:20.598901 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:20.598886    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5d5l7\" (UniqueName: \"kubernetes.io/projected/9d8fce6c-74ee-4766-8f8c-c1d4a05337de-kube-api-access-5d5l7\") pod \"alertmanager-main-0\" (UID: \"9d8fce6c-74ee-4766-8f8c-c1d4a05337de\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:56:20.712736 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:20.712702    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:56:20.992991 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:20.992955    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 20:56:20.996285 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:56:20.996258    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d8fce6c_74ee_4766_8f8c_c1d4a05337de.slice/crio-edcf64f3d32e0a56484767477adb085f59c4b2cfcf2cb859b7155d6bb8ec2491 WatchSource:0}: Error finding container edcf64f3d32e0a56484767477adb085f59c4b2cfcf2cb859b7155d6bb8ec2491: Status 404 returned error can't find the container with id edcf64f3d32e0a56484767477adb085f59c4b2cfcf2cb859b7155d6bb8ec2491
Sep 09 20:56:21.335142 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:21.335045    2576 generic.go:356] "Generic (PLEG): container finished" podID="9d8fce6c-74ee-4766-8f8c-c1d4a05337de" containerID="de009ea466593417efe2204e30fe5626c77f6efc37af07244b97e03cb92ca8db" exitCode=0
Sep 09 20:56:21.335534 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:21.335135    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9d8fce6c-74ee-4766-8f8c-c1d4a05337de","Type":"ContainerDied","Data":"de009ea466593417efe2204e30fe5626c77f6efc37af07244b97e03cb92ca8db"}
Sep 09 20:56:21.335534 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:21.335202    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9d8fce6c-74ee-4766-8f8c-c1d4a05337de","Type":"ContainerStarted","Data":"edcf64f3d32e0a56484767477adb085f59c4b2cfcf2cb859b7155d6bb8ec2491"}
Sep 09 20:56:21.408718 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:21.408691    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="22c8a115-79a9-4fc5-9cfa-c27836bf9e87" path="/var/lib/kubelet/pods/22c8a115-79a9-4fc5-9cfa-c27836bf9e87/volumes"
Sep 09 20:56:22.123809 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:22.123777    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-6c5555b889-cjqvq"]
Sep 09 20:56:22.129182 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:22.126516    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6c5555b889-cjqvq"
Sep 09 20:56:22.129828 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:22.129803    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 09 20:56:22.129958 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:22.129813    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 09 20:56:22.129958 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:22.129871    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-phr98\""
Sep 09 20:56:22.129958 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:22.129882    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 09 20:56:22.129958 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:22.129887    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 09 20:56:22.130335 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:22.130319    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 09 20:56:22.136041 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:22.136018    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 09 20:56:22.141175 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:22.141155    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6c5555b889-cjqvq"]
Sep 09 20:56:22.202515 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:22.202481    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c40f0ae4-b1ca-4c1e-b2b8-c6c122c185dc-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6c5555b889-cjqvq\" (UID: \"c40f0ae4-b1ca-4c1e-b2b8-c6c122c185dc\") " pod="openshift-monitoring/telemeter-client-6c5555b889-cjqvq"
Sep 09 20:56:22.202647 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:22.202523    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c40f0ae4-b1ca-4c1e-b2b8-c6c122c185dc-serving-certs-ca-bundle\") pod \"telemeter-client-6c5555b889-cjqvq\" (UID: \"c40f0ae4-b1ca-4c1e-b2b8-c6c122c185dc\") " pod="openshift-monitoring/telemeter-client-6c5555b889-cjqvq"
Sep 09 20:56:22.202647 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:22.202555    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/c40f0ae4-b1ca-4c1e-b2b8-c6c122c185dc-secret-telemeter-client\") pod \"telemeter-client-6c5555b889-cjqvq\" (UID: \"c40f0ae4-b1ca-4c1e-b2b8-c6c122c185dc\") " pod="openshift-monitoring/telemeter-client-6c5555b889-cjqvq"
Sep 09 20:56:22.202647 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:22.202584    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/c40f0ae4-b1ca-4c1e-b2b8-c6c122c185dc-federate-client-tls\") pod \"telemeter-client-6c5555b889-cjqvq\" (UID: \"c40f0ae4-b1ca-4c1e-b2b8-c6c122c185dc\") " pod="openshift-monitoring/telemeter-client-6c5555b889-cjqvq"
Sep 09 20:56:22.202647 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:22.202624    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c40f0ae4-b1ca-4c1e-b2b8-c6c122c185dc-metrics-client-ca\") pod \"telemeter-client-6c5555b889-cjqvq\" (UID: \"c40f0ae4-b1ca-4c1e-b2b8-c6c122c185dc\") " pod="openshift-monitoring/telemeter-client-6c5555b889-cjqvq"
Sep 09 20:56:22.202784 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:22.202651    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9fxq\" (UniqueName: \"kubernetes.io/projected/c40f0ae4-b1ca-4c1e-b2b8-c6c122c185dc-kube-api-access-z9fxq\") pod \"telemeter-client-6c5555b889-cjqvq\" (UID: \"c40f0ae4-b1ca-4c1e-b2b8-c6c122c185dc\") " pod="openshift-monitoring/telemeter-client-6c5555b889-cjqvq"
Sep 09 20:56:22.202784 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:22.202674    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c40f0ae4-b1ca-4c1e-b2b8-c6c122c185dc-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6c5555b889-cjqvq\" (UID: \"c40f0ae4-b1ca-4c1e-b2b8-c6c122c185dc\") " pod="openshift-monitoring/telemeter-client-6c5555b889-cjqvq"
Sep 09 20:56:22.202784 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:22.202699    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/c40f0ae4-b1ca-4c1e-b2b8-c6c122c185dc-telemeter-client-tls\") pod \"telemeter-client-6c5555b889-cjqvq\" (UID: \"c40f0ae4-b1ca-4c1e-b2b8-c6c122c185dc\") " pod="openshift-monitoring/telemeter-client-6c5555b889-cjqvq"
Sep 09 20:56:22.303699 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:22.303660    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c40f0ae4-b1ca-4c1e-b2b8-c6c122c185dc-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6c5555b889-cjqvq\" (UID: \"c40f0ae4-b1ca-4c1e-b2b8-c6c122c185dc\") " pod="openshift-monitoring/telemeter-client-6c5555b889-cjqvq"
Sep 09 20:56:22.303699 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:22.303706    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c40f0ae4-b1ca-4c1e-b2b8-c6c122c185dc-serving-certs-ca-bundle\") pod \"telemeter-client-6c5555b889-cjqvq\" (UID: \"c40f0ae4-b1ca-4c1e-b2b8-c6c122c185dc\") " pod="openshift-monitoring/telemeter-client-6c5555b889-cjqvq"
Sep 09 20:56:22.303840 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:22.303737    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/c40f0ae4-b1ca-4c1e-b2b8-c6c122c185dc-secret-telemeter-client\") pod \"telemeter-client-6c5555b889-cjqvq\" (UID: \"c40f0ae4-b1ca-4c1e-b2b8-c6c122c185dc\") " pod="openshift-monitoring/telemeter-client-6c5555b889-cjqvq"
Sep 09 20:56:22.303840 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:22.303759    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/c40f0ae4-b1ca-4c1e-b2b8-c6c122c185dc-federate-client-tls\") pod \"telemeter-client-6c5555b889-cjqvq\" (UID: \"c40f0ae4-b1ca-4c1e-b2b8-c6c122c185dc\") " pod="openshift-monitoring/telemeter-client-6c5555b889-cjqvq"
Sep 09 20:56:22.303840 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:22.303789    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c40f0ae4-b1ca-4c1e-b2b8-c6c122c185dc-metrics-client-ca\") pod \"telemeter-client-6c5555b889-cjqvq\" (UID: \"c40f0ae4-b1ca-4c1e-b2b8-c6c122c185dc\") " pod="openshift-monitoring/telemeter-client-6c5555b889-cjqvq"
Sep 09 20:56:22.303840 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:22.303824    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9fxq\" (UniqueName: \"kubernetes.io/projected/c40f0ae4-b1ca-4c1e-b2b8-c6c122c185dc-kube-api-access-z9fxq\") pod \"telemeter-client-6c5555b889-cjqvq\" (UID: \"c40f0ae4-b1ca-4c1e-b2b8-c6c122c185dc\") " pod="openshift-monitoring/telemeter-client-6c5555b889-cjqvq"
Sep 09 20:56:22.304021 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:22.303855    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c40f0ae4-b1ca-4c1e-b2b8-c6c122c185dc-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6c5555b889-cjqvq\" (UID: \"c40f0ae4-b1ca-4c1e-b2b8-c6c122c185dc\") " pod="openshift-monitoring/telemeter-client-6c5555b889-cjqvq"
Sep 09 20:56:22.304021 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:22.303894    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/c40f0ae4-b1ca-4c1e-b2b8-c6c122c185dc-telemeter-client-tls\") pod \"telemeter-client-6c5555b889-cjqvq\" (UID: \"c40f0ae4-b1ca-4c1e-b2b8-c6c122c185dc\") " pod="openshift-monitoring/telemeter-client-6c5555b889-cjqvq"
Sep 09 20:56:22.304538 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:22.304512    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c40f0ae4-b1ca-4c1e-b2b8-c6c122c185dc-serving-certs-ca-bundle\") pod \"telemeter-client-6c5555b889-cjqvq\" (UID: \"c40f0ae4-b1ca-4c1e-b2b8-c6c122c185dc\") " pod="openshift-monitoring/telemeter-client-6c5555b889-cjqvq"
Sep 09 20:56:22.304801 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:22.304578    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c40f0ae4-b1ca-4c1e-b2b8-c6c122c185dc-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6c5555b889-cjqvq\" (UID: \"c40f0ae4-b1ca-4c1e-b2b8-c6c122c185dc\") " pod="openshift-monitoring/telemeter-client-6c5555b889-cjqvq"
Sep 09 20:56:22.304977 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:22.304955    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c40f0ae4-b1ca-4c1e-b2b8-c6c122c185dc-metrics-client-ca\") pod \"telemeter-client-6c5555b889-cjqvq\" (UID: \"c40f0ae4-b1ca-4c1e-b2b8-c6c122c185dc\") " pod="openshift-monitoring/telemeter-client-6c5555b889-cjqvq"
Sep 09 20:56:22.306655 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:22.306628    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c40f0ae4-b1ca-4c1e-b2b8-c6c122c185dc-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6c5555b889-cjqvq\" (UID: \"c40f0ae4-b1ca-4c1e-b2b8-c6c122c185dc\") " pod="openshift-monitoring/telemeter-client-6c5555b889-cjqvq"
Sep 09 20:56:22.306755 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:22.306635    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/c40f0ae4-b1ca-4c1e-b2b8-c6c122c185dc-telemeter-client-tls\") pod \"telemeter-client-6c5555b889-cjqvq\" (UID: \"c40f0ae4-b1ca-4c1e-b2b8-c6c122c185dc\") " pod="openshift-monitoring/telemeter-client-6c5555b889-cjqvq"
Sep 09 20:56:22.306820 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:22.306798    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/c40f0ae4-b1ca-4c1e-b2b8-c6c122c185dc-secret-telemeter-client\") pod \"telemeter-client-6c5555b889-cjqvq\" (UID: \"c40f0ae4-b1ca-4c1e-b2b8-c6c122c185dc\") " pod="openshift-monitoring/telemeter-client-6c5555b889-cjqvq"
Sep 09 20:56:22.306879 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:22.306803    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/c40f0ae4-b1ca-4c1e-b2b8-c6c122c185dc-federate-client-tls\") pod \"telemeter-client-6c5555b889-cjqvq\" (UID: \"c40f0ae4-b1ca-4c1e-b2b8-c6c122c185dc\") " pod="openshift-monitoring/telemeter-client-6c5555b889-cjqvq"
Sep 09 20:56:22.312934 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:22.312914    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9fxq\" (UniqueName: \"kubernetes.io/projected/c40f0ae4-b1ca-4c1e-b2b8-c6c122c185dc-kube-api-access-z9fxq\") pod \"telemeter-client-6c5555b889-cjqvq\" (UID: \"c40f0ae4-b1ca-4c1e-b2b8-c6c122c185dc\") " pod="openshift-monitoring/telemeter-client-6c5555b889-cjqvq"
Sep 09 20:56:22.343220 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:22.343189    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9d8fce6c-74ee-4766-8f8c-c1d4a05337de","Type":"ContainerStarted","Data":"9a08aad21e54441cfdd832cf3e9e8397e291e3f9ebc3778c235483d8d66fdcbf"}
Sep 09 20:56:22.343649 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:22.343266    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9d8fce6c-74ee-4766-8f8c-c1d4a05337de","Type":"ContainerStarted","Data":"8d2d3cd4d8b4e411ac144811ac0641089c2591ab80ace1b5f9318c94e103e870"}
Sep 09 20:56:22.343649 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:22.343311    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9d8fce6c-74ee-4766-8f8c-c1d4a05337de","Type":"ContainerStarted","Data":"0a6a5ab91f73b4d77bb68c8ee2179394cec91677959f02526d74aa7aba10d9ec"}
Sep 09 20:56:22.343649 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:22.343355    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9d8fce6c-74ee-4766-8f8c-c1d4a05337de","Type":"ContainerStarted","Data":"76a8b5854971cdec3404407ae932ceade06a7fb2d5c46e97e34192a8f030f0cb"}
Sep 09 20:56:22.343649 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:22.343425    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9d8fce6c-74ee-4766-8f8c-c1d4a05337de","Type":"ContainerStarted","Data":"88c02894e374e2e2e570fcc098975ef7618733afeb13555f01f8c097e52c5234"}
Sep 09 20:56:22.343649 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:22.343467    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9d8fce6c-74ee-4766-8f8c-c1d4a05337de","Type":"ContainerStarted","Data":"d08c1d0d3b4774902574f4ec2e44aeb921c31b0d8d97a01d2b70fe66cbd5c7d3"}
Sep 09 20:56:22.372705 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:22.372659    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.353060421 podStartE2EDuration="2.372643813s" totalImagesPullingTime="0s" totalInitContainerRuntime="19.583392ms" isStatefulPod=false podCreationTimestamp="2026-09-09 20:56:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:56:22.370825473 +0000 UTC m=+225.484511332" watchObservedRunningTime="2026-09-09 20:56:22.372643813 +0000 UTC m=+225.486329684"
Sep 09 20:56:22.438189 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:22.438103    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6c5555b889-cjqvq"
Sep 09 20:56:22.730777 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:22.730753    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6c5555b889-cjqvq"]
Sep 09 20:56:22.732787 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:56:22.732759    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc40f0ae4_b1ca_4c1e_b2b8_c6c122c185dc.slice/crio-fe453a0fa18da3b3f1495af13934fed55eda1c09497e3230bd966ebb989ec509 WatchSource:0}: Error finding container fe453a0fa18da3b3f1495af13934fed55eda1c09497e3230bd966ebb989ec509: Status 404 returned error can't find the container with id fe453a0fa18da3b3f1495af13934fed55eda1c09497e3230bd966ebb989ec509
Sep 09 20:56:23.349593 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:23.349559    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6c5555b889-cjqvq" event={"ID":"c40f0ae4-b1ca-4c1e-b2b8-c6c122c185dc","Type":"ContainerStarted","Data":"fe453a0fa18da3b3f1495af13934fed55eda1c09497e3230bd966ebb989ec509"}
Sep 09 20:56:25.361360 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:25.361323    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6c5555b889-cjqvq" event={"ID":"c40f0ae4-b1ca-4c1e-b2b8-c6c122c185dc","Type":"ContainerStarted","Data":"09c51e813a045ef817decf2882c930ea7777a97c315651275f223e0164ad3572"}
Sep 09 20:56:25.361851 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:25.361393    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6c5555b889-cjqvq" event={"ID":"c40f0ae4-b1ca-4c1e-b2b8-c6c122c185dc","Type":"ContainerStarted","Data":"2a73d9a0e9b95d7208cb75a48273b9d2e57a5e88cc2cbe78ab162ca96c356579"}
Sep 09 20:56:25.361851 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:25.361412    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6c5555b889-cjqvq" event={"ID":"c40f0ae4-b1ca-4c1e-b2b8-c6c122c185dc","Type":"ContainerStarted","Data":"00a65f516f36aefa45c35d30a4f3a06197b16c4654cfa425582f9ccc726f9c78"}
Sep 09 20:56:25.391549 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:25.391498    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-6c5555b889-cjqvq" podStartSLOduration=1.5753394539999999 podStartE2EDuration="3.391482648s" totalImagesPullingTime="1.816143194s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:56:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:56:25.390315074 +0000 UTC m=+228.504000943" watchObservedRunningTime="2026-09-09 20:56:25.391482648 +0000 UTC m=+228.505168506"
Sep 09 20:56:25.489522 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:25.489483    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-b78f8dfc8-4brqr"]
Sep 09 20:56:25.491564 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:25.491543    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b78f8dfc8-4brqr"
Sep 09 20:56:25.509221 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:25.509199    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-b78f8dfc8-4brqr"]
Sep 09 20:56:25.565676 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:25.565642    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:56:25.625511 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:25.625432    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:56:25.632348 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:25.632320    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e65af6f5-ae07-4b83-b79d-53f2bddc8947-console-config\") pod \"console-b78f8dfc8-4brqr\" (UID: \"e65af6f5-ae07-4b83-b79d-53f2bddc8947\") " pod="openshift-console/console-b78f8dfc8-4brqr"
Sep 09 20:56:25.632510 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:25.632380    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e65af6f5-ae07-4b83-b79d-53f2bddc8947-console-oauth-config\") pod \"console-b78f8dfc8-4brqr\" (UID: \"e65af6f5-ae07-4b83-b79d-53f2bddc8947\") " pod="openshift-console/console-b78f8dfc8-4brqr"
Sep 09 20:56:25.632510 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:25.632443    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e65af6f5-ae07-4b83-b79d-53f2bddc8947-oauth-serving-cert\") pod \"console-b78f8dfc8-4brqr\" (UID: \"e65af6f5-ae07-4b83-b79d-53f2bddc8947\") " pod="openshift-console/console-b78f8dfc8-4brqr"
Sep 09 20:56:25.632625 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:25.632513    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e65af6f5-ae07-4b83-b79d-53f2bddc8947-trusted-ca-bundle\") pod \"console-b78f8dfc8-4brqr\" (UID: \"e65af6f5-ae07-4b83-b79d-53f2bddc8947\") " pod="openshift-console/console-b78f8dfc8-4brqr"
Sep 09 20:56:25.632625 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:25.632572    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpjzk\" (UniqueName: \"kubernetes.io/projected/e65af6f5-ae07-4b83-b79d-53f2bddc8947-kube-api-access-cpjzk\") pod \"console-b78f8dfc8-4brqr\" (UID: \"e65af6f5-ae07-4b83-b79d-53f2bddc8947\") " pod="openshift-console/console-b78f8dfc8-4brqr"
Sep 09 20:56:25.632625 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:25.632617    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e65af6f5-ae07-4b83-b79d-53f2bddc8947-console-serving-cert\") pod \"console-b78f8dfc8-4brqr\" (UID: \"e65af6f5-ae07-4b83-b79d-53f2bddc8947\") " pod="openshift-console/console-b78f8dfc8-4brqr"
Sep 09 20:56:25.632781 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:25.632637    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e65af6f5-ae07-4b83-b79d-53f2bddc8947-service-ca\") pod \"console-b78f8dfc8-4brqr\" (UID: \"e65af6f5-ae07-4b83-b79d-53f2bddc8947\") " pod="openshift-console/console-b78f8dfc8-4brqr"
Sep 09 20:56:25.699606 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:25.699563    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:56:25.733795 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:25.733768    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e65af6f5-ae07-4b83-b79d-53f2bddc8947-console-oauth-config\") pod \"console-b78f8dfc8-4brqr\" (UID: \"e65af6f5-ae07-4b83-b79d-53f2bddc8947\") " pod="openshift-console/console-b78f8dfc8-4brqr"
Sep 09 20:56:25.733903 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:25.733806    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e65af6f5-ae07-4b83-b79d-53f2bddc8947-oauth-serving-cert\") pod \"console-b78f8dfc8-4brqr\" (UID: \"e65af6f5-ae07-4b83-b79d-53f2bddc8947\") " pod="openshift-console/console-b78f8dfc8-4brqr"
Sep 09 20:56:25.733947 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:25.733918    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e65af6f5-ae07-4b83-b79d-53f2bddc8947-trusted-ca-bundle\") pod \"console-b78f8dfc8-4brqr\" (UID: \"e65af6f5-ae07-4b83-b79d-53f2bddc8947\") " pod="openshift-console/console-b78f8dfc8-4brqr"
Sep 09 20:56:25.733999 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:25.733970    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpjzk\" (UniqueName: \"kubernetes.io/projected/e65af6f5-ae07-4b83-b79d-53f2bddc8947-kube-api-access-cpjzk\") pod \"console-b78f8dfc8-4brqr\" (UID: \"e65af6f5-ae07-4b83-b79d-53f2bddc8947\") " pod="openshift-console/console-b78f8dfc8-4brqr"
Sep 09 20:56:25.734051 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:25.734016    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e65af6f5-ae07-4b83-b79d-53f2bddc8947-console-serving-cert\") pod \"console-b78f8dfc8-4brqr\" (UID: \"e65af6f5-ae07-4b83-b79d-53f2bddc8947\") " pod="openshift-console/console-b78f8dfc8-4brqr"
Sep 09 20:56:25.734051 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:25.734036    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e65af6f5-ae07-4b83-b79d-53f2bddc8947-service-ca\") pod \"console-b78f8dfc8-4brqr\" (UID: \"e65af6f5-ae07-4b83-b79d-53f2bddc8947\") " pod="openshift-console/console-b78f8dfc8-4brqr"
Sep 09 20:56:25.734159 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:25.734067    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e65af6f5-ae07-4b83-b79d-53f2bddc8947-console-config\") pod \"console-b78f8dfc8-4brqr\" (UID: \"e65af6f5-ae07-4b83-b79d-53f2bddc8947\") " pod="openshift-console/console-b78f8dfc8-4brqr"
Sep 09 20:56:25.734533 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:25.734509    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e65af6f5-ae07-4b83-b79d-53f2bddc8947-oauth-serving-cert\") pod \"console-b78f8dfc8-4brqr\" (UID: \"e65af6f5-ae07-4b83-b79d-53f2bddc8947\") " pod="openshift-console/console-b78f8dfc8-4brqr"
Sep 09 20:56:25.734952 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:25.734928    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e65af6f5-ae07-4b83-b79d-53f2bddc8947-trusted-ca-bundle\") pod \"console-b78f8dfc8-4brqr\" (UID: \"e65af6f5-ae07-4b83-b79d-53f2bddc8947\") " pod="openshift-console/console-b78f8dfc8-4brqr"
Sep 09 20:56:25.734952 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:25.734941    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e65af6f5-ae07-4b83-b79d-53f2bddc8947-service-ca\") pod \"console-b78f8dfc8-4brqr\" (UID: \"e65af6f5-ae07-4b83-b79d-53f2bddc8947\") " pod="openshift-console/console-b78f8dfc8-4brqr"
Sep 09 20:56:25.735058 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:25.734971    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e65af6f5-ae07-4b83-b79d-53f2bddc8947-console-config\") pod \"console-b78f8dfc8-4brqr\" (UID: \"e65af6f5-ae07-4b83-b79d-53f2bddc8947\") " pod="openshift-console/console-b78f8dfc8-4brqr"
Sep 09 20:56:25.736424 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:25.736403    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e65af6f5-ae07-4b83-b79d-53f2bddc8947-console-oauth-config\") pod \"console-b78f8dfc8-4brqr\" (UID: \"e65af6f5-ae07-4b83-b79d-53f2bddc8947\") " pod="openshift-console/console-b78f8dfc8-4brqr"
Sep 09 20:56:25.736586 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:25.736571    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e65af6f5-ae07-4b83-b79d-53f2bddc8947-console-serving-cert\") pod \"console-b78f8dfc8-4brqr\" (UID: \"e65af6f5-ae07-4b83-b79d-53f2bddc8947\") " pod="openshift-console/console-b78f8dfc8-4brqr"
Sep 09 20:56:25.744451 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:25.744428    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpjzk\" (UniqueName: \"kubernetes.io/projected/e65af6f5-ae07-4b83-b79d-53f2bddc8947-kube-api-access-cpjzk\") pod \"console-b78f8dfc8-4brqr\" (UID: \"e65af6f5-ae07-4b83-b79d-53f2bddc8947\") " pod="openshift-console/console-b78f8dfc8-4brqr"
Sep 09 20:56:25.800260 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:25.800232    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b78f8dfc8-4brqr"
Sep 09 20:56:26.084470 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:26.084445    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-b78f8dfc8-4brqr"]
Sep 09 20:56:26.086980 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:56:26.086953    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode65af6f5_ae07_4b83_b79d_53f2bddc8947.slice/crio-fb67932171da3c41940d5e5e1273e73e5efb366b0944fb117acd1266a2e72a38 WatchSource:0}: Error finding container fb67932171da3c41940d5e5e1273e73e5efb366b0944fb117acd1266a2e72a38: Status 404 returned error can't find the container with id fb67932171da3c41940d5e5e1273e73e5efb366b0944fb117acd1266a2e72a38
Sep 09 20:56:26.301997 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:26.301962    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:56:26.366442 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:26.366398    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b78f8dfc8-4brqr" event={"ID":"e65af6f5-ae07-4b83-b79d-53f2bddc8947","Type":"ContainerStarted","Data":"f97ad6ffea1fdf1671ee943eac4107c0178bd502859fbb43dd899baad78a3ded"}
Sep 09 20:56:26.366904 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:26.366463    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b78f8dfc8-4brqr" event={"ID":"e65af6f5-ae07-4b83-b79d-53f2bddc8947","Type":"ContainerStarted","Data":"fb67932171da3c41940d5e5e1273e73e5efb366b0944fb117acd1266a2e72a38"}
Sep 09 20:56:26.378741 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:26.378688    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:56:26.391727 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:26.391683    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-b78f8dfc8-4brqr" podStartSLOduration=1.3916674549999999 podStartE2EDuration="1.391667455s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:56:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:56:26.390727079 +0000 UTC m=+229.504412938" watchObservedRunningTime="2026-09-09 20:56:26.391667455 +0000 UTC m=+229.505353314"
Sep 09 20:56:35.801228 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:35.801181    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-b78f8dfc8-4brqr"
Sep 09 20:56:35.801641 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:35.801279    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-b78f8dfc8-4brqr"
Sep 09 20:56:35.805945 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:35.805923    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-b78f8dfc8-4brqr"
Sep 09 20:56:36.406256 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:36.406227    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-b78f8dfc8-4brqr"
Sep 09 20:56:36.455801 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:36.455766    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-74dffcb5f7-84lks"]
Sep 09 20:56:36.566850 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:36.566818    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:56:36.668137 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:36.668046    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:56:36.728594 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:56:36.728566    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:57:01.476030 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:57:01.475945    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-74dffcb5f7-84lks" podUID="55572f58-18d5-4dc3-b3d0-483f96fc9564" containerName="console" containerID="cri-o://878367a807ac6565e8e0aa9a332f33df6ec2dff0831169c31baa7e5c8c8635c7" gracePeriod=15
Sep 09 20:57:01.712676 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:57:01.712653    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74dffcb5f7-84lks"
Sep 09 20:57:01.820558 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:57:01.820461    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/55572f58-18d5-4dc3-b3d0-483f96fc9564-service-ca\") pod \"55572f58-18d5-4dc3-b3d0-483f96fc9564\" (UID: \"55572f58-18d5-4dc3-b3d0-483f96fc9564\") "
Sep 09 20:57:01.820558 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:57:01.820505    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/55572f58-18d5-4dc3-b3d0-483f96fc9564-oauth-serving-cert\") pod \"55572f58-18d5-4dc3-b3d0-483f96fc9564\" (UID: \"55572f58-18d5-4dc3-b3d0-483f96fc9564\") "
Sep 09 20:57:01.820558 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:57:01.820556    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/55572f58-18d5-4dc3-b3d0-483f96fc9564-console-config\") pod \"55572f58-18d5-4dc3-b3d0-483f96fc9564\" (UID: \"55572f58-18d5-4dc3-b3d0-483f96fc9564\") "
Sep 09 20:57:01.820840 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:57:01.820614    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rzjg8\" (UniqueName: \"kubernetes.io/projected/55572f58-18d5-4dc3-b3d0-483f96fc9564-kube-api-access-rzjg8\") pod \"55572f58-18d5-4dc3-b3d0-483f96fc9564\" (UID: \"55572f58-18d5-4dc3-b3d0-483f96fc9564\") "
Sep 09 20:57:01.820840 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:57:01.820760    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/55572f58-18d5-4dc3-b3d0-483f96fc9564-trusted-ca-bundle\") pod \"55572f58-18d5-4dc3-b3d0-483f96fc9564\" (UID: \"55572f58-18d5-4dc3-b3d0-483f96fc9564\") "
Sep 09 20:57:01.820840 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:57:01.820806    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/55572f58-18d5-4dc3-b3d0-483f96fc9564-console-serving-cert\") pod \"55572f58-18d5-4dc3-b3d0-483f96fc9564\" (UID: \"55572f58-18d5-4dc3-b3d0-483f96fc9564\") "
Sep 09 20:57:01.820840 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:57:01.820835    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/55572f58-18d5-4dc3-b3d0-483f96fc9564-console-oauth-config\") pod \"55572f58-18d5-4dc3-b3d0-483f96fc9564\" (UID: \"55572f58-18d5-4dc3-b3d0-483f96fc9564\") "
Sep 09 20:57:01.821033 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:57:01.820884    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55572f58-18d5-4dc3-b3d0-483f96fc9564-service-ca" (OuterVolumeSpecName: "service-ca") pod "55572f58-18d5-4dc3-b3d0-483f96fc9564" (UID: "55572f58-18d5-4dc3-b3d0-483f96fc9564"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:57:01.821033 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:57:01.820949    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55572f58-18d5-4dc3-b3d0-483f96fc9564-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "55572f58-18d5-4dc3-b3d0-483f96fc9564" (UID: "55572f58-18d5-4dc3-b3d0-483f96fc9564"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:57:01.821131 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:57:01.820965    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55572f58-18d5-4dc3-b3d0-483f96fc9564-console-config" (OuterVolumeSpecName: "console-config") pod "55572f58-18d5-4dc3-b3d0-483f96fc9564" (UID: "55572f58-18d5-4dc3-b3d0-483f96fc9564"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:57:01.821180 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:57:01.821159    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55572f58-18d5-4dc3-b3d0-483f96fc9564-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "55572f58-18d5-4dc3-b3d0-483f96fc9564" (UID: "55572f58-18d5-4dc3-b3d0-483f96fc9564"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:57:01.821232 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:57:01.821191    2576 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/55572f58-18d5-4dc3-b3d0-483f96fc9564-service-ca\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 20:57:01.821232 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:57:01.821208    2576 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/55572f58-18d5-4dc3-b3d0-483f96fc9564-oauth-serving-cert\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 20:57:01.821232 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:57:01.821217    2576 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/55572f58-18d5-4dc3-b3d0-483f96fc9564-console-config\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 20:57:01.822920 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:57:01.822899    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55572f58-18d5-4dc3-b3d0-483f96fc9564-kube-api-access-rzjg8" (OuterVolumeSpecName: "kube-api-access-rzjg8") pod "55572f58-18d5-4dc3-b3d0-483f96fc9564" (UID: "55572f58-18d5-4dc3-b3d0-483f96fc9564"). InnerVolumeSpecName "kube-api-access-rzjg8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:57:01.823006 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:57:01.822943    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55572f58-18d5-4dc3-b3d0-483f96fc9564-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "55572f58-18d5-4dc3-b3d0-483f96fc9564" (UID: "55572f58-18d5-4dc3-b3d0-483f96fc9564"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:57:01.823061 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:57:01.823045    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55572f58-18d5-4dc3-b3d0-483f96fc9564-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "55572f58-18d5-4dc3-b3d0-483f96fc9564" (UID: "55572f58-18d5-4dc3-b3d0-483f96fc9564"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:57:01.921603 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:57:01.921564    2576 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/55572f58-18d5-4dc3-b3d0-483f96fc9564-trusted-ca-bundle\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 20:57:01.921603 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:57:01.921595    2576 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/55572f58-18d5-4dc3-b3d0-483f96fc9564-console-serving-cert\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 20:57:01.921603 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:57:01.921604    2576 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/55572f58-18d5-4dc3-b3d0-483f96fc9564-console-oauth-config\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 20:57:01.921603 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:57:01.921612    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzjg8\" (UniqueName: \"kubernetes.io/projected/55572f58-18d5-4dc3-b3d0-483f96fc9564-kube-api-access-rzjg8\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 20:57:02.495722 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:57:02.495674    2576 generic.go:356] "Generic (PLEG): container finished" podID="55572f58-18d5-4dc3-b3d0-483f96fc9564" containerID="878367a807ac6565e8e0aa9a332f33df6ec2dff0831169c31baa7e5c8c8635c7" exitCode=0
Sep 09 20:57:02.496133 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:57:02.495723    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74dffcb5f7-84lks" event={"ID":"55572f58-18d5-4dc3-b3d0-483f96fc9564","Type":"ContainerDied","Data":"878367a807ac6565e8e0aa9a332f33df6ec2dff0831169c31baa7e5c8c8635c7"}
Sep 09 20:57:02.496133 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:57:02.495775    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74dffcb5f7-84lks" event={"ID":"55572f58-18d5-4dc3-b3d0-483f96fc9564","Type":"ContainerDied","Data":"8df0df82ecf788eff96e73cc61c0d9c41f08132bbad48bb1e44c368f27def660"}
Sep 09 20:57:02.496133 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:57:02.495796    2576 scope.go:117] "RemoveContainer" containerID="878367a807ac6565e8e0aa9a332f33df6ec2dff0831169c31baa7e5c8c8635c7"
Sep 09 20:57:02.496133 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:57:02.495734    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74dffcb5f7-84lks"
Sep 09 20:57:02.505234 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:57:02.505215    2576 scope.go:117] "RemoveContainer" containerID="878367a807ac6565e8e0aa9a332f33df6ec2dff0831169c31baa7e5c8c8635c7"
Sep 09 20:57:02.505561 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:57:02.505538    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"878367a807ac6565e8e0aa9a332f33df6ec2dff0831169c31baa7e5c8c8635c7\": container with ID starting with 878367a807ac6565e8e0aa9a332f33df6ec2dff0831169c31baa7e5c8c8635c7 not found: ID does not exist" containerID="878367a807ac6565e8e0aa9a332f33df6ec2dff0831169c31baa7e5c8c8635c7"
Sep 09 20:57:02.505612 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:57:02.505577    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"878367a807ac6565e8e0aa9a332f33df6ec2dff0831169c31baa7e5c8c8635c7"} err="failed to get container status \"878367a807ac6565e8e0aa9a332f33df6ec2dff0831169c31baa7e5c8c8635c7\": rpc error: code = NotFound desc = could not find container \"878367a807ac6565e8e0aa9a332f33df6ec2dff0831169c31baa7e5c8c8635c7\": container with ID starting with 878367a807ac6565e8e0aa9a332f33df6ec2dff0831169c31baa7e5c8c8635c7 not found: ID does not exist"
Sep 09 20:57:02.518277 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:57:02.518250    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-74dffcb5f7-84lks"]
Sep 09 20:57:02.522631 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:57:02.522606    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-74dffcb5f7-84lks"]
Sep 09 20:57:03.408488 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:57:03.408455    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="55572f58-18d5-4dc3-b3d0-483f96fc9564" path="/var/lib/kubelet/pods/55572f58-18d5-4dc3-b3d0-483f96fc9564/volumes"
Sep 09 20:57:08.331076 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:57:08.331046    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:57:08.403909 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:57:08.403876    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:57:32.323853 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:57:32.323810    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:57:32.415806 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:57:32.415768    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:57:37.318006 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:57:37.317979    2576 kubelet.go:1648] "Image garbage collection succeeded"
Sep 09 20:57:37.697248 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:57:37.697214    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-865c7cb8cb-kk6xg"]
Sep 09 20:57:37.699451 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:57:37.697559    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55572f58-18d5-4dc3-b3d0-483f96fc9564" containerName="console"
Sep 09 20:57:37.699451 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:57:37.697573    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="55572f58-18d5-4dc3-b3d0-483f96fc9564" containerName="console"
Sep 09 20:57:37.699451 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:57:37.697624    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="55572f58-18d5-4dc3-b3d0-483f96fc9564" containerName="console"
Sep 09 20:57:37.700296 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:57:37.700280    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-865c7cb8cb-kk6xg"
Sep 09 20:57:37.714161 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:57:37.714136    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-865c7cb8cb-kk6xg"]
Sep 09 20:57:37.772189 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:57:37.772158    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:57:37.783261 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:57:37.783233    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/228889d4-a02e-4f1e-bc6d-efb354b1d7e0-console-config\") pod \"console-865c7cb8cb-kk6xg\" (UID: \"228889d4-a02e-4f1e-bc6d-efb354b1d7e0\") " pod="openshift-console/console-865c7cb8cb-kk6xg"
Sep 09 20:57:37.783401 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:57:37.783266    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/228889d4-a02e-4f1e-bc6d-efb354b1d7e0-service-ca\") pod \"console-865c7cb8cb-kk6xg\" (UID: \"228889d4-a02e-4f1e-bc6d-efb354b1d7e0\") " pod="openshift-console/console-865c7cb8cb-kk6xg"
Sep 09 20:57:37.783401 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:57:37.783304    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/228889d4-a02e-4f1e-bc6d-efb354b1d7e0-console-serving-cert\") pod \"console-865c7cb8cb-kk6xg\" (UID: \"228889d4-a02e-4f1e-bc6d-efb354b1d7e0\") " pod="openshift-console/console-865c7cb8cb-kk6xg"
Sep 09 20:57:37.783401 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:57:37.783391    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/228889d4-a02e-4f1e-bc6d-efb354b1d7e0-console-oauth-config\") pod \"console-865c7cb8cb-kk6xg\" (UID: \"228889d4-a02e-4f1e-bc6d-efb354b1d7e0\") " pod="openshift-console/console-865c7cb8cb-kk6xg"
Sep 09 20:57:37.783549 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:57:37.783438    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/228889d4-a02e-4f1e-bc6d-efb354b1d7e0-oauth-serving-cert\") pod \"console-865c7cb8cb-kk6xg\" (UID: \"228889d4-a02e-4f1e-bc6d-efb354b1d7e0\") " pod="openshift-console/console-865c7cb8cb-kk6xg"
Sep 09 20:57:37.783549 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:57:37.783465    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/228889d4-a02e-4f1e-bc6d-efb354b1d7e0-trusted-ca-bundle\") pod \"console-865c7cb8cb-kk6xg\" (UID: \"228889d4-a02e-4f1e-bc6d-efb354b1d7e0\") " pod="openshift-console/console-865c7cb8cb-kk6xg"
Sep 09 20:57:37.783549 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:57:37.783486    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lflr2\" (UniqueName: \"kubernetes.io/projected/228889d4-a02e-4f1e-bc6d-efb354b1d7e0-kube-api-access-lflr2\") pod \"console-865c7cb8cb-kk6xg\" (UID: \"228889d4-a02e-4f1e-bc6d-efb354b1d7e0\") " pod="openshift-console/console-865c7cb8cb-kk6xg"
Sep 09 20:57:37.834008 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:57:37.833969    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:57:37.884574 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:57:37.884546    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/228889d4-a02e-4f1e-bc6d-efb354b1d7e0-console-serving-cert\") pod \"console-865c7cb8cb-kk6xg\" (UID: \"228889d4-a02e-4f1e-bc6d-efb354b1d7e0\") " pod="openshift-console/console-865c7cb8cb-kk6xg"
Sep 09 20:57:37.884698 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:57:37.884579    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/228889d4-a02e-4f1e-bc6d-efb354b1d7e0-console-oauth-config\") pod \"console-865c7cb8cb-kk6xg\" (UID: \"228889d4-a02e-4f1e-bc6d-efb354b1d7e0\") " pod="openshift-console/console-865c7cb8cb-kk6xg"
Sep 09 20:57:37.884698 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:57:37.884611    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/228889d4-a02e-4f1e-bc6d-efb354b1d7e0-oauth-serving-cert\") pod \"console-865c7cb8cb-kk6xg\" (UID: \"228889d4-a02e-4f1e-bc6d-efb354b1d7e0\") " pod="openshift-console/console-865c7cb8cb-kk6xg"
Sep 09 20:57:37.884698 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:57:37.884627    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/228889d4-a02e-4f1e-bc6d-efb354b1d7e0-trusted-ca-bundle\") pod \"console-865c7cb8cb-kk6xg\" (UID: \"228889d4-a02e-4f1e-bc6d-efb354b1d7e0\") " pod="openshift-console/console-865c7cb8cb-kk6xg"
Sep 09 20:57:37.884698 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:57:37.884646    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lflr2\" (UniqueName: \"kubernetes.io/projected/228889d4-a02e-4f1e-bc6d-efb354b1d7e0-kube-api-access-lflr2\") pod \"console-865c7cb8cb-kk6xg\" (UID: \"228889d4-a02e-4f1e-bc6d-efb354b1d7e0\") " pod="openshift-console/console-865c7cb8cb-kk6xg"
Sep 09 20:57:37.884698 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:57:37.884692    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/228889d4-a02e-4f1e-bc6d-efb354b1d7e0-console-config\") pod \"console-865c7cb8cb-kk6xg\" (UID: \"228889d4-a02e-4f1e-bc6d-efb354b1d7e0\") " pod="openshift-console/console-865c7cb8cb-kk6xg"
Sep 09 20:57:37.884936 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:57:37.884731    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/228889d4-a02e-4f1e-bc6d-efb354b1d7e0-service-ca\") pod \"console-865c7cb8cb-kk6xg\" (UID: \"228889d4-a02e-4f1e-bc6d-efb354b1d7e0\") " pod="openshift-console/console-865c7cb8cb-kk6xg"
Sep 09 20:57:37.885337 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:57:37.885309    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/228889d4-a02e-4f1e-bc6d-efb354b1d7e0-oauth-serving-cert\") pod \"console-865c7cb8cb-kk6xg\" (UID: \"228889d4-a02e-4f1e-bc6d-efb354b1d7e0\") " pod="openshift-console/console-865c7cb8cb-kk6xg"
Sep 09 20:57:37.885466 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:57:37.885444    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/228889d4-a02e-4f1e-bc6d-efb354b1d7e0-service-ca\") pod \"console-865c7cb8cb-kk6xg\" (UID: \"228889d4-a02e-4f1e-bc6d-efb354b1d7e0\") " pod="openshift-console/console-865c7cb8cb-kk6xg"
Sep 09 20:57:37.885602 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:57:37.885499    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/228889d4-a02e-4f1e-bc6d-efb354b1d7e0-trusted-ca-bundle\") pod \"console-865c7cb8cb-kk6xg\" (UID: \"228889d4-a02e-4f1e-bc6d-efb354b1d7e0\") " pod="openshift-console/console-865c7cb8cb-kk6xg"
Sep 09 20:57:37.885602 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:57:37.885571    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/228889d4-a02e-4f1e-bc6d-efb354b1d7e0-console-config\") pod \"console-865c7cb8cb-kk6xg\" (UID: \"228889d4-a02e-4f1e-bc6d-efb354b1d7e0\") " pod="openshift-console/console-865c7cb8cb-kk6xg"
Sep 09 20:57:37.887721 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:57:37.887706    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/228889d4-a02e-4f1e-bc6d-efb354b1d7e0-console-serving-cert\") pod \"console-865c7cb8cb-kk6xg\" (UID: \"228889d4-a02e-4f1e-bc6d-efb354b1d7e0\") " pod="openshift-console/console-865c7cb8cb-kk6xg"
Sep 09 20:57:37.887845 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:57:37.887825    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/228889d4-a02e-4f1e-bc6d-efb354b1d7e0-console-oauth-config\") pod \"console-865c7cb8cb-kk6xg\" (UID: \"228889d4-a02e-4f1e-bc6d-efb354b1d7e0\") " pod="openshift-console/console-865c7cb8cb-kk6xg"
Sep 09 20:57:37.894037 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:57:37.894016    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lflr2\" (UniqueName: \"kubernetes.io/projected/228889d4-a02e-4f1e-bc6d-efb354b1d7e0-kube-api-access-lflr2\") pod \"console-865c7cb8cb-kk6xg\" (UID: \"228889d4-a02e-4f1e-bc6d-efb354b1d7e0\") " pod="openshift-console/console-865c7cb8cb-kk6xg"
Sep 09 20:57:38.010149 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:57:38.010054    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-865c7cb8cb-kk6xg"
Sep 09 20:57:38.301184 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:57:38.301108    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-865c7cb8cb-kk6xg"]
Sep 09 20:57:38.304115 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:57:38.304082    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod228889d4_a02e_4f1e_bc6d_efb354b1d7e0.slice/crio-2ed896d455b8f2bd647f5a347caa67f78f5eae4c13b58d432b69bc71a6ceb1e2 WatchSource:0}: Error finding container 2ed896d455b8f2bd647f5a347caa67f78f5eae4c13b58d432b69bc71a6ceb1e2: Status 404 returned error can't find the container with id 2ed896d455b8f2bd647f5a347caa67f78f5eae4c13b58d432b69bc71a6ceb1e2
Sep 09 20:57:38.625469 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:57:38.625435    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-865c7cb8cb-kk6xg" event={"ID":"228889d4-a02e-4f1e-bc6d-efb354b1d7e0","Type":"ContainerStarted","Data":"d4ace748662f2789899ea27e746fce9ff3f8db5a5ad12849ce39e8a7caf41803"}
Sep 09 20:57:38.625820 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:57:38.625491    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-865c7cb8cb-kk6xg" event={"ID":"228889d4-a02e-4f1e-bc6d-efb354b1d7e0","Type":"ContainerStarted","Data":"2ed896d455b8f2bd647f5a347caa67f78f5eae4c13b58d432b69bc71a6ceb1e2"}
Sep 09 20:57:38.645245 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:57:38.645194    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-865c7cb8cb-kk6xg" podStartSLOduration=1.645174263 podStartE2EDuration="1.645174263s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:57:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:57:38.644703272 +0000 UTC m=+301.758389127" watchObservedRunningTime="2026-09-09 20:57:38.645174263 +0000 UTC m=+301.758860125"
Sep 09 20:57:42.799855 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:57:42.799817    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:57:44.312722 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:57:44.312651    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:57:44.384834 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:57:44.384799    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:57:48.011175 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:57:48.011128    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-865c7cb8cb-kk6xg"
Sep 09 20:57:48.011175 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:57:48.011191    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-865c7cb8cb-kk6xg"
Sep 09 20:57:48.015907 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:57:48.015887    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-865c7cb8cb-kk6xg"
Sep 09 20:57:48.664947 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:57:48.664921    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-865c7cb8cb-kk6xg"
Sep 09 20:57:48.717308 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:57:48.717268    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-b78f8dfc8-4brqr"]
Sep 09 20:57:48.829024 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:57:48.828991    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:57:48.930027 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:57:48.929938    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:57:48.991638 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:57:48.991611    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:57:59.211905 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:57:59.211868    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:57:59.300532 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:57:59.300482    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:57:59.399940 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:57:59.399908    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:58:13.737951 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:58:13.737904    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-b78f8dfc8-4brqr" podUID="e65af6f5-ae07-4b83-b79d-53f2bddc8947" containerName="console" containerID="cri-o://f97ad6ffea1fdf1671ee943eac4107c0178bd502859fbb43dd899baad78a3ded" gracePeriod=15
Sep 09 20:58:13.973547 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:58:13.973520    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b78f8dfc8-4brqr"
Sep 09 20:58:13.984644 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:58:13.984623    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cpjzk\" (UniqueName: \"kubernetes.io/projected/e65af6f5-ae07-4b83-b79d-53f2bddc8947-kube-api-access-cpjzk\") pod \"e65af6f5-ae07-4b83-b79d-53f2bddc8947\" (UID: \"e65af6f5-ae07-4b83-b79d-53f2bddc8947\") "
Sep 09 20:58:13.984750 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:58:13.984656    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e65af6f5-ae07-4b83-b79d-53f2bddc8947-service-ca\") pod \"e65af6f5-ae07-4b83-b79d-53f2bddc8947\" (UID: \"e65af6f5-ae07-4b83-b79d-53f2bddc8947\") "
Sep 09 20:58:13.984750 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:58:13.984701    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e65af6f5-ae07-4b83-b79d-53f2bddc8947-oauth-serving-cert\") pod \"e65af6f5-ae07-4b83-b79d-53f2bddc8947\" (UID: \"e65af6f5-ae07-4b83-b79d-53f2bddc8947\") "
Sep 09 20:58:13.984750 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:58:13.984746    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e65af6f5-ae07-4b83-b79d-53f2bddc8947-console-serving-cert\") pod \"e65af6f5-ae07-4b83-b79d-53f2bddc8947\" (UID: \"e65af6f5-ae07-4b83-b79d-53f2bddc8947\") "
Sep 09 20:58:13.984893 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:58:13.984772    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e65af6f5-ae07-4b83-b79d-53f2bddc8947-console-oauth-config\") pod \"e65af6f5-ae07-4b83-b79d-53f2bddc8947\" (UID: \"e65af6f5-ae07-4b83-b79d-53f2bddc8947\") "
Sep 09 20:58:13.984893 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:58:13.984851    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e65af6f5-ae07-4b83-b79d-53f2bddc8947-trusted-ca-bundle\") pod \"e65af6f5-ae07-4b83-b79d-53f2bddc8947\" (UID: \"e65af6f5-ae07-4b83-b79d-53f2bddc8947\") "
Sep 09 20:58:13.984993 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:58:13.984931    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e65af6f5-ae07-4b83-b79d-53f2bddc8947-console-config\") pod \"e65af6f5-ae07-4b83-b79d-53f2bddc8947\" (UID: \"e65af6f5-ae07-4b83-b79d-53f2bddc8947\") "
Sep 09 20:58:13.985191 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:58:13.985166    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e65af6f5-ae07-4b83-b79d-53f2bddc8947-service-ca" (OuterVolumeSpecName: "service-ca") pod "e65af6f5-ae07-4b83-b79d-53f2bddc8947" (UID: "e65af6f5-ae07-4b83-b79d-53f2bddc8947"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:58:13.985326 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:58:13.985245    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e65af6f5-ae07-4b83-b79d-53f2bddc8947-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "e65af6f5-ae07-4b83-b79d-53f2bddc8947" (UID: "e65af6f5-ae07-4b83-b79d-53f2bddc8947"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:58:13.985431 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:58:13.985409    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e65af6f5-ae07-4b83-b79d-53f2bddc8947-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "e65af6f5-ae07-4b83-b79d-53f2bddc8947" (UID: "e65af6f5-ae07-4b83-b79d-53f2bddc8947"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:58:13.985483 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:58:13.985424    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e65af6f5-ae07-4b83-b79d-53f2bddc8947-console-config" (OuterVolumeSpecName: "console-config") pod "e65af6f5-ae07-4b83-b79d-53f2bddc8947" (UID: "e65af6f5-ae07-4b83-b79d-53f2bddc8947"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:58:13.986875 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:58:13.986859    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e65af6f5-ae07-4b83-b79d-53f2bddc8947-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "e65af6f5-ae07-4b83-b79d-53f2bddc8947" (UID: "e65af6f5-ae07-4b83-b79d-53f2bddc8947"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:58:13.987354 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:58:13.987332    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e65af6f5-ae07-4b83-b79d-53f2bddc8947-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "e65af6f5-ae07-4b83-b79d-53f2bddc8947" (UID: "e65af6f5-ae07-4b83-b79d-53f2bddc8947"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:58:13.987354 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:58:13.987342    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e65af6f5-ae07-4b83-b79d-53f2bddc8947-kube-api-access-cpjzk" (OuterVolumeSpecName: "kube-api-access-cpjzk") pod "e65af6f5-ae07-4b83-b79d-53f2bddc8947" (UID: "e65af6f5-ae07-4b83-b79d-53f2bddc8947"). InnerVolumeSpecName "kube-api-access-cpjzk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:58:14.086448 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:58:14.086417    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cpjzk\" (UniqueName: \"kubernetes.io/projected/e65af6f5-ae07-4b83-b79d-53f2bddc8947-kube-api-access-cpjzk\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 20:58:14.086448 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:58:14.086442    2576 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e65af6f5-ae07-4b83-b79d-53f2bddc8947-service-ca\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 20:58:14.086448 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:58:14.086451    2576 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e65af6f5-ae07-4b83-b79d-53f2bddc8947-oauth-serving-cert\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 20:58:14.086671 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:58:14.086460    2576 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e65af6f5-ae07-4b83-b79d-53f2bddc8947-console-serving-cert\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 20:58:14.086671 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:58:14.086469    2576 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e65af6f5-ae07-4b83-b79d-53f2bddc8947-console-oauth-config\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 20:58:14.086671 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:58:14.086477    2576 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e65af6f5-ae07-4b83-b79d-53f2bddc8947-trusted-ca-bundle\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 20:58:14.086671 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:58:14.086485    2576 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e65af6f5-ae07-4b83-b79d-53f2bddc8947-console-config\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 20:58:14.756145 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:58:14.756109    2576 generic.go:356] "Generic (PLEG): container finished" podID="e65af6f5-ae07-4b83-b79d-53f2bddc8947" containerID="f97ad6ffea1fdf1671ee943eac4107c0178bd502859fbb43dd899baad78a3ded" exitCode=0
Sep 09 20:58:14.756628 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:58:14.756162    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b78f8dfc8-4brqr"
Sep 09 20:58:14.756628 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:58:14.756197    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b78f8dfc8-4brqr" event={"ID":"e65af6f5-ae07-4b83-b79d-53f2bddc8947","Type":"ContainerDied","Data":"f97ad6ffea1fdf1671ee943eac4107c0178bd502859fbb43dd899baad78a3ded"}
Sep 09 20:58:14.756628 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:58:14.756248    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b78f8dfc8-4brqr" event={"ID":"e65af6f5-ae07-4b83-b79d-53f2bddc8947","Type":"ContainerDied","Data":"fb67932171da3c41940d5e5e1273e73e5efb366b0944fb117acd1266a2e72a38"}
Sep 09 20:58:14.756628 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:58:14.756272    2576 scope.go:117] "RemoveContainer" containerID="f97ad6ffea1fdf1671ee943eac4107c0178bd502859fbb43dd899baad78a3ded"
Sep 09 20:58:14.768899 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:58:14.768878    2576 scope.go:117] "RemoveContainer" containerID="f97ad6ffea1fdf1671ee943eac4107c0178bd502859fbb43dd899baad78a3ded"
Sep 09 20:58:14.769166 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:58:14.769150    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f97ad6ffea1fdf1671ee943eac4107c0178bd502859fbb43dd899baad78a3ded\": container with ID starting with f97ad6ffea1fdf1671ee943eac4107c0178bd502859fbb43dd899baad78a3ded not found: ID does not exist" containerID="f97ad6ffea1fdf1671ee943eac4107c0178bd502859fbb43dd899baad78a3ded"
Sep 09 20:58:14.769210 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:58:14.769173    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f97ad6ffea1fdf1671ee943eac4107c0178bd502859fbb43dd899baad78a3ded"} err="failed to get container status \"f97ad6ffea1fdf1671ee943eac4107c0178bd502859fbb43dd899baad78a3ded\": rpc error: code = NotFound desc = could not find container \"f97ad6ffea1fdf1671ee943eac4107c0178bd502859fbb43dd899baad78a3ded\": container with ID starting with f97ad6ffea1fdf1671ee943eac4107c0178bd502859fbb43dd899baad78a3ded not found: ID does not exist"
Sep 09 20:58:14.781449 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:58:14.781427    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-b78f8dfc8-4brqr"]
Sep 09 20:58:14.784176 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:58:14.784133    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-b78f8dfc8-4brqr"]
Sep 09 20:58:15.408207 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:58:15.408178    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e65af6f5-ae07-4b83-b79d-53f2bddc8947" path="/var/lib/kubelet/pods/e65af6f5-ae07-4b83-b79d-53f2bddc8947/volumes"
Sep 09 20:58:30.293991 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:58:30.293904    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:58:30.392570 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:58:30.392540    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:58:30.493759 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:58:30.493715    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:58:31.511429 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:58:31.511392    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1af8whp"]
Sep 09 20:58:31.511774 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:58:31.511658    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e65af6f5-ae07-4b83-b79d-53f2bddc8947" containerName="console"
Sep 09 20:58:31.511774 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:58:31.511669    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="e65af6f5-ae07-4b83-b79d-53f2bddc8947" containerName="console"
Sep 09 20:58:31.511774 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:58:31.511727    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="e65af6f5-ae07-4b83-b79d-53f2bddc8947" containerName="console"
Sep 09 20:58:31.514688 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:58:31.514673    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1af8whp"
Sep 09 20:58:31.517776 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:58:31.517756    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 09 20:58:31.517906 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:58:31.517892    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 09 20:58:31.518893 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:58:31.518877    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-lbxks\""
Sep 09 20:58:31.524050 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:58:31.524029    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1af8whp"]
Sep 09 20:58:31.610352 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:58:31.610327    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e8abad66-0ee0-41b5-b00f-68819a2cf85c-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1af8whp\" (UID: \"e8abad66-0ee0-41b5-b00f-68819a2cf85c\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1af8whp"
Sep 09 20:58:31.610479 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:58:31.610386    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gr9gr\" (UniqueName: \"kubernetes.io/projected/e8abad66-0ee0-41b5-b00f-68819a2cf85c-kube-api-access-gr9gr\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1af8whp\" (UID: \"e8abad66-0ee0-41b5-b00f-68819a2cf85c\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1af8whp"
Sep 09 20:58:31.610479 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:58:31.610457    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e8abad66-0ee0-41b5-b00f-68819a2cf85c-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1af8whp\" (UID: \"e8abad66-0ee0-41b5-b00f-68819a2cf85c\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1af8whp"
Sep 09 20:58:31.711629 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:58:31.711596    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e8abad66-0ee0-41b5-b00f-68819a2cf85c-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1af8whp\" (UID: \"e8abad66-0ee0-41b5-b00f-68819a2cf85c\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1af8whp"
Sep 09 20:58:31.711734 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:58:31.711657    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e8abad66-0ee0-41b5-b00f-68819a2cf85c-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1af8whp\" (UID: \"e8abad66-0ee0-41b5-b00f-68819a2cf85c\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1af8whp"
Sep 09 20:58:31.711734 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:58:31.711699    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gr9gr\" (UniqueName: \"kubernetes.io/projected/e8abad66-0ee0-41b5-b00f-68819a2cf85c-kube-api-access-gr9gr\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1af8whp\" (UID: \"e8abad66-0ee0-41b5-b00f-68819a2cf85c\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1af8whp"
Sep 09 20:58:31.712040 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:58:31.712009    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e8abad66-0ee0-41b5-b00f-68819a2cf85c-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1af8whp\" (UID: \"e8abad66-0ee0-41b5-b00f-68819a2cf85c\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1af8whp"
Sep 09 20:58:31.712040 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:58:31.712030    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e8abad66-0ee0-41b5-b00f-68819a2cf85c-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1af8whp\" (UID: \"e8abad66-0ee0-41b5-b00f-68819a2cf85c\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1af8whp"
Sep 09 20:58:31.721616 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:58:31.721591    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gr9gr\" (UniqueName: \"kubernetes.io/projected/e8abad66-0ee0-41b5-b00f-68819a2cf85c-kube-api-access-gr9gr\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1af8whp\" (UID: \"e8abad66-0ee0-41b5-b00f-68819a2cf85c\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1af8whp"
Sep 09 20:58:31.824290 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:58:31.824203    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1af8whp"
Sep 09 20:58:32.094114 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:58:32.094083    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1af8whp"]
Sep 09 20:58:32.097109 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:58:32.097084    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8abad66_0ee0_41b5_b00f_68819a2cf85c.slice/crio-43c6ba45d91d67789d7aefcb2e6578436f0048abefaa8c71098fde96aac9e139 WatchSource:0}: Error finding container 43c6ba45d91d67789d7aefcb2e6578436f0048abefaa8c71098fde96aac9e139: Status 404 returned error can't find the container with id 43c6ba45d91d67789d7aefcb2e6578436f0048abefaa8c71098fde96aac9e139
Sep 09 20:58:32.099092 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:58:32.099077    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 20:58:32.823936 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:58:32.823900    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1af8whp" event={"ID":"e8abad66-0ee0-41b5-b00f-68819a2cf85c","Type":"ContainerStarted","Data":"43c6ba45d91d67789d7aefcb2e6578436f0048abefaa8c71098fde96aac9e139"}
Sep 09 20:58:38.849185 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:58:38.849149    2576 generic.go:356] "Generic (PLEG): container finished" podID="e8abad66-0ee0-41b5-b00f-68819a2cf85c" containerID="d0684ff6832b3e13fb54790c221e5d8a4c763489cad87ae298a00b1b9aa9bd1d" exitCode=0
Sep 09 20:58:38.849578 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:58:38.849206    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1af8whp" event={"ID":"e8abad66-0ee0-41b5-b00f-68819a2cf85c","Type":"ContainerDied","Data":"d0684ff6832b3e13fb54790c221e5d8a4c763489cad87ae298a00b1b9aa9bd1d"}
Sep 09 20:58:40.859681 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:58:40.859651    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1af8whp" event={"ID":"e8abad66-0ee0-41b5-b00f-68819a2cf85c","Type":"ContainerStarted","Data":"a03e3e605a9c4c9b41f0d853a70a84b48845c0cb006ba2cc570a8e74aef324a0"}
Sep 09 20:58:41.865326 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:58:41.865292    2576 generic.go:356] "Generic (PLEG): container finished" podID="e8abad66-0ee0-41b5-b00f-68819a2cf85c" containerID="a03e3e605a9c4c9b41f0d853a70a84b48845c0cb006ba2cc570a8e74aef324a0" exitCode=0
Sep 09 20:58:41.865706 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:58:41.865399    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1af8whp" event={"ID":"e8abad66-0ee0-41b5-b00f-68819a2cf85c","Type":"ContainerDied","Data":"a03e3e605a9c4c9b41f0d853a70a84b48845c0cb006ba2cc570a8e74aef324a0"}
Sep 09 20:58:49.900428 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:58:49.900384    2576 generic.go:356] "Generic (PLEG): container finished" podID="e8abad66-0ee0-41b5-b00f-68819a2cf85c" containerID="da6fd83447ef718400eb64c173ac221ad9ce1b93ac3fffd5a4809f2cefed7a39" exitCode=0
Sep 09 20:58:49.900428 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:58:49.900396    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1af8whp" event={"ID":"e8abad66-0ee0-41b5-b00f-68819a2cf85c","Type":"ContainerDied","Data":"da6fd83447ef718400eb64c173ac221ad9ce1b93ac3fffd5a4809f2cefed7a39"}
Sep 09 20:58:51.064379 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:58:51.064351    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1af8whp"
Sep 09 20:58:51.171297 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:58:51.171224    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e8abad66-0ee0-41b5-b00f-68819a2cf85c-bundle\") pod \"e8abad66-0ee0-41b5-b00f-68819a2cf85c\" (UID: \"e8abad66-0ee0-41b5-b00f-68819a2cf85c\") "
Sep 09 20:58:51.171297 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:58:51.171283    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gr9gr\" (UniqueName: \"kubernetes.io/projected/e8abad66-0ee0-41b5-b00f-68819a2cf85c-kube-api-access-gr9gr\") pod \"e8abad66-0ee0-41b5-b00f-68819a2cf85c\" (UID: \"e8abad66-0ee0-41b5-b00f-68819a2cf85c\") "
Sep 09 20:58:51.171539 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:58:51.171330    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e8abad66-0ee0-41b5-b00f-68819a2cf85c-util\") pod \"e8abad66-0ee0-41b5-b00f-68819a2cf85c\" (UID: \"e8abad66-0ee0-41b5-b00f-68819a2cf85c\") "
Sep 09 20:58:51.171892 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:58:51.171867    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8abad66-0ee0-41b5-b00f-68819a2cf85c-bundle" (OuterVolumeSpecName: "bundle") pod "e8abad66-0ee0-41b5-b00f-68819a2cf85c" (UID: "e8abad66-0ee0-41b5-b00f-68819a2cf85c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:58:51.173667 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:58:51.173650    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8abad66-0ee0-41b5-b00f-68819a2cf85c-kube-api-access-gr9gr" (OuterVolumeSpecName: "kube-api-access-gr9gr") pod "e8abad66-0ee0-41b5-b00f-68819a2cf85c" (UID: "e8abad66-0ee0-41b5-b00f-68819a2cf85c"). InnerVolumeSpecName "kube-api-access-gr9gr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:58:51.175497 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:58:51.175478    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8abad66-0ee0-41b5-b00f-68819a2cf85c-util" (OuterVolumeSpecName: "util") pod "e8abad66-0ee0-41b5-b00f-68819a2cf85c" (UID: "e8abad66-0ee0-41b5-b00f-68819a2cf85c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:58:51.272214 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:58:51.272183    2576 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e8abad66-0ee0-41b5-b00f-68819a2cf85c-util\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 20:58:51.272214 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:58:51.272209    2576 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e8abad66-0ee0-41b5-b00f-68819a2cf85c-bundle\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 20:58:51.272214 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:58:51.272218    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gr9gr\" (UniqueName: \"kubernetes.io/projected/e8abad66-0ee0-41b5-b00f-68819a2cf85c-kube-api-access-gr9gr\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 20:58:51.909418 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:58:51.909359    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1af8whp" event={"ID":"e8abad66-0ee0-41b5-b00f-68819a2cf85c","Type":"ContainerDied","Data":"43c6ba45d91d67789d7aefcb2e6578436f0048abefaa8c71098fde96aac9e139"}
Sep 09 20:58:51.909613 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:58:51.909443    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="43c6ba45d91d67789d7aefcb2e6578436f0048abefaa8c71098fde96aac9e139"
Sep 09 20:58:51.909613 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:58:51.909402    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1af8whp"
Sep 09 20:58:52.895269 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:58:52.895229    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:58:54.040244 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:58:54.040209    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:58:55.026610 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:58:55.026570    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-xbgh9"]
Sep 09 20:58:55.027222 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:58:55.027202    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8abad66-0ee0-41b5-b00f-68819a2cf85c" containerName="pull"
Sep 09 20:58:55.027316 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:58:55.027224    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8abad66-0ee0-41b5-b00f-68819a2cf85c" containerName="pull"
Sep 09 20:58:55.027316 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:58:55.027253    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8abad66-0ee0-41b5-b00f-68819a2cf85c" containerName="util"
Sep 09 20:58:55.027316 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:58:55.027261    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8abad66-0ee0-41b5-b00f-68819a2cf85c" containerName="util"
Sep 09 20:58:55.027316 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:58:55.027272    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8abad66-0ee0-41b5-b00f-68819a2cf85c" containerName="extract"
Sep 09 20:58:55.027316 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:58:55.027280    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8abad66-0ee0-41b5-b00f-68819a2cf85c" containerName="extract"
Sep 09 20:58:55.027554 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:58:55.027434    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8abad66-0ee0-41b5-b00f-68819a2cf85c" containerName="extract"
Sep 09 20:58:55.029872 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:58:55.029849    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-xbgh9"
Sep 09 20:58:55.032959 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:58:55.032938    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Sep 09 20:58:55.033093 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:58:55.032976    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Sep 09 20:58:55.033093 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:58:55.032979    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-s9ms2\""
Sep 09 20:58:55.033235 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:58:55.033221    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Sep 09 20:58:55.042264 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:58:55.042244    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-xbgh9"]
Sep 09 20:58:55.102427 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:58:55.102400    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4jcb\" (UniqueName: \"kubernetes.io/projected/0aa9bba2-f984-4e2a-8058-7886733c6607-kube-api-access-s4jcb\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-xbgh9\" (UID: \"0aa9bba2-f984-4e2a-8058-7886733c6607\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-xbgh9"
Sep 09 20:58:55.102548 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:58:55.102476    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/0aa9bba2-f984-4e2a-8058-7886733c6607-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-xbgh9\" (UID: \"0aa9bba2-f984-4e2a-8058-7886733c6607\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-xbgh9"
Sep 09 20:58:55.203176 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:58:55.203142    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4jcb\" (UniqueName: \"kubernetes.io/projected/0aa9bba2-f984-4e2a-8058-7886733c6607-kube-api-access-s4jcb\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-xbgh9\" (UID: \"0aa9bba2-f984-4e2a-8058-7886733c6607\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-xbgh9"
Sep 09 20:58:55.203314 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:58:55.203217    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/0aa9bba2-f984-4e2a-8058-7886733c6607-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-xbgh9\" (UID: \"0aa9bba2-f984-4e2a-8058-7886733c6607\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-xbgh9"
Sep 09 20:58:55.205629 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:58:55.205611    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/0aa9bba2-f984-4e2a-8058-7886733c6607-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-xbgh9\" (UID: \"0aa9bba2-f984-4e2a-8058-7886733c6607\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-xbgh9"
Sep 09 20:58:55.212604 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:58:55.212582    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4jcb\" (UniqueName: \"kubernetes.io/projected/0aa9bba2-f984-4e2a-8058-7886733c6607-kube-api-access-s4jcb\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-xbgh9\" (UID: \"0aa9bba2-f984-4e2a-8058-7886733c6607\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-xbgh9"
Sep 09 20:58:55.340248 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:58:55.340224    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-xbgh9"
Sep 09 20:58:55.614275 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:58:55.614201    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-xbgh9"]
Sep 09 20:58:55.617114 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:58:55.617088    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0aa9bba2_f984_4e2a_8058_7886733c6607.slice/crio-4f09ad7a5001ec62d54adb419f9ed577d34c38c00b0b66e26fab4b70bc21a330 WatchSource:0}: Error finding container 4f09ad7a5001ec62d54adb419f9ed577d34c38c00b0b66e26fab4b70bc21a330: Status 404 returned error can't find the container with id 4f09ad7a5001ec62d54adb419f9ed577d34c38c00b0b66e26fab4b70bc21a330
Sep 09 20:58:55.924724 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:58:55.924650    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-xbgh9" event={"ID":"0aa9bba2-f984-4e2a-8058-7886733c6607","Type":"ContainerStarted","Data":"4f09ad7a5001ec62d54adb419f9ed577d34c38c00b0b66e26fab4b70bc21a330"}
Sep 09 20:58:58.938129 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:58:58.938052    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-xbgh9" event={"ID":"0aa9bba2-f984-4e2a-8058-7886733c6607","Type":"ContainerStarted","Data":"e2e9c8e75d9a6f6f9bcbefe2f036531ff4648ff228577d6f61746daeebaa0be0"}
Sep 09 20:58:58.938501 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:58:58.938177    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-xbgh9"
Sep 09 20:58:58.962313 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:58:58.962272    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-xbgh9" podStartSLOduration=0.926688742 podStartE2EDuration="3.962257323s" totalImagesPullingTime="3.035568581s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:58:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:58:58.96048293 +0000 UTC m=+382.074168788" watchObservedRunningTime="2026-09-09 20:58:58.962257323 +0000 UTC m=+382.075943183"
Sep 09 20:58:59.282157 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:58:59.282075    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:58:59.544770 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:58:59.544679    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:58:59.562504 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:58:59.562476    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-579c8958f7-5z57d"]
Sep 09 20:58:59.565327 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:58:59.565308    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-5z57d"
Sep 09 20:58:59.568208 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:58:59.568180    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
Sep 09 20:58:59.568313 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:58:59.568184    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Sep 09 20:58:59.568489 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:58:59.568473    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-rd4t9\""
Sep 09 20:58:59.580471 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:58:59.580452    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-5z57d"]
Sep 09 20:58:59.642349 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:58:59.642316    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/2ecf51bc-7914-4aaf-b4aa-b1fbed72d1dd-cabundle0\") pod \"keda-operator-579c8958f7-5z57d\" (UID: \"2ecf51bc-7914-4aaf-b4aa-b1fbed72d1dd\") " pod="openshift-keda/keda-operator-579c8958f7-5z57d"
Sep 09 20:58:59.642349 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:58:59.642352    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljg96\" (UniqueName: \"kubernetes.io/projected/2ecf51bc-7914-4aaf-b4aa-b1fbed72d1dd-kube-api-access-ljg96\") pod \"keda-operator-579c8958f7-5z57d\" (UID: \"2ecf51bc-7914-4aaf-b4aa-b1fbed72d1dd\") " pod="openshift-keda/keda-operator-579c8958f7-5z57d"
Sep 09 20:58:59.642575 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:58:59.642383    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/2ecf51bc-7914-4aaf-b4aa-b1fbed72d1dd-certificates\") pod \"keda-operator-579c8958f7-5z57d\" (UID: \"2ecf51bc-7914-4aaf-b4aa-b1fbed72d1dd\") " pod="openshift-keda/keda-operator-579c8958f7-5z57d"
Sep 09 20:58:59.710602 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:58:59.710565    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:58:59.743095 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:58:59.743067    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/2ecf51bc-7914-4aaf-b4aa-b1fbed72d1dd-cabundle0\") pod \"keda-operator-579c8958f7-5z57d\" (UID: \"2ecf51bc-7914-4aaf-b4aa-b1fbed72d1dd\") " pod="openshift-keda/keda-operator-579c8958f7-5z57d"
Sep 09 20:58:59.743236 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:58:59.743102    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljg96\" (UniqueName: \"kubernetes.io/projected/2ecf51bc-7914-4aaf-b4aa-b1fbed72d1dd-kube-api-access-ljg96\") pod \"keda-operator-579c8958f7-5z57d\" (UID: \"2ecf51bc-7914-4aaf-b4aa-b1fbed72d1dd\") " pod="openshift-keda/keda-operator-579c8958f7-5z57d"
Sep 09 20:58:59.743236 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:58:59.743129    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/2ecf51bc-7914-4aaf-b4aa-b1fbed72d1dd-certificates\") pod \"keda-operator-579c8958f7-5z57d\" (UID: \"2ecf51bc-7914-4aaf-b4aa-b1fbed72d1dd\") " pod="openshift-keda/keda-operator-579c8958f7-5z57d"
Sep 09 20:58:59.743312 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:58:59.743291    2576 secret.go:281] references non-existent secret key: ca.crt
Sep 09 20:58:59.743344 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:58:59.743313    2576 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Sep 09 20:58:59.743344 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:58:59.743322    2576 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-579c8958f7-5z57d: references non-existent secret key: ca.crt
Sep 09 20:58:59.743462 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:58:59.743419    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2ecf51bc-7914-4aaf-b4aa-b1fbed72d1dd-certificates podName:2ecf51bc-7914-4aaf-b4aa-b1fbed72d1dd nodeName:}" failed. No retries permitted until 2026-09-09 20:59:00.243402981 +0000 UTC m=+383.357088830 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/2ecf51bc-7914-4aaf-b4aa-b1fbed72d1dd-certificates") pod "keda-operator-579c8958f7-5z57d" (UID: "2ecf51bc-7914-4aaf-b4aa-b1fbed72d1dd") : references non-existent secret key: ca.crt
Sep 09 20:58:59.743706 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:58:59.743690    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/2ecf51bc-7914-4aaf-b4aa-b1fbed72d1dd-cabundle0\") pod \"keda-operator-579c8958f7-5z57d\" (UID: \"2ecf51bc-7914-4aaf-b4aa-b1fbed72d1dd\") " pod="openshift-keda/keda-operator-579c8958f7-5z57d"
Sep 09 20:58:59.755663 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:58:59.755640    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljg96\" (UniqueName: \"kubernetes.io/projected/2ecf51bc-7914-4aaf-b4aa-b1fbed72d1dd-kube-api-access-ljg96\") pod \"keda-operator-579c8958f7-5z57d\" (UID: \"2ecf51bc-7914-4aaf-b4aa-b1fbed72d1dd\") " pod="openshift-keda/keda-operator-579c8958f7-5z57d"
Sep 09 20:58:59.946329 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:58:59.946290    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:58:59.981445 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:58:59.981413    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-s97xk"]
Sep 09 20:58:59.985107 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:58:59.985086    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-s97xk"]
Sep 09 20:58:59.985210 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:58:59.985186    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-s97xk"
Sep 09 20:58:59.988344 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:58:59.988327    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\""
Sep 09 20:59:00.094080 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:59:00.094053    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:59:00.147083 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:59:00.147051    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k75hc\" (UniqueName: \"kubernetes.io/projected/f778fcb7-c329-423a-9a43-8530e67ad862-kube-api-access-k75hc\") pod \"keda-metrics-apiserver-77857b7b98-s97xk\" (UID: \"f778fcb7-c329-423a-9a43-8530e67ad862\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-s97xk"
Sep 09 20:59:00.147259 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:59:00.147098    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/f778fcb7-c329-423a-9a43-8530e67ad862-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-s97xk\" (UID: \"f778fcb7-c329-423a-9a43-8530e67ad862\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-s97xk"
Sep 09 20:59:00.147259 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:59:00.147125    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/f778fcb7-c329-423a-9a43-8530e67ad862-certificates\") pod \"keda-metrics-apiserver-77857b7b98-s97xk\" (UID: \"f778fcb7-c329-423a-9a43-8530e67ad862\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-s97xk"
Sep 09 20:59:00.247757 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:59:00.247673    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/f778fcb7-c329-423a-9a43-8530e67ad862-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-s97xk\" (UID: \"f778fcb7-c329-423a-9a43-8530e67ad862\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-s97xk"
Sep 09 20:59:00.247757 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:59:00.247724    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/f778fcb7-c329-423a-9a43-8530e67ad862-certificates\") pod \"keda-metrics-apiserver-77857b7b98-s97xk\" (UID: \"f778fcb7-c329-423a-9a43-8530e67ad862\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-s97xk"
Sep 09 20:59:00.247968 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:59:00.247785    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/2ecf51bc-7914-4aaf-b4aa-b1fbed72d1dd-certificates\") pod \"keda-operator-579c8958f7-5z57d\" (UID: \"2ecf51bc-7914-4aaf-b4aa-b1fbed72d1dd\") " pod="openshift-keda/keda-operator-579c8958f7-5z57d"
Sep 09 20:59:00.247968 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:59:00.247820    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k75hc\" (UniqueName: \"kubernetes.io/projected/f778fcb7-c329-423a-9a43-8530e67ad862-kube-api-access-k75hc\") pod \"keda-metrics-apiserver-77857b7b98-s97xk\" (UID: \"f778fcb7-c329-423a-9a43-8530e67ad862\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-s97xk"
Sep 09 20:59:00.248065 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:59:00.248044    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/f778fcb7-c329-423a-9a43-8530e67ad862-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-s97xk\" (UID: \"f778fcb7-c329-423a-9a43-8530e67ad862\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-s97xk"
Sep 09 20:59:00.250331 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:59:00.250310    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/2ecf51bc-7914-4aaf-b4aa-b1fbed72d1dd-certificates\") pod \"keda-operator-579c8958f7-5z57d\" (UID: \"2ecf51bc-7914-4aaf-b4aa-b1fbed72d1dd\") " pod="openshift-keda/keda-operator-579c8958f7-5z57d"
Sep 09 20:59:00.250533 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:59:00.250515    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/f778fcb7-c329-423a-9a43-8530e67ad862-certificates\") pod \"keda-metrics-apiserver-77857b7b98-s97xk\" (UID: \"f778fcb7-c329-423a-9a43-8530e67ad862\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-s97xk"
Sep 09 20:59:00.253282 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:59:00.253259    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:59:00.260151 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:59:00.260130    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k75hc\" (UniqueName: \"kubernetes.io/projected/f778fcb7-c329-423a-9a43-8530e67ad862-kube-api-access-k75hc\") pod \"keda-metrics-apiserver-77857b7b98-s97xk\" (UID: \"f778fcb7-c329-423a-9a43-8530e67ad862\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-s97xk"
Sep 09 20:59:00.263537 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:59:00.263517    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-dnqpw"]
Sep 09 20:59:00.266798 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:59:00.266783    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-dnqpw"
Sep 09 20:59:00.269863 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:59:00.269848    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Sep 09 20:59:00.287575 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:59:00.287552    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-dnqpw"]
Sep 09 20:59:00.296170 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:59:00.296152    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-s97xk"
Sep 09 20:59:00.346870 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:59:00.346674    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:59:00.348264 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:59:00.348230    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ws8rx\" (UniqueName: \"kubernetes.io/projected/086a4a06-b2fa-4b9d-b052-6f5f4015e573-kube-api-access-ws8rx\") pod \"keda-admission-6c4d864b86-dnqpw\" (UID: \"086a4a06-b2fa-4b9d-b052-6f5f4015e573\") " pod="openshift-keda/keda-admission-6c4d864b86-dnqpw"
Sep 09 20:59:00.348474 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:59:00.348282    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/086a4a06-b2fa-4b9d-b052-6f5f4015e573-certificates\") pod \"keda-admission-6c4d864b86-dnqpw\" (UID: \"086a4a06-b2fa-4b9d-b052-6f5f4015e573\") " pod="openshift-keda/keda-admission-6c4d864b86-dnqpw"
Sep 09 20:59:00.449442 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:59:00.449388    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ws8rx\" (UniqueName: \"kubernetes.io/projected/086a4a06-b2fa-4b9d-b052-6f5f4015e573-kube-api-access-ws8rx\") pod \"keda-admission-6c4d864b86-dnqpw\" (UID: \"086a4a06-b2fa-4b9d-b052-6f5f4015e573\") " pod="openshift-keda/keda-admission-6c4d864b86-dnqpw"
Sep 09 20:59:00.449442 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:59:00.449444    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/086a4a06-b2fa-4b9d-b052-6f5f4015e573-certificates\") pod \"keda-admission-6c4d864b86-dnqpw\" (UID: \"086a4a06-b2fa-4b9d-b052-6f5f4015e573\") " pod="openshift-keda/keda-admission-6c4d864b86-dnqpw"
Sep 09 20:59:00.449698 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:59:00.449571    2576 projected.go:266] Couldn't get secret openshift-keda/keda-admission-webhooks-certs: secret "keda-admission-webhooks-certs" not found
Sep 09 20:59:00.449698 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:59:00.449599    2576 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-admission-6c4d864b86-dnqpw: secret "keda-admission-webhooks-certs" not found
Sep 09 20:59:00.449698 ip-10-0-134-72 kubenswrapper[2576]: E0909 20:59:00.449660    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/086a4a06-b2fa-4b9d-b052-6f5f4015e573-certificates podName:086a4a06-b2fa-4b9d-b052-6f5f4015e573 nodeName:}" failed. No retries permitted until 2026-09-09 20:59:00.949641403 +0000 UTC m=+384.063327243 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/086a4a06-b2fa-4b9d-b052-6f5f4015e573-certificates") pod "keda-admission-6c4d864b86-dnqpw" (UID: "086a4a06-b2fa-4b9d-b052-6f5f4015e573") : secret "keda-admission-webhooks-certs" not found
Sep 09 20:59:00.461809 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:59:00.461783    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ws8rx\" (UniqueName: \"kubernetes.io/projected/086a4a06-b2fa-4b9d-b052-6f5f4015e573-kube-api-access-ws8rx\") pod \"keda-admission-6c4d864b86-dnqpw\" (UID: \"086a4a06-b2fa-4b9d-b052-6f5f4015e573\") " pod="openshift-keda/keda-admission-6c4d864b86-dnqpw"
Sep 09 20:59:00.475494 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:59:00.475470    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-5z57d"
Sep 09 20:59:00.573546 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:59:00.573520    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-s97xk"]
Sep 09 20:59:00.579770 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:59:00.579733    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf778fcb7_c329_423a_9a43_8530e67ad862.slice/crio-38d52dca3e8eae644737b06afcf697980e5393bdd5687220d7a287e1e87248b5 WatchSource:0}: Error finding container 38d52dca3e8eae644737b06afcf697980e5393bdd5687220d7a287e1e87248b5: Status 404 returned error can't find the container with id 38d52dca3e8eae644737b06afcf697980e5393bdd5687220d7a287e1e87248b5
Sep 09 20:59:00.747290 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:59:00.747264    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-5z57d"]
Sep 09 20:59:00.749550 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:59:00.749523    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ecf51bc_7914_4aaf_b4aa_b1fbed72d1dd.slice/crio-34b9fafd6961745358bd0778045455547217d25d1c41e5a74ab279d841227e0b WatchSource:0}: Error finding container 34b9fafd6961745358bd0778045455547217d25d1c41e5a74ab279d841227e0b: Status 404 returned error can't find the container with id 34b9fafd6961745358bd0778045455547217d25d1c41e5a74ab279d841227e0b
Sep 09 20:59:00.871252 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:59:00.871223    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:59:00.946610 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:59:00.946572    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-5z57d" event={"ID":"2ecf51bc-7914-4aaf-b4aa-b1fbed72d1dd","Type":"ContainerStarted","Data":"34b9fafd6961745358bd0778045455547217d25d1c41e5a74ab279d841227e0b"}
Sep 09 20:59:00.947750 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:59:00.947729    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-s97xk" event={"ID":"f778fcb7-c329-423a-9a43-8530e67ad862","Type":"ContainerStarted","Data":"38d52dca3e8eae644737b06afcf697980e5393bdd5687220d7a287e1e87248b5"}
Sep 09 20:59:00.955119 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:59:00.955098    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/086a4a06-b2fa-4b9d-b052-6f5f4015e573-certificates\") pod \"keda-admission-6c4d864b86-dnqpw\" (UID: \"086a4a06-b2fa-4b9d-b052-6f5f4015e573\") " pod="openshift-keda/keda-admission-6c4d864b86-dnqpw"
Sep 09 20:59:00.957676 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:59:00.957654    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/086a4a06-b2fa-4b9d-b052-6f5f4015e573-certificates\") pod \"keda-admission-6c4d864b86-dnqpw\" (UID: \"086a4a06-b2fa-4b9d-b052-6f5f4015e573\") " pod="openshift-keda/keda-admission-6c4d864b86-dnqpw"
Sep 09 20:59:00.969255 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:59:00.969232    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:59:01.099782 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:59:01.099748    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:59:01.180150 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:59:01.180067    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-dnqpw"
Sep 09 20:59:01.377903 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:59:01.377793    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:59:01.465090 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:59:01.464684    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-dnqpw"]
Sep 09 20:59:01.467965 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:59:01.467936    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:59:01.468507 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:59:01.468482    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod086a4a06_b2fa_4b9d_b052_6f5f4015e573.slice/crio-2078a78d2414daf38be584dca99aa55f9d2d46c9aa19f6f1a2fcde2574b7acec WatchSource:0}: Error finding container 2078a78d2414daf38be584dca99aa55f9d2d46c9aa19f6f1a2fcde2574b7acec: Status 404 returned error can't find the container with id 2078a78d2414daf38be584dca99aa55f9d2d46c9aa19f6f1a2fcde2574b7acec
Sep 09 20:59:01.954680 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:59:01.954641    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-dnqpw" event={"ID":"086a4a06-b2fa-4b9d-b052-6f5f4015e573","Type":"ContainerStarted","Data":"2078a78d2414daf38be584dca99aa55f9d2d46c9aa19f6f1a2fcde2574b7acec"}
Sep 09 20:59:04.971882 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:59:04.971784    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-dnqpw" event={"ID":"086a4a06-b2fa-4b9d-b052-6f5f4015e573","Type":"ContainerStarted","Data":"20d90b30dd49c1b404371480cec0b2a23d18580ff5eb1e95d7cb84482b17645c"}
Sep 09 20:59:04.972306 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:59:04.972110    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-6c4d864b86-dnqpw"
Sep 09 20:59:04.973339 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:59:04.973314    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-5z57d" event={"ID":"2ecf51bc-7914-4aaf-b4aa-b1fbed72d1dd","Type":"ContainerStarted","Data":"248aa4c76e64b7edee381fba4eecdbec72b1f9f18b513fe8557936f5f490726b"}
Sep 09 20:59:04.973468 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:59:04.973435    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-579c8958f7-5z57d"
Sep 09 20:59:04.974956 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:59:04.974933    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-s97xk" event={"ID":"f778fcb7-c329-423a-9a43-8530e67ad862","Type":"ContainerStarted","Data":"4b4f391d9d9715f90c29adcb3e3a2549e8599ef597b7ead4dfd2f3c98fa475e5"}
Sep 09 20:59:04.975150 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:59:04.975136    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-s97xk"
Sep 09 20:59:04.990469 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:59:04.990425    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-6c4d864b86-dnqpw" podStartSLOduration=1.717095319 podStartE2EDuration="4.990408244s" totalImagesPullingTime="3.273312925s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:59:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:59:04.988924719 +0000 UTC m=+388.102610585" watchObservedRunningTime="2026-09-09 20:59:04.990408244 +0000 UTC m=+388.104094101"
Sep 09 20:59:05.006066 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:59:05.006016    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-s97xk" podStartSLOduration=1.843683489 podStartE2EDuration="6.006003396s" totalImagesPullingTime="4.162319907s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:58:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:59:05.004827456 +0000 UTC m=+388.118513317" watchObservedRunningTime="2026-09-09 20:59:05.006003396 +0000 UTC m=+388.119689254"
Sep 09 20:59:05.022553 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:59:05.022500    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-operator-579c8958f7-5z57d" podStartSLOduration=2.019482114 podStartE2EDuration="6.022486518s" totalImagesPullingTime="4.003004404s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:58:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:59:05.021098748 +0000 UTC m=+388.134784607" watchObservedRunningTime="2026-09-09 20:59:05.022486518 +0000 UTC m=+388.136172376"
Sep 09 20:59:15.984968 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:59:15.984934    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-s97xk"
Sep 09 20:59:19.944240 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:59:19.944212    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-xbgh9"
Sep 09 20:59:25.982137 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:59:25.982092    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-579c8958f7-5z57d"
Sep 09 20:59:25.982660 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:59:25.982644    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-6c4d864b86-dnqpw"
Sep 09 20:59:51.663459 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:59:51.663424    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:59:51.767458 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:59:51.767421    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 20:59:51.926270 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:59:51.926192    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wm9rw"]
Sep 09 20:59:51.932751 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:59:51.932724    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wm9rw"
Sep 09 20:59:51.935675 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:59:51.935653    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 09 20:59:51.936440 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:59:51.936422    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-lbxks\""
Sep 09 20:59:51.936529 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:59:51.936445    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 09 20:59:51.940447 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:59:51.940427    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wm9rw"]
Sep 09 20:59:52.078052 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:59:52.078019    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjj2s\" (UniqueName: \"kubernetes.io/projected/c4cb8260-cc45-442a-9037-5cfad45c4d78-kube-api-access-vjj2s\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wm9rw\" (UID: \"c4cb8260-cc45-442a-9037-5cfad45c4d78\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wm9rw"
Sep 09 20:59:52.078218 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:59:52.078077    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c4cb8260-cc45-442a-9037-5cfad45c4d78-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wm9rw\" (UID: \"c4cb8260-cc45-442a-9037-5cfad45c4d78\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wm9rw"
Sep 09 20:59:52.078218 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:59:52.078097    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c4cb8260-cc45-442a-9037-5cfad45c4d78-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wm9rw\" (UID: \"c4cb8260-cc45-442a-9037-5cfad45c4d78\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wm9rw"
Sep 09 20:59:52.179317 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:59:52.179238    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c4cb8260-cc45-442a-9037-5cfad45c4d78-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wm9rw\" (UID: \"c4cb8260-cc45-442a-9037-5cfad45c4d78\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wm9rw"
Sep 09 20:59:52.179317 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:59:52.179268    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c4cb8260-cc45-442a-9037-5cfad45c4d78-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wm9rw\" (UID: \"c4cb8260-cc45-442a-9037-5cfad45c4d78\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wm9rw"
Sep 09 20:59:52.179515 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:59:52.179318    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjj2s\" (UniqueName: \"kubernetes.io/projected/c4cb8260-cc45-442a-9037-5cfad45c4d78-kube-api-access-vjj2s\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wm9rw\" (UID: \"c4cb8260-cc45-442a-9037-5cfad45c4d78\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wm9rw"
Sep 09 20:59:52.179671 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:59:52.179655    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c4cb8260-cc45-442a-9037-5cfad45c4d78-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wm9rw\" (UID: \"c4cb8260-cc45-442a-9037-5cfad45c4d78\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wm9rw"
Sep 09 20:59:52.179708 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:59:52.179684    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c4cb8260-cc45-442a-9037-5cfad45c4d78-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wm9rw\" (UID: \"c4cb8260-cc45-442a-9037-5cfad45c4d78\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wm9rw"
Sep 09 20:59:52.189325 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:59:52.189293    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjj2s\" (UniqueName: \"kubernetes.io/projected/c4cb8260-cc45-442a-9037-5cfad45c4d78-kube-api-access-vjj2s\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wm9rw\" (UID: \"c4cb8260-cc45-442a-9037-5cfad45c4d78\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wm9rw"
Sep 09 20:59:52.242512 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:59:52.242483    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wm9rw"
Sep 09 20:59:52.515299 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:59:52.515276    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wm9rw"]
Sep 09 20:59:52.517792 ip-10-0-134-72 kubenswrapper[2576]: W0909 20:59:52.517762    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4cb8260_cc45_442a_9037_5cfad45c4d78.slice/crio-b502679b503113990fb6b7bccc09a78d682d2a3131136b570fd43db3269410b6 WatchSource:0}: Error finding container b502679b503113990fb6b7bccc09a78d682d2a3131136b570fd43db3269410b6: Status 404 returned error can't find the container with id b502679b503113990fb6b7bccc09a78d682d2a3131136b570fd43db3269410b6
Sep 09 20:59:53.183562 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:59:53.183529    2576 generic.go:356] "Generic (PLEG): container finished" podID="c4cb8260-cc45-442a-9037-5cfad45c4d78" containerID="0c664fa988273bf0ba959a1bcb9dac14741df4d31a3083b1d76aa146b47d1dcc" exitCode=0
Sep 09 20:59:53.183932 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:59:53.183612    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wm9rw" event={"ID":"c4cb8260-cc45-442a-9037-5cfad45c4d78","Type":"ContainerDied","Data":"0c664fa988273bf0ba959a1bcb9dac14741df4d31a3083b1d76aa146b47d1dcc"}
Sep 09 20:59:53.183932 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:59:53.183659    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wm9rw" event={"ID":"c4cb8260-cc45-442a-9037-5cfad45c4d78","Type":"ContainerStarted","Data":"b502679b503113990fb6b7bccc09a78d682d2a3131136b570fd43db3269410b6"}
Sep 09 20:59:55.195066 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:59:55.195011    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wm9rw" event={"ID":"c4cb8260-cc45-442a-9037-5cfad45c4d78","Type":"ContainerStarted","Data":"baa362e76be1d469267ee9809f93e0c9644b1fb603cb7274ed857a3c01968793"}
Sep 09 20:59:56.201860 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:59:56.201820    2576 generic.go:356] "Generic (PLEG): container finished" podID="c4cb8260-cc45-442a-9037-5cfad45c4d78" containerID="baa362e76be1d469267ee9809f93e0c9644b1fb603cb7274ed857a3c01968793" exitCode=0
Sep 09 20:59:56.202226 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:59:56.201912    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wm9rw" event={"ID":"c4cb8260-cc45-442a-9037-5cfad45c4d78","Type":"ContainerDied","Data":"baa362e76be1d469267ee9809f93e0c9644b1fb603cb7274ed857a3c01968793"}
Sep 09 20:59:57.208588 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:59:57.208550    2576 generic.go:356] "Generic (PLEG): container finished" podID="c4cb8260-cc45-442a-9037-5cfad45c4d78" containerID="10348e002c71e2d7712144a730c0098bcef38476741be9b76810b6d931dfd717" exitCode=0
Sep 09 20:59:57.208958 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:59:57.208632    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wm9rw" event={"ID":"c4cb8260-cc45-442a-9037-5cfad45c4d78","Type":"ContainerDied","Data":"10348e002c71e2d7712144a730c0098bcef38476741be9b76810b6d931dfd717"}
Sep 09 20:59:58.341162 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:59:58.341139    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wm9rw"
Sep 09 20:59:58.432331 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:59:58.432305    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c4cb8260-cc45-442a-9037-5cfad45c4d78-util\") pod \"c4cb8260-cc45-442a-9037-5cfad45c4d78\" (UID: \"c4cb8260-cc45-442a-9037-5cfad45c4d78\") "
Sep 09 20:59:58.432473 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:59:58.432344    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vjj2s\" (UniqueName: \"kubernetes.io/projected/c4cb8260-cc45-442a-9037-5cfad45c4d78-kube-api-access-vjj2s\") pod \"c4cb8260-cc45-442a-9037-5cfad45c4d78\" (UID: \"c4cb8260-cc45-442a-9037-5cfad45c4d78\") "
Sep 09 20:59:58.432473 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:59:58.432395    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c4cb8260-cc45-442a-9037-5cfad45c4d78-bundle\") pod \"c4cb8260-cc45-442a-9037-5cfad45c4d78\" (UID: \"c4cb8260-cc45-442a-9037-5cfad45c4d78\") "
Sep 09 20:59:58.433152 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:59:58.433038    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4cb8260-cc45-442a-9037-5cfad45c4d78-bundle" (OuterVolumeSpecName: "bundle") pod "c4cb8260-cc45-442a-9037-5cfad45c4d78" (UID: "c4cb8260-cc45-442a-9037-5cfad45c4d78"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:59:58.434557 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:59:58.434531    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4cb8260-cc45-442a-9037-5cfad45c4d78-kube-api-access-vjj2s" (OuterVolumeSpecName: "kube-api-access-vjj2s") pod "c4cb8260-cc45-442a-9037-5cfad45c4d78" (UID: "c4cb8260-cc45-442a-9037-5cfad45c4d78"). InnerVolumeSpecName "kube-api-access-vjj2s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:59:58.437051 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:59:58.437033    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4cb8260-cc45-442a-9037-5cfad45c4d78-util" (OuterVolumeSpecName: "util") pod "c4cb8260-cc45-442a-9037-5cfad45c4d78" (UID: "c4cb8260-cc45-442a-9037-5cfad45c4d78"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:59:58.533352 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:59:58.533242    2576 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c4cb8260-cc45-442a-9037-5cfad45c4d78-util\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 20:59:58.533352 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:59:58.533277    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjj2s\" (UniqueName: \"kubernetes.io/projected/c4cb8260-cc45-442a-9037-5cfad45c4d78-kube-api-access-vjj2s\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 20:59:58.533352 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:59:58.533292    2576 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c4cb8260-cc45-442a-9037-5cfad45c4d78-bundle\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 20:59:59.219686 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:59:59.219658    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wm9rw"
Sep 09 20:59:59.219877 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:59:59.219658    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wm9rw" event={"ID":"c4cb8260-cc45-442a-9037-5cfad45c4d78","Type":"ContainerDied","Data":"b502679b503113990fb6b7bccc09a78d682d2a3131136b570fd43db3269410b6"}
Sep 09 20:59:59.219877 ip-10-0-134-72 kubenswrapper[2576]: I0909 20:59:59.219771    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b502679b503113990fb6b7bccc09a78d682d2a3131136b570fd43db3269410b6"
Sep 09 21:00:02.971430 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:02.971389    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:00:03.887186 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:03.887152    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:00:04.953884 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:04.953850    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-k4rkv"]
Sep 09 21:00:04.954246 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:04.954148    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4cb8260-cc45-442a-9037-5cfad45c4d78" containerName="util"
Sep 09 21:00:04.954246 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:04.954159    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4cb8260-cc45-442a-9037-5cfad45c4d78" containerName="util"
Sep 09 21:00:04.954246 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:04.954167    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4cb8260-cc45-442a-9037-5cfad45c4d78" containerName="extract"
Sep 09 21:00:04.954246 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:04.954173    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4cb8260-cc45-442a-9037-5cfad45c4d78" containerName="extract"
Sep 09 21:00:04.954246 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:04.954185    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4cb8260-cc45-442a-9037-5cfad45c4d78" containerName="pull"
Sep 09 21:00:04.954246 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:04.954191    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4cb8260-cc45-442a-9037-5cfad45c4d78" containerName="pull"
Sep 09 21:00:04.954246 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:04.954247    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4cb8260-cc45-442a-9037-5cfad45c4d78" containerName="extract"
Sep 09 21:00:04.957141 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:04.957125    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-k4rkv"
Sep 09 21:00:04.959905 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:04.959885    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Sep 09 21:00:04.960246 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:04.960230    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Sep 09 21:00:04.960358 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:04.960261    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-p2tc8\""
Sep 09 21:00:04.971320 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:04.971298    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-k4rkv"]
Sep 09 21:00:05.086977 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:05.086942    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kbkf\" (UniqueName: \"kubernetes.io/projected/9a251cab-b4fb-4dd7-8e00-97dff5476384-kube-api-access-4kbkf\") pod \"cert-manager-operator-controller-manager-684f79c9b9-k4rkv\" (UID: \"9a251cab-b4fb-4dd7-8e00-97dff5476384\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-k4rkv"
Sep 09 21:00:05.087145 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:05.087080    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9a251cab-b4fb-4dd7-8e00-97dff5476384-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-k4rkv\" (UID: \"9a251cab-b4fb-4dd7-8e00-97dff5476384\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-k4rkv"
Sep 09 21:00:05.188270 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:05.188233    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9a251cab-b4fb-4dd7-8e00-97dff5476384-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-k4rkv\" (UID: \"9a251cab-b4fb-4dd7-8e00-97dff5476384\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-k4rkv"
Sep 09 21:00:05.188447 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:05.188279    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kbkf\" (UniqueName: \"kubernetes.io/projected/9a251cab-b4fb-4dd7-8e00-97dff5476384-kube-api-access-4kbkf\") pod \"cert-manager-operator-controller-manager-684f79c9b9-k4rkv\" (UID: \"9a251cab-b4fb-4dd7-8e00-97dff5476384\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-k4rkv"
Sep 09 21:00:05.188720 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:05.188696    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9a251cab-b4fb-4dd7-8e00-97dff5476384-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-k4rkv\" (UID: \"9a251cab-b4fb-4dd7-8e00-97dff5476384\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-k4rkv"
Sep 09 21:00:05.202812 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:05.202781    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kbkf\" (UniqueName: \"kubernetes.io/projected/9a251cab-b4fb-4dd7-8e00-97dff5476384-kube-api-access-4kbkf\") pod \"cert-manager-operator-controller-manager-684f79c9b9-k4rkv\" (UID: \"9a251cab-b4fb-4dd7-8e00-97dff5476384\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-k4rkv"
Sep 09 21:00:05.265935 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:05.265844    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-k4rkv"
Sep 09 21:00:05.546881 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:05.546812    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-k4rkv"]
Sep 09 21:00:05.551163 ip-10-0-134-72 kubenswrapper[2576]: W0909 21:00:05.551138    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a251cab_b4fb_4dd7_8e00_97dff5476384.slice/crio-f26b8827ab8ff18a204ab4cf3663896c875f74b6c0e70f377e73e426f92f291d WatchSource:0}: Error finding container f26b8827ab8ff18a204ab4cf3663896c875f74b6c0e70f377e73e426f92f291d: Status 404 returned error can't find the container with id f26b8827ab8ff18a204ab4cf3663896c875f74b6c0e70f377e73e426f92f291d
Sep 09 21:00:06.251360 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:06.251317    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-k4rkv" event={"ID":"9a251cab-b4fb-4dd7-8e00-97dff5476384","Type":"ContainerStarted","Data":"f26b8827ab8ff18a204ab4cf3663896c875f74b6c0e70f377e73e426f92f291d"}
Sep 09 21:00:06.758762 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:06.758729    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:00:06.865015 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:06.864986    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:00:11.071507 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:11.071464    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:00:11.651758 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:11.651722    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7cb4cf479c-564b2"]
Sep 09 21:00:11.656071 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:11.656055    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7cb4cf479c-564b2"
Sep 09 21:00:11.669469 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:11.669449    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7cb4cf479c-564b2"]
Sep 09 21:00:11.713343 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:11.713311    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:00:11.738558 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:11.738535    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5vwd\" (UniqueName: \"kubernetes.io/projected/584b0d1e-fd29-4d8c-bff4-c268361542fe-kube-api-access-h5vwd\") pod \"console-7cb4cf479c-564b2\" (UID: \"584b0d1e-fd29-4d8c-bff4-c268361542fe\") " pod="openshift-console/console-7cb4cf479c-564b2"
Sep 09 21:00:11.738674 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:11.738600    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/584b0d1e-fd29-4d8c-bff4-c268361542fe-console-config\") pod \"console-7cb4cf479c-564b2\" (UID: \"584b0d1e-fd29-4d8c-bff4-c268361542fe\") " pod="openshift-console/console-7cb4cf479c-564b2"
Sep 09 21:00:11.738674 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:11.738638    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/584b0d1e-fd29-4d8c-bff4-c268361542fe-console-oauth-config\") pod \"console-7cb4cf479c-564b2\" (UID: \"584b0d1e-fd29-4d8c-bff4-c268361542fe\") " pod="openshift-console/console-7cb4cf479c-564b2"
Sep 09 21:00:11.738763 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:11.738732    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/584b0d1e-fd29-4d8c-bff4-c268361542fe-service-ca\") pod \"console-7cb4cf479c-564b2\" (UID: \"584b0d1e-fd29-4d8c-bff4-c268361542fe\") " pod="openshift-console/console-7cb4cf479c-564b2"
Sep 09 21:00:11.738763 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:11.738758    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/584b0d1e-fd29-4d8c-bff4-c268361542fe-console-serving-cert\") pod \"console-7cb4cf479c-564b2\" (UID: \"584b0d1e-fd29-4d8c-bff4-c268361542fe\") " pod="openshift-console/console-7cb4cf479c-564b2"
Sep 09 21:00:11.738839 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:11.738774    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/584b0d1e-fd29-4d8c-bff4-c268361542fe-trusted-ca-bundle\") pod \"console-7cb4cf479c-564b2\" (UID: \"584b0d1e-fd29-4d8c-bff4-c268361542fe\") " pod="openshift-console/console-7cb4cf479c-564b2"
Sep 09 21:00:11.738839 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:11.738813    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/584b0d1e-fd29-4d8c-bff4-c268361542fe-oauth-serving-cert\") pod \"console-7cb4cf479c-564b2\" (UID: \"584b0d1e-fd29-4d8c-bff4-c268361542fe\") " pod="openshift-console/console-7cb4cf479c-564b2"
Sep 09 21:00:11.815067 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:11.815036    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:00:11.839887 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:11.839863    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/584b0d1e-fd29-4d8c-bff4-c268361542fe-service-ca\") pod \"console-7cb4cf479c-564b2\" (UID: \"584b0d1e-fd29-4d8c-bff4-c268361542fe\") " pod="openshift-console/console-7cb4cf479c-564b2"
Sep 09 21:00:11.839976 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:11.839894    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/584b0d1e-fd29-4d8c-bff4-c268361542fe-console-serving-cert\") pod \"console-7cb4cf479c-564b2\" (UID: \"584b0d1e-fd29-4d8c-bff4-c268361542fe\") " pod="openshift-console/console-7cb4cf479c-564b2"
Sep 09 21:00:11.839976 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:11.839912    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/584b0d1e-fd29-4d8c-bff4-c268361542fe-trusted-ca-bundle\") pod \"console-7cb4cf479c-564b2\" (UID: \"584b0d1e-fd29-4d8c-bff4-c268361542fe\") " pod="openshift-console/console-7cb4cf479c-564b2"
Sep 09 21:00:11.839976 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:11.839948    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/584b0d1e-fd29-4d8c-bff4-c268361542fe-oauth-serving-cert\") pod \"console-7cb4cf479c-564b2\" (UID: \"584b0d1e-fd29-4d8c-bff4-c268361542fe\") " pod="openshift-console/console-7cb4cf479c-564b2"
Sep 09 21:00:11.840133 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:11.839988    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5vwd\" (UniqueName: \"kubernetes.io/projected/584b0d1e-fd29-4d8c-bff4-c268361542fe-kube-api-access-h5vwd\") pod \"console-7cb4cf479c-564b2\" (UID: \"584b0d1e-fd29-4d8c-bff4-c268361542fe\") " pod="openshift-console/console-7cb4cf479c-564b2"
Sep 09 21:00:11.840133 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:11.840005    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/584b0d1e-fd29-4d8c-bff4-c268361542fe-console-config\") pod \"console-7cb4cf479c-564b2\" (UID: \"584b0d1e-fd29-4d8c-bff4-c268361542fe\") " pod="openshift-console/console-7cb4cf479c-564b2"
Sep 09 21:00:11.840133 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:11.840022    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/584b0d1e-fd29-4d8c-bff4-c268361542fe-console-oauth-config\") pod \"console-7cb4cf479c-564b2\" (UID: \"584b0d1e-fd29-4d8c-bff4-c268361542fe\") " pod="openshift-console/console-7cb4cf479c-564b2"
Sep 09 21:00:11.840798 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:11.840767    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/584b0d1e-fd29-4d8c-bff4-c268361542fe-service-ca\") pod \"console-7cb4cf479c-564b2\" (UID: \"584b0d1e-fd29-4d8c-bff4-c268361542fe\") " pod="openshift-console/console-7cb4cf479c-564b2"
Sep 09 21:00:11.840896 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:11.840785    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/584b0d1e-fd29-4d8c-bff4-c268361542fe-oauth-serving-cert\") pod \"console-7cb4cf479c-564b2\" (UID: \"584b0d1e-fd29-4d8c-bff4-c268361542fe\") " pod="openshift-console/console-7cb4cf479c-564b2"
Sep 09 21:00:11.840985 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:11.840962    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/584b0d1e-fd29-4d8c-bff4-c268361542fe-console-config\") pod \"console-7cb4cf479c-564b2\" (UID: \"584b0d1e-fd29-4d8c-bff4-c268361542fe\") " pod="openshift-console/console-7cb4cf479c-564b2"
Sep 09 21:00:11.841024 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:11.840975    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/584b0d1e-fd29-4d8c-bff4-c268361542fe-trusted-ca-bundle\") pod \"console-7cb4cf479c-564b2\" (UID: \"584b0d1e-fd29-4d8c-bff4-c268361542fe\") " pod="openshift-console/console-7cb4cf479c-564b2"
Sep 09 21:00:11.842473 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:11.842446    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/584b0d1e-fd29-4d8c-bff4-c268361542fe-console-oauth-config\") pod \"console-7cb4cf479c-564b2\" (UID: \"584b0d1e-fd29-4d8c-bff4-c268361542fe\") " pod="openshift-console/console-7cb4cf479c-564b2"
Sep 09 21:00:11.842662 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:11.842641    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/584b0d1e-fd29-4d8c-bff4-c268361542fe-console-serving-cert\") pod \"console-7cb4cf479c-564b2\" (UID: \"584b0d1e-fd29-4d8c-bff4-c268361542fe\") " pod="openshift-console/console-7cb4cf479c-564b2"
Sep 09 21:00:11.847804 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:11.847787    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5vwd\" (UniqueName: \"kubernetes.io/projected/584b0d1e-fd29-4d8c-bff4-c268361542fe-kube-api-access-h5vwd\") pod \"console-7cb4cf479c-564b2\" (UID: \"584b0d1e-fd29-4d8c-bff4-c268361542fe\") " pod="openshift-console/console-7cb4cf479c-564b2"
Sep 09 21:00:11.965956 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:11.965879    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7cb4cf479c-564b2"
Sep 09 21:00:12.237957 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:12.237934    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7cb4cf479c-564b2"]
Sep 09 21:00:12.240025 ip-10-0-134-72 kubenswrapper[2576]: W0909 21:00:12.239999    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod584b0d1e_fd29_4d8c_bff4_c268361542fe.slice/crio-83e1489759b79f421dd9bcbbbc785aad40226d0d47e3423ff8003835f07bbb9d WatchSource:0}: Error finding container 83e1489759b79f421dd9bcbbbc785aad40226d0d47e3423ff8003835f07bbb9d: Status 404 returned error can't find the container with id 83e1489759b79f421dd9bcbbbc785aad40226d0d47e3423ff8003835f07bbb9d
Sep 09 21:00:12.283429 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:12.283401    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7cb4cf479c-564b2" event={"ID":"584b0d1e-fd29-4d8c-bff4-c268361542fe","Type":"ContainerStarted","Data":"83e1489759b79f421dd9bcbbbc785aad40226d0d47e3423ff8003835f07bbb9d"}
Sep 09 21:00:13.291388 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:13.291323    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7cb4cf479c-564b2" event={"ID":"584b0d1e-fd29-4d8c-bff4-c268361542fe","Type":"ContainerStarted","Data":"2bd4653ba7ad48e74a730cbaa6c6733d06610621423a0171f6d7a9d272e7e6b6"}
Sep 09 21:00:13.312822 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:13.312768    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7cb4cf479c-564b2" podStartSLOduration=2.31274794 podStartE2EDuration="2.31274794s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:00:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:00:13.31158691 +0000 UTC m=+456.425272780" watchObservedRunningTime="2026-09-09 21:00:13.31274794 +0000 UTC m=+456.426433800"
Sep 09 21:00:14.278181 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:14.278146    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:00:14.297972 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:14.297942    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-k4rkv" event={"ID":"9a251cab-b4fb-4dd7-8e00-97dff5476384","Type":"ContainerStarted","Data":"8f72210daad25e494804eab614630338a40565ba31e0029f9db0a9d35e8ff36b"}
Sep 09 21:00:14.320740 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:14.320684    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-k4rkv" podStartSLOduration=2.147455854 podStartE2EDuration="10.320665528s" totalImagesPullingTime="8.173209674s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:00:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:00:14.319236506 +0000 UTC m=+457.432922366" watchObservedRunningTime="2026-09-09 21:00:14.320665528 +0000 UTC m=+457.434351387"
Sep 09 21:00:14.378419 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:14.378388    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:00:16.496112 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:16.496075    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:00:16.672662 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:16.672575    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:00:16.850678 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:16.850640    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-fvcdd"]
Sep 09 21:00:16.854004 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:16.853980    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-fvcdd"
Sep 09 21:00:16.858139 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:16.858120    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 09 21:00:16.858246 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:16.858118    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-4ffbm\""
Sep 09 21:00:16.858246 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:16.858125    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 09 21:00:16.864277 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:16.864258    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-fvcdd"]
Sep 09 21:00:16.881889 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:16.881864    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/53087643-3d8e-4628-922c-35b9ec6acffa-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-fvcdd\" (UID: \"53087643-3d8e-4628-922c-35b9ec6acffa\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-fvcdd"
Sep 09 21:00:16.882005 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:16.881985    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzb5r\" (UniqueName: \"kubernetes.io/projected/53087643-3d8e-4628-922c-35b9ec6acffa-kube-api-access-kzb5r\") pod \"cert-manager-cainjector-559cdd8c87-fvcdd\" (UID: \"53087643-3d8e-4628-922c-35b9ec6acffa\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-fvcdd"
Sep 09 21:00:16.983300 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:16.983229    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzb5r\" (UniqueName: \"kubernetes.io/projected/53087643-3d8e-4628-922c-35b9ec6acffa-kube-api-access-kzb5r\") pod \"cert-manager-cainjector-559cdd8c87-fvcdd\" (UID: \"53087643-3d8e-4628-922c-35b9ec6acffa\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-fvcdd"
Sep 09 21:00:16.983300 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:16.983276    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/53087643-3d8e-4628-922c-35b9ec6acffa-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-fvcdd\" (UID: \"53087643-3d8e-4628-922c-35b9ec6acffa\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-fvcdd"
Sep 09 21:00:16.993792 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:16.993768    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/53087643-3d8e-4628-922c-35b9ec6acffa-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-fvcdd\" (UID: \"53087643-3d8e-4628-922c-35b9ec6acffa\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-fvcdd"
Sep 09 21:00:16.993981 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:16.993963    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzb5r\" (UniqueName: \"kubernetes.io/projected/53087643-3d8e-4628-922c-35b9ec6acffa-kube-api-access-kzb5r\") pod \"cert-manager-cainjector-559cdd8c87-fvcdd\" (UID: \"53087643-3d8e-4628-922c-35b9ec6acffa\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-fvcdd"
Sep 09 21:00:17.173649 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:17.173613    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-fvcdd"
Sep 09 21:00:17.445066 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:17.444944    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-fvcdd"]
Sep 09 21:00:17.447202 ip-10-0-134-72 kubenswrapper[2576]: W0909 21:00:17.447179    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod53087643_3d8e_4628_922c_35b9ec6acffa.slice/crio-eccf8c2e00b8bc3a6e2b08ea9ea302f15d3105fa12c07191b78e9009ef376bda WatchSource:0}: Error finding container eccf8c2e00b8bc3a6e2b08ea9ea302f15d3105fa12c07191b78e9009ef376bda: Status 404 returned error can't find the container with id eccf8c2e00b8bc3a6e2b08ea9ea302f15d3105fa12c07191b78e9009ef376bda
Sep 09 21:00:18.318166 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:18.318132    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-fvcdd" event={"ID":"53087643-3d8e-4628-922c-35b9ec6acffa","Type":"ContainerStarted","Data":"eccf8c2e00b8bc3a6e2b08ea9ea302f15d3105fa12c07191b78e9009ef376bda"}
Sep 09 21:00:18.422850 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:18.422806    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:00:18.523431 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:18.523396    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:00:18.622953 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:18.622908    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:00:19.502176 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:19.502145    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fql4q8"]
Sep 09 21:00:19.505865 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:19.505846    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fql4q8"
Sep 09 21:00:19.509709 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:19.509685    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 09 21:00:19.509709 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:19.509699    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 09 21:00:19.509874 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:19.509731    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-lbxks\""
Sep 09 21:00:19.519444 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:19.519412    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fql4q8"]
Sep 09 21:00:19.605083 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:19.605045    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/51fd7730-6860-4a8b-a1a8-41f2b37e2fe5-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fql4q8\" (UID: \"51fd7730-6860-4a8b-a1a8-41f2b37e2fe5\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fql4q8"
Sep 09 21:00:19.605267 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:19.605125    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sns7p\" (UniqueName: \"kubernetes.io/projected/51fd7730-6860-4a8b-a1a8-41f2b37e2fe5-kube-api-access-sns7p\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fql4q8\" (UID: \"51fd7730-6860-4a8b-a1a8-41f2b37e2fe5\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fql4q8"
Sep 09 21:00:19.605267 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:19.605183    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/51fd7730-6860-4a8b-a1a8-41f2b37e2fe5-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fql4q8\" (UID: \"51fd7730-6860-4a8b-a1a8-41f2b37e2fe5\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fql4q8"
Sep 09 21:00:19.706232 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:19.706195    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/51fd7730-6860-4a8b-a1a8-41f2b37e2fe5-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fql4q8\" (UID: \"51fd7730-6860-4a8b-a1a8-41f2b37e2fe5\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fql4q8"
Sep 09 21:00:19.706432 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:19.706280    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/51fd7730-6860-4a8b-a1a8-41f2b37e2fe5-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fql4q8\" (UID: \"51fd7730-6860-4a8b-a1a8-41f2b37e2fe5\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fql4q8"
Sep 09 21:00:19.706432 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:19.706345    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sns7p\" (UniqueName: \"kubernetes.io/projected/51fd7730-6860-4a8b-a1a8-41f2b37e2fe5-kube-api-access-sns7p\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fql4q8\" (UID: \"51fd7730-6860-4a8b-a1a8-41f2b37e2fe5\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fql4q8"
Sep 09 21:00:19.706704 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:19.706668    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/51fd7730-6860-4a8b-a1a8-41f2b37e2fe5-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fql4q8\" (UID: \"51fd7730-6860-4a8b-a1a8-41f2b37e2fe5\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fql4q8"
Sep 09 21:00:19.706704 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:19.706694    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/51fd7730-6860-4a8b-a1a8-41f2b37e2fe5-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fql4q8\" (UID: \"51fd7730-6860-4a8b-a1a8-41f2b37e2fe5\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fql4q8"
Sep 09 21:00:19.720567 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:19.720540    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sns7p\" (UniqueName: \"kubernetes.io/projected/51fd7730-6860-4a8b-a1a8-41f2b37e2fe5-kube-api-access-sns7p\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fql4q8\" (UID: \"51fd7730-6860-4a8b-a1a8-41f2b37e2fe5\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fql4q8"
Sep 09 21:00:19.817301 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:19.817221    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fql4q8"
Sep 09 21:00:20.328516 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:20.328484    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-fvcdd" event={"ID":"53087643-3d8e-4628-922c-35b9ec6acffa","Type":"ContainerStarted","Data":"99df553be4ae3b7c5287d3a44d2b52a12355922c41c9a04d7b94313aed30666f"}
Sep 09 21:00:20.348647 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:20.348598    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-fvcdd" podStartSLOduration=1.613605981 podStartE2EDuration="4.348583708s" totalImagesPullingTime="2.734977727s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:00:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:00:20.346960831 +0000 UTC m=+463.460646691" watchObservedRunningTime="2026-09-09 21:00:20.348583708 +0000 UTC m=+463.462269568"
Sep 09 21:00:20.389329 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:20.389258    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fql4q8"]
Sep 09 21:00:20.392921 ip-10-0-134-72 kubenswrapper[2576]: W0909 21:00:20.392890    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51fd7730_6860_4a8b_a1a8_41f2b37e2fe5.slice/crio-d95836646f3befe3469356508ffcafee54ca93c1c64992a6519d9d32a3d4d8ca WatchSource:0}: Error finding container d95836646f3befe3469356508ffcafee54ca93c1c64992a6519d9d32a3d4d8ca: Status 404 returned error can't find the container with id d95836646f3befe3469356508ffcafee54ca93c1c64992a6519d9d32a3d4d8ca
Sep 09 21:00:21.065487 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:21.065448    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:00:21.133092 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:21.133055    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:00:21.335432 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:21.335300    2576 generic.go:356] "Generic (PLEG): container finished" podID="51fd7730-6860-4a8b-a1a8-41f2b37e2fe5" containerID="2ed9184055622ce6cf634479e0938742454c798fd2ba3c6bb8fd1ac903be0aae" exitCode=0
Sep 09 21:00:21.335614 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:21.335414    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fql4q8" event={"ID":"51fd7730-6860-4a8b-a1a8-41f2b37e2fe5","Type":"ContainerDied","Data":"2ed9184055622ce6cf634479e0938742454c798fd2ba3c6bb8fd1ac903be0aae"}
Sep 09 21:00:21.335614 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:21.335515    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fql4q8" event={"ID":"51fd7730-6860-4a8b-a1a8-41f2b37e2fe5","Type":"ContainerStarted","Data":"d95836646f3befe3469356508ffcafee54ca93c1c64992a6519d9d32a3d4d8ca"}
Sep 09 21:00:21.966756 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:21.966724    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7cb4cf479c-564b2"
Sep 09 21:00:21.966938 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:21.966778    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7cb4cf479c-564b2"
Sep 09 21:00:21.971696 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:21.971672    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7cb4cf479c-564b2"
Sep 09 21:00:22.344389 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:22.344347    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7cb4cf479c-564b2"
Sep 09 21:00:22.404345 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:22.404311    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-865c7cb8cb-kk6xg"]
Sep 09 21:00:22.529228 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:22.529199    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:00:22.629921 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:22.629832    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:00:22.698104 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:22.698071    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:00:24.351981 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:24.351944    2576 generic.go:356] "Generic (PLEG): container finished" podID="51fd7730-6860-4a8b-a1a8-41f2b37e2fe5" containerID="6a55ef1bdce6ff81a27afd51b861538efc802b24ffea3016001310d476db17d7" exitCode=0
Sep 09 21:00:24.352508 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:24.351994    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fql4q8" event={"ID":"51fd7730-6860-4a8b-a1a8-41f2b37e2fe5","Type":"ContainerDied","Data":"6a55ef1bdce6ff81a27afd51b861538efc802b24ffea3016001310d476db17d7"}
Sep 09 21:00:24.881233 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:24.881197    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:00:25.359593 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:25.359560    2576 generic.go:356] "Generic (PLEG): container finished" podID="51fd7730-6860-4a8b-a1a8-41f2b37e2fe5" containerID="74f6ddcd6e304c769bbaec8fa1e5d789091dcbaece02efd02f4d9f34086ae53c" exitCode=0
Sep 09 21:00:25.359939 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:25.359607    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fql4q8" event={"ID":"51fd7730-6860-4a8b-a1a8-41f2b37e2fe5","Type":"ContainerDied","Data":"74f6ddcd6e304c769bbaec8fa1e5d789091dcbaece02efd02f4d9f34086ae53c"}
Sep 09 21:00:26.491095 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:26.491072    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fql4q8"
Sep 09 21:00:26.563902 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:26.563874    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sns7p\" (UniqueName: \"kubernetes.io/projected/51fd7730-6860-4a8b-a1a8-41f2b37e2fe5-kube-api-access-sns7p\") pod \"51fd7730-6860-4a8b-a1a8-41f2b37e2fe5\" (UID: \"51fd7730-6860-4a8b-a1a8-41f2b37e2fe5\") "
Sep 09 21:00:26.564092 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:26.563984    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/51fd7730-6860-4a8b-a1a8-41f2b37e2fe5-bundle\") pod \"51fd7730-6860-4a8b-a1a8-41f2b37e2fe5\" (UID: \"51fd7730-6860-4a8b-a1a8-41f2b37e2fe5\") "
Sep 09 21:00:26.564092 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:26.564041    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/51fd7730-6860-4a8b-a1a8-41f2b37e2fe5-util\") pod \"51fd7730-6860-4a8b-a1a8-41f2b37e2fe5\" (UID: \"51fd7730-6860-4a8b-a1a8-41f2b37e2fe5\") "
Sep 09 21:00:26.564427 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:26.564404    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51fd7730-6860-4a8b-a1a8-41f2b37e2fe5-bundle" (OuterVolumeSpecName: "bundle") pod "51fd7730-6860-4a8b-a1a8-41f2b37e2fe5" (UID: "51fd7730-6860-4a8b-a1a8-41f2b37e2fe5"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:00:26.566112 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:26.566081    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51fd7730-6860-4a8b-a1a8-41f2b37e2fe5-kube-api-access-sns7p" (OuterVolumeSpecName: "kube-api-access-sns7p") pod "51fd7730-6860-4a8b-a1a8-41f2b37e2fe5" (UID: "51fd7730-6860-4a8b-a1a8-41f2b37e2fe5"). InnerVolumeSpecName "kube-api-access-sns7p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:00:26.568088 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:26.568066    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51fd7730-6860-4a8b-a1a8-41f2b37e2fe5-util" (OuterVolumeSpecName: "util") pod "51fd7730-6860-4a8b-a1a8-41f2b37e2fe5" (UID: "51fd7730-6860-4a8b-a1a8-41f2b37e2fe5"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:00:26.665507 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:26.665432    2576 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/51fd7730-6860-4a8b-a1a8-41f2b37e2fe5-bundle\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:00:26.665507 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:26.665456    2576 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/51fd7730-6860-4a8b-a1a8-41f2b37e2fe5-util\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:00:26.665507 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:26.665465    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sns7p\" (UniqueName: \"kubernetes.io/projected/51fd7730-6860-4a8b-a1a8-41f2b37e2fe5-kube-api-access-sns7p\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:00:27.371220 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:27.371190    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fql4q8" event={"ID":"51fd7730-6860-4a8b-a1a8-41f2b37e2fe5","Type":"ContainerDied","Data":"d95836646f3befe3469356508ffcafee54ca93c1c64992a6519d9d32a3d4d8ca"}
Sep 09 21:00:27.371403 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:27.371231    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fql4q8"
Sep 09 21:00:27.371403 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:27.371242    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d95836646f3befe3469356508ffcafee54ca93c1c64992a6519d9d32a3d4d8ca"
Sep 09 21:00:27.596182 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:27.596130    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:00:32.116824 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:32.116782    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:00:32.216807 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:32.216753    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:00:42.275348 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:42.275300    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:00:46.945699 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:46.945656    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:00:47.031673 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:47.031631    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:00:47.424355 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:47.424314    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-865c7cb8cb-kk6xg" podUID="228889d4-a02e-4f1e-bc6d-efb354b1d7e0" containerName="console" containerID="cri-o://d4ace748662f2789899ea27e746fce9ff3f8db5a5ad12849ce39e8a7caf41803" gracePeriod=15
Sep 09 21:00:47.671986 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:47.671966    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-865c7cb8cb-kk6xg"
Sep 09 21:00:47.745057 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:47.744980    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lflr2\" (UniqueName: \"kubernetes.io/projected/228889d4-a02e-4f1e-bc6d-efb354b1d7e0-kube-api-access-lflr2\") pod \"228889d4-a02e-4f1e-bc6d-efb354b1d7e0\" (UID: \"228889d4-a02e-4f1e-bc6d-efb354b1d7e0\") "
Sep 09 21:00:47.745057 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:47.745039    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/228889d4-a02e-4f1e-bc6d-efb354b1d7e0-oauth-serving-cert\") pod \"228889d4-a02e-4f1e-bc6d-efb354b1d7e0\" (UID: \"228889d4-a02e-4f1e-bc6d-efb354b1d7e0\") "
Sep 09 21:00:47.745245 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:47.745063    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/228889d4-a02e-4f1e-bc6d-efb354b1d7e0-trusted-ca-bundle\") pod \"228889d4-a02e-4f1e-bc6d-efb354b1d7e0\" (UID: \"228889d4-a02e-4f1e-bc6d-efb354b1d7e0\") "
Sep 09 21:00:47.745245 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:47.745093    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/228889d4-a02e-4f1e-bc6d-efb354b1d7e0-console-config\") pod \"228889d4-a02e-4f1e-bc6d-efb354b1d7e0\" (UID: \"228889d4-a02e-4f1e-bc6d-efb354b1d7e0\") "
Sep 09 21:00:47.745245 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:47.745108    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/228889d4-a02e-4f1e-bc6d-efb354b1d7e0-service-ca\") pod \"228889d4-a02e-4f1e-bc6d-efb354b1d7e0\" (UID: \"228889d4-a02e-4f1e-bc6d-efb354b1d7e0\") "
Sep 09 21:00:47.745245 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:47.745144    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/228889d4-a02e-4f1e-bc6d-efb354b1d7e0-console-serving-cert\") pod \"228889d4-a02e-4f1e-bc6d-efb354b1d7e0\" (UID: \"228889d4-a02e-4f1e-bc6d-efb354b1d7e0\") "
Sep 09 21:00:47.745245 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:47.745168    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/228889d4-a02e-4f1e-bc6d-efb354b1d7e0-console-oauth-config\") pod \"228889d4-a02e-4f1e-bc6d-efb354b1d7e0\" (UID: \"228889d4-a02e-4f1e-bc6d-efb354b1d7e0\") "
Sep 09 21:00:47.745607 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:47.745574    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/228889d4-a02e-4f1e-bc6d-efb354b1d7e0-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "228889d4-a02e-4f1e-bc6d-efb354b1d7e0" (UID: "228889d4-a02e-4f1e-bc6d-efb354b1d7e0"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:00:47.745722 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:47.745626    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/228889d4-a02e-4f1e-bc6d-efb354b1d7e0-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "228889d4-a02e-4f1e-bc6d-efb354b1d7e0" (UID: "228889d4-a02e-4f1e-bc6d-efb354b1d7e0"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:00:47.745722 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:47.745670    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/228889d4-a02e-4f1e-bc6d-efb354b1d7e0-console-config" (OuterVolumeSpecName: "console-config") pod "228889d4-a02e-4f1e-bc6d-efb354b1d7e0" (UID: "228889d4-a02e-4f1e-bc6d-efb354b1d7e0"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:00:47.745722 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:47.745678    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/228889d4-a02e-4f1e-bc6d-efb354b1d7e0-service-ca" (OuterVolumeSpecName: "service-ca") pod "228889d4-a02e-4f1e-bc6d-efb354b1d7e0" (UID: "228889d4-a02e-4f1e-bc6d-efb354b1d7e0"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:00:47.747423 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:47.747393    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/228889d4-a02e-4f1e-bc6d-efb354b1d7e0-kube-api-access-lflr2" (OuterVolumeSpecName: "kube-api-access-lflr2") pod "228889d4-a02e-4f1e-bc6d-efb354b1d7e0" (UID: "228889d4-a02e-4f1e-bc6d-efb354b1d7e0"). InnerVolumeSpecName "kube-api-access-lflr2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:00:47.747576 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:47.747555    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/228889d4-a02e-4f1e-bc6d-efb354b1d7e0-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "228889d4-a02e-4f1e-bc6d-efb354b1d7e0" (UID: "228889d4-a02e-4f1e-bc6d-efb354b1d7e0"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:00:47.747626 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:47.747571    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/228889d4-a02e-4f1e-bc6d-efb354b1d7e0-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "228889d4-a02e-4f1e-bc6d-efb354b1d7e0" (UID: "228889d4-a02e-4f1e-bc6d-efb354b1d7e0"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:00:47.846317 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:47.846267    2576 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/228889d4-a02e-4f1e-bc6d-efb354b1d7e0-oauth-serving-cert\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:00:47.846317 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:47.846311    2576 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/228889d4-a02e-4f1e-bc6d-efb354b1d7e0-trusted-ca-bundle\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:00:47.846317 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:47.846322    2576 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/228889d4-a02e-4f1e-bc6d-efb354b1d7e0-console-config\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:00:47.846317 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:47.846334    2576 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/228889d4-a02e-4f1e-bc6d-efb354b1d7e0-service-ca\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:00:47.846604 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:47.846342    2576 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/228889d4-a02e-4f1e-bc6d-efb354b1d7e0-console-serving-cert\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:00:47.846604 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:47.846350    2576 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/228889d4-a02e-4f1e-bc6d-efb354b1d7e0-console-oauth-config\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:00:47.846604 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:47.846359    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lflr2\" (UniqueName: \"kubernetes.io/projected/228889d4-a02e-4f1e-bc6d-efb354b1d7e0-kube-api-access-lflr2\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:00:48.464578 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:48.464545    2576 generic.go:356] "Generic (PLEG): container finished" podID="228889d4-a02e-4f1e-bc6d-efb354b1d7e0" containerID="d4ace748662f2789899ea27e746fce9ff3f8db5a5ad12849ce39e8a7caf41803" exitCode=0
Sep 09 21:00:48.464961 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:48.464589    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-865c7cb8cb-kk6xg"
Sep 09 21:00:48.464961 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:48.464634    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-865c7cb8cb-kk6xg" event={"ID":"228889d4-a02e-4f1e-bc6d-efb354b1d7e0","Type":"ContainerDied","Data":"d4ace748662f2789899ea27e746fce9ff3f8db5a5ad12849ce39e8a7caf41803"}
Sep 09 21:00:48.464961 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:48.464686    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-865c7cb8cb-kk6xg" event={"ID":"228889d4-a02e-4f1e-bc6d-efb354b1d7e0","Type":"ContainerDied","Data":"2ed896d455b8f2bd647f5a347caa67f78f5eae4c13b58d432b69bc71a6ceb1e2"}
Sep 09 21:00:48.464961 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:48.464711    2576 scope.go:117] "RemoveContainer" containerID="d4ace748662f2789899ea27e746fce9ff3f8db5a5ad12849ce39e8a7caf41803"
Sep 09 21:00:48.473529 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:48.473512    2576 scope.go:117] "RemoveContainer" containerID="d4ace748662f2789899ea27e746fce9ff3f8db5a5ad12849ce39e8a7caf41803"
Sep 09 21:00:48.473769 ip-10-0-134-72 kubenswrapper[2576]: E0909 21:00:48.473755    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4ace748662f2789899ea27e746fce9ff3f8db5a5ad12849ce39e8a7caf41803\": container with ID starting with d4ace748662f2789899ea27e746fce9ff3f8db5a5ad12849ce39e8a7caf41803 not found: ID does not exist" containerID="d4ace748662f2789899ea27e746fce9ff3f8db5a5ad12849ce39e8a7caf41803"
Sep 09 21:00:48.473807 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:48.473777    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4ace748662f2789899ea27e746fce9ff3f8db5a5ad12849ce39e8a7caf41803"} err="failed to get container status \"d4ace748662f2789899ea27e746fce9ff3f8db5a5ad12849ce39e8a7caf41803\": rpc error: code = NotFound desc = could not find container \"d4ace748662f2789899ea27e746fce9ff3f8db5a5ad12849ce39e8a7caf41803\": container with ID starting with d4ace748662f2789899ea27e746fce9ff3f8db5a5ad12849ce39e8a7caf41803 not found: ID does not exist"
Sep 09 21:00:48.502264 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:48.502237    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-865c7cb8cb-kk6xg"]
Sep 09 21:00:48.516615 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:48.516591    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-865c7cb8cb-kk6xg"]
Sep 09 21:00:48.581746 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:48.581715    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:00:49.409509 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:49.409474    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="228889d4-a02e-4f1e-bc6d-efb354b1d7e0" path="/var/lib/kubelet/pods/228889d4-a02e-4f1e-bc6d-efb354b1d7e0/volumes"
Sep 09 21:00:50.608540 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:50.608510    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-85b69767b4-kqss7"]
Sep 09 21:00:50.608906 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:50.608817    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51fd7730-6860-4a8b-a1a8-41f2b37e2fe5" containerName="extract"
Sep 09 21:00:50.608906 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:50.608827    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="51fd7730-6860-4a8b-a1a8-41f2b37e2fe5" containerName="extract"
Sep 09 21:00:50.608906 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:50.608836    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="228889d4-a02e-4f1e-bc6d-efb354b1d7e0" containerName="console"
Sep 09 21:00:50.608906 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:50.608841    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="228889d4-a02e-4f1e-bc6d-efb354b1d7e0" containerName="console"
Sep 09 21:00:50.608906 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:50.608849    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51fd7730-6860-4a8b-a1a8-41f2b37e2fe5" containerName="pull"
Sep 09 21:00:50.608906 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:50.608856    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="51fd7730-6860-4a8b-a1a8-41f2b37e2fe5" containerName="pull"
Sep 09 21:00:50.608906 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:50.608868    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51fd7730-6860-4a8b-a1a8-41f2b37e2fe5" containerName="util"
Sep 09 21:00:50.608906 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:50.608873    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="51fd7730-6860-4a8b-a1a8-41f2b37e2fe5" containerName="util"
Sep 09 21:00:50.609137 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:50.608923    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="51fd7730-6860-4a8b-a1a8-41f2b37e2fe5" containerName="extract"
Sep 09 21:00:50.609137 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:50.608933    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="228889d4-a02e-4f1e-bc6d-efb354b1d7e0" containerName="console"
Sep 09 21:00:50.612151 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:50.612136    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-85b69767b4-kqss7"
Sep 09 21:00:50.614633 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:50.614615    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 09 21:00:50.614850 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:50.614833    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 09 21:00:50.615613 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:50.615598    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 09 21:00:50.615762 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:50.615746    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 09 21:00:50.615961 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:50.615946    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 09 21:00:50.617263 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:50.617249    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-6kknb\""
Sep 09 21:00:50.622240 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:50.622215    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-85b69767b4-kqss7"]
Sep 09 21:00:50.768945 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:50.768912    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vp8f6\" (UniqueName: \"kubernetes.io/projected/60ac6fa1-b9fb-41dc-a64d-09c22503038c-kube-api-access-vp8f6\") pod \"lws-controller-manager-85b69767b4-kqss7\" (UID: \"60ac6fa1-b9fb-41dc-a64d-09c22503038c\") " pod="openshift-lws-operator/lws-controller-manager-85b69767b4-kqss7"
Sep 09 21:00:50.769129 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:50.768956    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/60ac6fa1-b9fb-41dc-a64d-09c22503038c-manager-config\") pod \"lws-controller-manager-85b69767b4-kqss7\" (UID: \"60ac6fa1-b9fb-41dc-a64d-09c22503038c\") " pod="openshift-lws-operator/lws-controller-manager-85b69767b4-kqss7"
Sep 09 21:00:50.769129 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:50.768981    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/60ac6fa1-b9fb-41dc-a64d-09c22503038c-cert\") pod \"lws-controller-manager-85b69767b4-kqss7\" (UID: \"60ac6fa1-b9fb-41dc-a64d-09c22503038c\") " pod="openshift-lws-operator/lws-controller-manager-85b69767b4-kqss7"
Sep 09 21:00:50.769129 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:50.768995    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/60ac6fa1-b9fb-41dc-a64d-09c22503038c-metrics-cert\") pod \"lws-controller-manager-85b69767b4-kqss7\" (UID: \"60ac6fa1-b9fb-41dc-a64d-09c22503038c\") " pod="openshift-lws-operator/lws-controller-manager-85b69767b4-kqss7"
Sep 09 21:00:50.870322 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:50.870245    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vp8f6\" (UniqueName: \"kubernetes.io/projected/60ac6fa1-b9fb-41dc-a64d-09c22503038c-kube-api-access-vp8f6\") pod \"lws-controller-manager-85b69767b4-kqss7\" (UID: \"60ac6fa1-b9fb-41dc-a64d-09c22503038c\") " pod="openshift-lws-operator/lws-controller-manager-85b69767b4-kqss7"
Sep 09 21:00:50.870322 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:50.870290    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/60ac6fa1-b9fb-41dc-a64d-09c22503038c-manager-config\") pod \"lws-controller-manager-85b69767b4-kqss7\" (UID: \"60ac6fa1-b9fb-41dc-a64d-09c22503038c\") " pod="openshift-lws-operator/lws-controller-manager-85b69767b4-kqss7"
Sep 09 21:00:50.870322 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:50.870317    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/60ac6fa1-b9fb-41dc-a64d-09c22503038c-cert\") pod \"lws-controller-manager-85b69767b4-kqss7\" (UID: \"60ac6fa1-b9fb-41dc-a64d-09c22503038c\") " pod="openshift-lws-operator/lws-controller-manager-85b69767b4-kqss7"
Sep 09 21:00:50.870624 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:50.870332    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/60ac6fa1-b9fb-41dc-a64d-09c22503038c-metrics-cert\") pod \"lws-controller-manager-85b69767b4-kqss7\" (UID: \"60ac6fa1-b9fb-41dc-a64d-09c22503038c\") " pod="openshift-lws-operator/lws-controller-manager-85b69767b4-kqss7"
Sep 09 21:00:50.870998 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:50.870967    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/60ac6fa1-b9fb-41dc-a64d-09c22503038c-manager-config\") pod \"lws-controller-manager-85b69767b4-kqss7\" (UID: \"60ac6fa1-b9fb-41dc-a64d-09c22503038c\") " pod="openshift-lws-operator/lws-controller-manager-85b69767b4-kqss7"
Sep 09 21:00:50.873027 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:50.873005    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/60ac6fa1-b9fb-41dc-a64d-09c22503038c-cert\") pod \"lws-controller-manager-85b69767b4-kqss7\" (UID: \"60ac6fa1-b9fb-41dc-a64d-09c22503038c\") " pod="openshift-lws-operator/lws-controller-manager-85b69767b4-kqss7"
Sep 09 21:00:50.873127 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:50.873035    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/60ac6fa1-b9fb-41dc-a64d-09c22503038c-metrics-cert\") pod \"lws-controller-manager-85b69767b4-kqss7\" (UID: \"60ac6fa1-b9fb-41dc-a64d-09c22503038c\") " pod="openshift-lws-operator/lws-controller-manager-85b69767b4-kqss7"
Sep 09 21:00:50.884334 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:50.884309    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vp8f6\" (UniqueName: \"kubernetes.io/projected/60ac6fa1-b9fb-41dc-a64d-09c22503038c-kube-api-access-vp8f6\") pod \"lws-controller-manager-85b69767b4-kqss7\" (UID: \"60ac6fa1-b9fb-41dc-a64d-09c22503038c\") " pod="openshift-lws-operator/lws-controller-manager-85b69767b4-kqss7"
Sep 09 21:00:50.921972 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:50.921950    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-85b69767b4-kqss7"
Sep 09 21:00:51.192318 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:51.192291    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-85b69767b4-kqss7"]
Sep 09 21:00:51.194725 ip-10-0-134-72 kubenswrapper[2576]: W0909 21:00:51.194703    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60ac6fa1_b9fb_41dc_a64d_09c22503038c.slice/crio-bf02f82398d83368957986f7c34792adf9340023cacfbec3ac93565bcd8ad3b6 WatchSource:0}: Error finding container bf02f82398d83368957986f7c34792adf9340023cacfbec3ac93565bcd8ad3b6: Status 404 returned error can't find the container with id bf02f82398d83368957986f7c34792adf9340023cacfbec3ac93565bcd8ad3b6
Sep 09 21:00:51.224442 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:51.224412    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:00:51.325639 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:51.325606    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:00:51.482226 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:51.482153    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-85b69767b4-kqss7" event={"ID":"60ac6fa1-b9fb-41dc-a64d-09c22503038c","Type":"ContainerStarted","Data":"bf02f82398d83368957986f7c34792adf9340023cacfbec3ac93565bcd8ad3b6"}
Sep 09 21:00:53.493848 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:53.493818    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-85b69767b4-kqss7" event={"ID":"60ac6fa1-b9fb-41dc-a64d-09c22503038c","Type":"ContainerStarted","Data":"aa7529959144c167eb005eccde661a7ff981df500d2d918d11dd4a87645bfc8b"}
Sep 09 21:00:53.494229 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:53.493877    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-85b69767b4-kqss7"
Sep 09 21:00:53.536817 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:53.536767    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-85b69767b4-kqss7" podStartSLOduration=1.84900418 podStartE2EDuration="3.536752029s" totalImagesPullingTime="1.687747849s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:00:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:00:53.536225994 +0000 UTC m=+496.649911852" watchObservedRunningTime="2026-09-09 21:00:53.536752029 +0000 UTC m=+496.650437942"
Sep 09 21:00:53.792944 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:53.792856    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:00:53.886570 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:53.886532    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:00:57.357730 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:00:57.357694    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:01:01.308635 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:01.308598    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:01:01.405231 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:01.405198    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:01:04.501101 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:04.501070    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-85b69767b4-kqss7"
Sep 09 21:01:05.043486 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:05.043444    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpvgmk"]
Sep 09 21:01:05.046882 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:05.046859    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpvgmk"
Sep 09 21:01:05.049312 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:05.049286    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 09 21:01:05.049466 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:05.049444    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 09 21:01:05.049557 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:05.049483    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-lbxks\""
Sep 09 21:01:05.056899 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:05.056881    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpvgmk"]
Sep 09 21:01:05.092210 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:05.092184    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rk6wd\" (UniqueName: \"kubernetes.io/projected/fa0a535c-3226-43e7-96df-0d9372908774-kube-api-access-rk6wd\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpvgmk\" (UID: \"fa0a535c-3226-43e7-96df-0d9372908774\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpvgmk"
Sep 09 21:01:05.092315 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:05.092215    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fa0a535c-3226-43e7-96df-0d9372908774-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpvgmk\" (UID: \"fa0a535c-3226-43e7-96df-0d9372908774\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpvgmk"
Sep 09 21:01:05.092315 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:05.092282    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fa0a535c-3226-43e7-96df-0d9372908774-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpvgmk\" (UID: \"fa0a535c-3226-43e7-96df-0d9372908774\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpvgmk"
Sep 09 21:01:05.128979 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:05.128950    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6t2glg"]
Sep 09 21:01:05.132393 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:05.132359    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6t2glg"
Sep 09 21:01:05.141819 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:05.141795    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6t2glg"]
Sep 09 21:01:05.193403 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:05.193351    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rk6wd\" (UniqueName: \"kubernetes.io/projected/fa0a535c-3226-43e7-96df-0d9372908774-kube-api-access-rk6wd\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpvgmk\" (UID: \"fa0a535c-3226-43e7-96df-0d9372908774\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpvgmk"
Sep 09 21:01:05.193512 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:05.193418    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fa0a535c-3226-43e7-96df-0d9372908774-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpvgmk\" (UID: \"fa0a535c-3226-43e7-96df-0d9372908774\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpvgmk"
Sep 09 21:01:05.193512 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:05.193468    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/50d6e3f2-1fe7-4697-87ef-cbb615ae4aae-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6t2glg\" (UID: \"50d6e3f2-1fe7-4697-87ef-cbb615ae4aae\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6t2glg"
Sep 09 21:01:05.193512 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:05.193495    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpb24\" (UniqueName: \"kubernetes.io/projected/50d6e3f2-1fe7-4697-87ef-cbb615ae4aae-kube-api-access-qpb24\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6t2glg\" (UID: \"50d6e3f2-1fe7-4697-87ef-cbb615ae4aae\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6t2glg"
Sep 09 21:01:05.193684 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:05.193597    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/50d6e3f2-1fe7-4697-87ef-cbb615ae4aae-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6t2glg\" (UID: \"50d6e3f2-1fe7-4697-87ef-cbb615ae4aae\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6t2glg"
Sep 09 21:01:05.193684 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:05.193647    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fa0a535c-3226-43e7-96df-0d9372908774-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpvgmk\" (UID: \"fa0a535c-3226-43e7-96df-0d9372908774\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpvgmk"
Sep 09 21:01:05.193839 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:05.193818    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fa0a535c-3226-43e7-96df-0d9372908774-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpvgmk\" (UID: \"fa0a535c-3226-43e7-96df-0d9372908774\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpvgmk"
Sep 09 21:01:05.193899 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:05.193886    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fa0a535c-3226-43e7-96df-0d9372908774-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpvgmk\" (UID: \"fa0a535c-3226-43e7-96df-0d9372908774\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpvgmk"
Sep 09 21:01:05.202177 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:05.202159    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rk6wd\" (UniqueName: \"kubernetes.io/projected/fa0a535c-3226-43e7-96df-0d9372908774-kube-api-access-rk6wd\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpvgmk\" (UID: \"fa0a535c-3226-43e7-96df-0d9372908774\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpvgmk"
Sep 09 21:01:05.231982 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:05.231952    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5mnw5"]
Sep 09 21:01:05.235467 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:05.235449    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5mnw5"
Sep 09 21:01:05.245319 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:05.245300    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5mnw5"]
Sep 09 21:01:05.294599 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:05.294543    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s88b8\" (UniqueName: \"kubernetes.io/projected/771c548e-b359-4ec8-af21-b68f97a6a1fa-kube-api-access-s88b8\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5mnw5\" (UID: \"771c548e-b359-4ec8-af21-b68f97a6a1fa\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5mnw5"
Sep 09 21:01:05.294599 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:05.294585    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/50d6e3f2-1fe7-4697-87ef-cbb615ae4aae-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6t2glg\" (UID: \"50d6e3f2-1fe7-4697-87ef-cbb615ae4aae\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6t2glg"
Sep 09 21:01:05.294725 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:05.294603    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpb24\" (UniqueName: \"kubernetes.io/projected/50d6e3f2-1fe7-4697-87ef-cbb615ae4aae-kube-api-access-qpb24\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6t2glg\" (UID: \"50d6e3f2-1fe7-4697-87ef-cbb615ae4aae\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6t2glg"
Sep 09 21:01:05.294725 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:05.294635    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/771c548e-b359-4ec8-af21-b68f97a6a1fa-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5mnw5\" (UID: \"771c548e-b359-4ec8-af21-b68f97a6a1fa\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5mnw5"
Sep 09 21:01:05.294725 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:05.294654    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/771c548e-b359-4ec8-af21-b68f97a6a1fa-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5mnw5\" (UID: \"771c548e-b359-4ec8-af21-b68f97a6a1fa\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5mnw5"
Sep 09 21:01:05.294725 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:05.294688    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/50d6e3f2-1fe7-4697-87ef-cbb615ae4aae-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6t2glg\" (UID: \"50d6e3f2-1fe7-4697-87ef-cbb615ae4aae\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6t2glg"
Sep 09 21:01:05.294883 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:05.294866    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/50d6e3f2-1fe7-4697-87ef-cbb615ae4aae-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6t2glg\" (UID: \"50d6e3f2-1fe7-4697-87ef-cbb615ae4aae\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6t2glg"
Sep 09 21:01:05.294930 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:05.294917    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/50d6e3f2-1fe7-4697-87ef-cbb615ae4aae-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6t2glg\" (UID: \"50d6e3f2-1fe7-4697-87ef-cbb615ae4aae\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6t2glg"
Sep 09 21:01:05.304984 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:05.304963    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpb24\" (UniqueName: \"kubernetes.io/projected/50d6e3f2-1fe7-4697-87ef-cbb615ae4aae-kube-api-access-qpb24\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6t2glg\" (UID: \"50d6e3f2-1fe7-4697-87ef-cbb615ae4aae\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6t2glg"
Sep 09 21:01:05.329991 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:05.329971    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c2fd2"]
Sep 09 21:01:05.333639 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:05.333622    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c2fd2"
Sep 09 21:01:05.345637 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:05.345615    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c2fd2"]
Sep 09 21:01:05.356995 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:05.356974    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpvgmk"
Sep 09 21:01:05.395142 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:05.395115    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/07652310-10fd-4d42-a8ff-e70294b9037d-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c2fd2\" (UID: \"07652310-10fd-4d42-a8ff-e70294b9037d\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c2fd2"
Sep 09 21:01:05.395296 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:05.395155    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s88b8\" (UniqueName: \"kubernetes.io/projected/771c548e-b359-4ec8-af21-b68f97a6a1fa-kube-api-access-s88b8\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5mnw5\" (UID: \"771c548e-b359-4ec8-af21-b68f97a6a1fa\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5mnw5"
Sep 09 21:01:05.395296 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:05.395175    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcnsr\" (UniqueName: \"kubernetes.io/projected/07652310-10fd-4d42-a8ff-e70294b9037d-kube-api-access-kcnsr\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c2fd2\" (UID: \"07652310-10fd-4d42-a8ff-e70294b9037d\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c2fd2"
Sep 09 21:01:05.395296 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:05.395204    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/07652310-10fd-4d42-a8ff-e70294b9037d-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c2fd2\" (UID: \"07652310-10fd-4d42-a8ff-e70294b9037d\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c2fd2"
Sep 09 21:01:05.395500 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:05.395317    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/771c548e-b359-4ec8-af21-b68f97a6a1fa-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5mnw5\" (UID: \"771c548e-b359-4ec8-af21-b68f97a6a1fa\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5mnw5"
Sep 09 21:01:05.395500 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:05.395386    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/771c548e-b359-4ec8-af21-b68f97a6a1fa-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5mnw5\" (UID: \"771c548e-b359-4ec8-af21-b68f97a6a1fa\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5mnw5"
Sep 09 21:01:05.395780 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:05.395754    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/771c548e-b359-4ec8-af21-b68f97a6a1fa-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5mnw5\" (UID: \"771c548e-b359-4ec8-af21-b68f97a6a1fa\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5mnw5"
Sep 09 21:01:05.395818 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:05.395761    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/771c548e-b359-4ec8-af21-b68f97a6a1fa-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5mnw5\" (UID: \"771c548e-b359-4ec8-af21-b68f97a6a1fa\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5mnw5"
Sep 09 21:01:05.405745 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:05.405696    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s88b8\" (UniqueName: \"kubernetes.io/projected/771c548e-b359-4ec8-af21-b68f97a6a1fa-kube-api-access-s88b8\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5mnw5\" (UID: \"771c548e-b359-4ec8-af21-b68f97a6a1fa\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5mnw5"
Sep 09 21:01:05.440887 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:05.440860    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6t2glg"
Sep 09 21:01:05.496146 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:05.496115    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/07652310-10fd-4d42-a8ff-e70294b9037d-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c2fd2\" (UID: \"07652310-10fd-4d42-a8ff-e70294b9037d\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c2fd2"
Sep 09 21:01:05.496235 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:05.496178    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcnsr\" (UniqueName: \"kubernetes.io/projected/07652310-10fd-4d42-a8ff-e70294b9037d-kube-api-access-kcnsr\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c2fd2\" (UID: \"07652310-10fd-4d42-a8ff-e70294b9037d\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c2fd2"
Sep 09 21:01:05.496235 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:05.496207    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/07652310-10fd-4d42-a8ff-e70294b9037d-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c2fd2\" (UID: \"07652310-10fd-4d42-a8ff-e70294b9037d\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c2fd2"
Sep 09 21:01:05.496527 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:05.496505    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/07652310-10fd-4d42-a8ff-e70294b9037d-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c2fd2\" (UID: \"07652310-10fd-4d42-a8ff-e70294b9037d\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c2fd2"
Sep 09 21:01:05.496627 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:05.496606    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/07652310-10fd-4d42-a8ff-e70294b9037d-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c2fd2\" (UID: \"07652310-10fd-4d42-a8ff-e70294b9037d\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c2fd2"
Sep 09 21:01:05.505434 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:05.505414    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcnsr\" (UniqueName: \"kubernetes.io/projected/07652310-10fd-4d42-a8ff-e70294b9037d-kube-api-access-kcnsr\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c2fd2\" (UID: \"07652310-10fd-4d42-a8ff-e70294b9037d\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c2fd2"
Sep 09 21:01:05.545261 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:05.545206    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5mnw5"
Sep 09 21:01:05.629979 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:05.629877    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpvgmk"]
Sep 09 21:01:05.632319 ip-10-0-134-72 kubenswrapper[2576]: W0909 21:01:05.632289    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa0a535c_3226_43e7_96df_0d9372908774.slice/crio-5584f3a5022c5ae87a1c4c791335b3e85417506c5608f1e026c15e2ecdc63bf7 WatchSource:0}: Error finding container 5584f3a5022c5ae87a1c4c791335b3e85417506c5608f1e026c15e2ecdc63bf7: Status 404 returned error can't find the container with id 5584f3a5022c5ae87a1c4c791335b3e85417506c5608f1e026c15e2ecdc63bf7
Sep 09 21:01:05.642645 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:05.642625    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c2fd2"
Sep 09 21:01:05.720398 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:05.719870    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6t2glg"]
Sep 09 21:01:05.816585 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:05.816560    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5mnw5"]
Sep 09 21:01:05.819662 ip-10-0-134-72 kubenswrapper[2576]: W0909 21:01:05.819639    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod771c548e_b359_4ec8_af21_b68f97a6a1fa.slice/crio-07df11626da71cc1300e7f416af703d7461e6aaf034079ee65b84dec22d53bc3 WatchSource:0}: Error finding container 07df11626da71cc1300e7f416af703d7461e6aaf034079ee65b84dec22d53bc3: Status 404 returned error can't find the container with id 07df11626da71cc1300e7f416af703d7461e6aaf034079ee65b84dec22d53bc3
Sep 09 21:01:05.922467 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:05.922445    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c2fd2"]
Sep 09 21:01:05.969173 ip-10-0-134-72 kubenswrapper[2576]: W0909 21:01:05.969147    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07652310_10fd_4d42_a8ff_e70294b9037d.slice/crio-3b1357dfedcdb331c96363141e83ad5af3b639f52179caf107401b166364bd1e WatchSource:0}: Error finding container 3b1357dfedcdb331c96363141e83ad5af3b639f52179caf107401b166364bd1e: Status 404 returned error can't find the container with id 3b1357dfedcdb331c96363141e83ad5af3b639f52179caf107401b166364bd1e
Sep 09 21:01:06.551888 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:06.551848    2576 generic.go:356] "Generic (PLEG): container finished" podID="50d6e3f2-1fe7-4697-87ef-cbb615ae4aae" containerID="ff81be77ec5e2fcf67afe26bf9007222a14ff21db98257a010ed9dbe91923742" exitCode=0
Sep 09 21:01:06.552319 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:06.551936    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6t2glg" event={"ID":"50d6e3f2-1fe7-4697-87ef-cbb615ae4aae","Type":"ContainerDied","Data":"ff81be77ec5e2fcf67afe26bf9007222a14ff21db98257a010ed9dbe91923742"}
Sep 09 21:01:06.552319 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:06.551996    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6t2glg" event={"ID":"50d6e3f2-1fe7-4697-87ef-cbb615ae4aae","Type":"ContainerStarted","Data":"acbd7e1eb424ec2e497fe95278a742ad60c8bc89924f0d02f5b803f72187c1d8"}
Sep 09 21:01:06.553701 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:06.553679    2576 generic.go:356] "Generic (PLEG): container finished" podID="07652310-10fd-4d42-a8ff-e70294b9037d" containerID="f36157119783948739d355ff30cfd1129188286d1239290dea9f969a3219b773" exitCode=0
Sep 09 21:01:06.553803 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:06.553706    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c2fd2" event={"ID":"07652310-10fd-4d42-a8ff-e70294b9037d","Type":"ContainerDied","Data":"f36157119783948739d355ff30cfd1129188286d1239290dea9f969a3219b773"}
Sep 09 21:01:06.553803 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:06.553766    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c2fd2" event={"ID":"07652310-10fd-4d42-a8ff-e70294b9037d","Type":"ContainerStarted","Data":"3b1357dfedcdb331c96363141e83ad5af3b639f52179caf107401b166364bd1e"}
Sep 09 21:01:06.555572 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:06.555455    2576 generic.go:356] "Generic (PLEG): container finished" podID="771c548e-b359-4ec8-af21-b68f97a6a1fa" containerID="93d8656146320e5d4acd2b3f71f6878ecc63dce676bb4fd1a0cf150d38ba620a" exitCode=0
Sep 09 21:01:06.555572 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:06.555539    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5mnw5" event={"ID":"771c548e-b359-4ec8-af21-b68f97a6a1fa","Type":"ContainerDied","Data":"93d8656146320e5d4acd2b3f71f6878ecc63dce676bb4fd1a0cf150d38ba620a"}
Sep 09 21:01:06.555702 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:06.555581    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5mnw5" event={"ID":"771c548e-b359-4ec8-af21-b68f97a6a1fa","Type":"ContainerStarted","Data":"07df11626da71cc1300e7f416af703d7461e6aaf034079ee65b84dec22d53bc3"}
Sep 09 21:01:06.557285 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:06.557267    2576 generic.go:356] "Generic (PLEG): container finished" podID="fa0a535c-3226-43e7-96df-0d9372908774" containerID="732458558b8b75e889fe46425749ee6d1887fe004ccf5639962bca75208805f2" exitCode=0
Sep 09 21:01:06.557398 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:06.557356    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpvgmk" event={"ID":"fa0a535c-3226-43e7-96df-0d9372908774","Type":"ContainerDied","Data":"732458558b8b75e889fe46425749ee6d1887fe004ccf5639962bca75208805f2"}
Sep 09 21:01:06.557465 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:06.557425    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpvgmk" event={"ID":"fa0a535c-3226-43e7-96df-0d9372908774","Type":"ContainerStarted","Data":"5584f3a5022c5ae87a1c4c791335b3e85417506c5608f1e026c15e2ecdc63bf7"}
Sep 09 21:01:07.566925 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:07.566893    2576 generic.go:356] "Generic (PLEG): container finished" podID="07652310-10fd-4d42-a8ff-e70294b9037d" containerID="8cd65ded13db49f83d01dc4368c388ef4ce4cc9354cf2e9b7cbcb39ac3a410bd" exitCode=0
Sep 09 21:01:07.567257 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:07.567026    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c2fd2" event={"ID":"07652310-10fd-4d42-a8ff-e70294b9037d","Type":"ContainerDied","Data":"8cd65ded13db49f83d01dc4368c388ef4ce4cc9354cf2e9b7cbcb39ac3a410bd"}
Sep 09 21:01:08.573835 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:08.573799    2576 generic.go:356] "Generic (PLEG): container finished" podID="50d6e3f2-1fe7-4697-87ef-cbb615ae4aae" containerID="eb6e85337145c65aac9d3497c659ed3355e62d4d245aa8944d6f47ce8e0a0408" exitCode=0
Sep 09 21:01:08.574224 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:08.573888    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6t2glg" event={"ID":"50d6e3f2-1fe7-4697-87ef-cbb615ae4aae","Type":"ContainerDied","Data":"eb6e85337145c65aac9d3497c659ed3355e62d4d245aa8944d6f47ce8e0a0408"}
Sep 09 21:01:08.576239 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:08.576216    2576 generic.go:356] "Generic (PLEG): container finished" podID="07652310-10fd-4d42-a8ff-e70294b9037d" containerID="1763799b8af21e208b5972a913625efddfadbe07a6cedc18508e5904fe053649" exitCode=0
Sep 09 21:01:08.576329 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:08.576281    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c2fd2" event={"ID":"07652310-10fd-4d42-a8ff-e70294b9037d","Type":"ContainerDied","Data":"1763799b8af21e208b5972a913625efddfadbe07a6cedc18508e5904fe053649"}
Sep 09 21:01:08.577965 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:08.577945    2576 generic.go:356] "Generic (PLEG): container finished" podID="771c548e-b359-4ec8-af21-b68f97a6a1fa" containerID="ee1efacc4c94f7eedb6dd81926d44ad82a759a818bec1c1944c51c6c0f8dd625" exitCode=0
Sep 09 21:01:08.578042 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:08.578015    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5mnw5" event={"ID":"771c548e-b359-4ec8-af21-b68f97a6a1fa","Type":"ContainerDied","Data":"ee1efacc4c94f7eedb6dd81926d44ad82a759a818bec1c1944c51c6c0f8dd625"}
Sep 09 21:01:08.580199 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:08.580180    2576 generic.go:356] "Generic (PLEG): container finished" podID="fa0a535c-3226-43e7-96df-0d9372908774" containerID="3a95c5f959d9606595b9b9eb599c61be959bc808fb65dc48d8f4b483dfa8db1d" exitCode=0
Sep 09 21:01:08.580286 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:08.580217    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpvgmk" event={"ID":"fa0a535c-3226-43e7-96df-0d9372908774","Type":"ContainerDied","Data":"3a95c5f959d9606595b9b9eb599c61be959bc808fb65dc48d8f4b483dfa8db1d"}
Sep 09 21:01:08.751793 ip-10-0-134-72 kubenswrapper[2576]: E0909 21:01:08.751762    2576 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50d6e3f2_1fe7_4697_87ef_cbb615ae4aae.slice/crio-37ffa8456204c6822fbe4f2d2891d1c4dd0ff896133a6d6174dc5e6511592d6f.scope\": RecentStats: unable to find data in memory cache]"
Sep 09 21:01:09.084073 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:09.083984    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:01:09.160703 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:09.160674    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:01:09.587204 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:09.587169    2576 generic.go:356] "Generic (PLEG): container finished" podID="50d6e3f2-1fe7-4697-87ef-cbb615ae4aae" containerID="37ffa8456204c6822fbe4f2d2891d1c4dd0ff896133a6d6174dc5e6511592d6f" exitCode=0
Sep 09 21:01:09.587620 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:09.587251    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6t2glg" event={"ID":"50d6e3f2-1fe7-4697-87ef-cbb615ae4aae","Type":"ContainerDied","Data":"37ffa8456204c6822fbe4f2d2891d1c4dd0ff896133a6d6174dc5e6511592d6f"}
Sep 09 21:01:09.589395 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:09.589351    2576 generic.go:356] "Generic (PLEG): container finished" podID="771c548e-b359-4ec8-af21-b68f97a6a1fa" containerID="ab5cf3cbe7e938115252b11d22ffe758045415c32e0c4daf09b1b2b1117b5e6e" exitCode=0
Sep 09 21:01:09.589556 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:09.589444    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5mnw5" event={"ID":"771c548e-b359-4ec8-af21-b68f97a6a1fa","Type":"ContainerDied","Data":"ab5cf3cbe7e938115252b11d22ffe758045415c32e0c4daf09b1b2b1117b5e6e"}
Sep 09 21:01:09.591301 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:09.591281    2576 generic.go:356] "Generic (PLEG): container finished" podID="fa0a535c-3226-43e7-96df-0d9372908774" containerID="5ef7d453d37c9aa6f16caf98ab800230a74ecef5c4b72bfc419676c7e962f330" exitCode=0
Sep 09 21:01:09.591412 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:09.591349    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpvgmk" event={"ID":"fa0a535c-3226-43e7-96df-0d9372908774","Type":"ContainerDied","Data":"5ef7d453d37c9aa6f16caf98ab800230a74ecef5c4b72bfc419676c7e962f330"}
Sep 09 21:01:09.718043 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:09.718016    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c2fd2"
Sep 09 21:01:09.735591 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:09.735566    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/07652310-10fd-4d42-a8ff-e70294b9037d-util\") pod \"07652310-10fd-4d42-a8ff-e70294b9037d\" (UID: \"07652310-10fd-4d42-a8ff-e70294b9037d\") "
Sep 09 21:01:09.735737 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:09.735666    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/07652310-10fd-4d42-a8ff-e70294b9037d-bundle\") pod \"07652310-10fd-4d42-a8ff-e70294b9037d\" (UID: \"07652310-10fd-4d42-a8ff-e70294b9037d\") "
Sep 09 21:01:09.735737 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:09.735713    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kcnsr\" (UniqueName: \"kubernetes.io/projected/07652310-10fd-4d42-a8ff-e70294b9037d-kube-api-access-kcnsr\") pod \"07652310-10fd-4d42-a8ff-e70294b9037d\" (UID: \"07652310-10fd-4d42-a8ff-e70294b9037d\") "
Sep 09 21:01:09.736082 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:09.736056    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07652310-10fd-4d42-a8ff-e70294b9037d-bundle" (OuterVolumeSpecName: "bundle") pod "07652310-10fd-4d42-a8ff-e70294b9037d" (UID: "07652310-10fd-4d42-a8ff-e70294b9037d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:01:09.738040 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:09.738017    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07652310-10fd-4d42-a8ff-e70294b9037d-kube-api-access-kcnsr" (OuterVolumeSpecName: "kube-api-access-kcnsr") pod "07652310-10fd-4d42-a8ff-e70294b9037d" (UID: "07652310-10fd-4d42-a8ff-e70294b9037d"). InnerVolumeSpecName "kube-api-access-kcnsr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:01:09.741312 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:09.741278    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07652310-10fd-4d42-a8ff-e70294b9037d-util" (OuterVolumeSpecName: "util") pod "07652310-10fd-4d42-a8ff-e70294b9037d" (UID: "07652310-10fd-4d42-a8ff-e70294b9037d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:01:09.836766 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:09.836736    2576 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/07652310-10fd-4d42-a8ff-e70294b9037d-bundle\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:01:09.836766 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:09.836760    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kcnsr\" (UniqueName: \"kubernetes.io/projected/07652310-10fd-4d42-a8ff-e70294b9037d-kube-api-access-kcnsr\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:01:09.836766 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:09.836770    2576 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/07652310-10fd-4d42-a8ff-e70294b9037d-util\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:01:10.597544 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:10.597496    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c2fd2"
Sep 09 21:01:10.597544 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:10.597531    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c2fd2" event={"ID":"07652310-10fd-4d42-a8ff-e70294b9037d","Type":"ContainerDied","Data":"3b1357dfedcdb331c96363141e83ad5af3b639f52179caf107401b166364bd1e"}
Sep 09 21:01:10.597995 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:10.597573    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3b1357dfedcdb331c96363141e83ad5af3b639f52179caf107401b166364bd1e"
Sep 09 21:01:10.769417 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:10.769397    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6t2glg"
Sep 09 21:01:10.772105 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:10.772085    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5mnw5"
Sep 09 21:01:10.793393 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:10.793361    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpvgmk"
Sep 09 21:01:10.845938 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:10.845912    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fa0a535c-3226-43e7-96df-0d9372908774-util\") pod \"fa0a535c-3226-43e7-96df-0d9372908774\" (UID: \"fa0a535c-3226-43e7-96df-0d9372908774\") "
Sep 09 21:01:10.846079 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:10.845960    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/50d6e3f2-1fe7-4697-87ef-cbb615ae4aae-bundle\") pod \"50d6e3f2-1fe7-4697-87ef-cbb615ae4aae\" (UID: \"50d6e3f2-1fe7-4697-87ef-cbb615ae4aae\") "
Sep 09 21:01:10.846079 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:10.845984    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s88b8\" (UniqueName: \"kubernetes.io/projected/771c548e-b359-4ec8-af21-b68f97a6a1fa-kube-api-access-s88b8\") pod \"771c548e-b359-4ec8-af21-b68f97a6a1fa\" (UID: \"771c548e-b359-4ec8-af21-b68f97a6a1fa\") "
Sep 09 21:01:10.846079 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:10.846018    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/50d6e3f2-1fe7-4697-87ef-cbb615ae4aae-util\") pod \"50d6e3f2-1fe7-4697-87ef-cbb615ae4aae\" (UID: \"50d6e3f2-1fe7-4697-87ef-cbb615ae4aae\") "
Sep 09 21:01:10.846079 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:10.846035    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/771c548e-b359-4ec8-af21-b68f97a6a1fa-util\") pod \"771c548e-b359-4ec8-af21-b68f97a6a1fa\" (UID: \"771c548e-b359-4ec8-af21-b68f97a6a1fa\") "
Sep 09 21:01:10.846079 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:10.846062    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/771c548e-b359-4ec8-af21-b68f97a6a1fa-bundle\") pod \"771c548e-b359-4ec8-af21-b68f97a6a1fa\" (UID: \"771c548e-b359-4ec8-af21-b68f97a6a1fa\") "
Sep 09 21:01:10.846337 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:10.846093    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rk6wd\" (UniqueName: \"kubernetes.io/projected/fa0a535c-3226-43e7-96df-0d9372908774-kube-api-access-rk6wd\") pod \"fa0a535c-3226-43e7-96df-0d9372908774\" (UID: \"fa0a535c-3226-43e7-96df-0d9372908774\") "
Sep 09 21:01:10.846337 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:10.846151    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qpb24\" (UniqueName: \"kubernetes.io/projected/50d6e3f2-1fe7-4697-87ef-cbb615ae4aae-kube-api-access-qpb24\") pod \"50d6e3f2-1fe7-4697-87ef-cbb615ae4aae\" (UID: \"50d6e3f2-1fe7-4697-87ef-cbb615ae4aae\") "
Sep 09 21:01:10.846337 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:10.846186    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fa0a535c-3226-43e7-96df-0d9372908774-bundle\") pod \"fa0a535c-3226-43e7-96df-0d9372908774\" (UID: \"fa0a535c-3226-43e7-96df-0d9372908774\") "
Sep 09 21:01:10.846613 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:10.846587    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50d6e3f2-1fe7-4697-87ef-cbb615ae4aae-bundle" (OuterVolumeSpecName: "bundle") pod "50d6e3f2-1fe7-4697-87ef-cbb615ae4aae" (UID: "50d6e3f2-1fe7-4697-87ef-cbb615ae4aae"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:01:10.846977 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:10.846956    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa0a535c-3226-43e7-96df-0d9372908774-bundle" (OuterVolumeSpecName: "bundle") pod "fa0a535c-3226-43e7-96df-0d9372908774" (UID: "fa0a535c-3226-43e7-96df-0d9372908774"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:01:10.847523 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:10.847480    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/771c548e-b359-4ec8-af21-b68f97a6a1fa-bundle" (OuterVolumeSpecName: "bundle") pod "771c548e-b359-4ec8-af21-b68f97a6a1fa" (UID: "771c548e-b359-4ec8-af21-b68f97a6a1fa"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:01:10.848612 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:10.848557    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/771c548e-b359-4ec8-af21-b68f97a6a1fa-kube-api-access-s88b8" (OuterVolumeSpecName: "kube-api-access-s88b8") pod "771c548e-b359-4ec8-af21-b68f97a6a1fa" (UID: "771c548e-b359-4ec8-af21-b68f97a6a1fa"). InnerVolumeSpecName "kube-api-access-s88b8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:01:10.848788 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:10.848771    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50d6e3f2-1fe7-4697-87ef-cbb615ae4aae-kube-api-access-qpb24" (OuterVolumeSpecName: "kube-api-access-qpb24") pod "50d6e3f2-1fe7-4697-87ef-cbb615ae4aae" (UID: "50d6e3f2-1fe7-4697-87ef-cbb615ae4aae"). InnerVolumeSpecName "kube-api-access-qpb24". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:01:10.849334 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:10.849309    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa0a535c-3226-43e7-96df-0d9372908774-kube-api-access-rk6wd" (OuterVolumeSpecName: "kube-api-access-rk6wd") pod "fa0a535c-3226-43e7-96df-0d9372908774" (UID: "fa0a535c-3226-43e7-96df-0d9372908774"). InnerVolumeSpecName "kube-api-access-rk6wd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:01:10.852716 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:10.852693    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/771c548e-b359-4ec8-af21-b68f97a6a1fa-util" (OuterVolumeSpecName: "util") pod "771c548e-b359-4ec8-af21-b68f97a6a1fa" (UID: "771c548e-b359-4ec8-af21-b68f97a6a1fa"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:01:10.854627 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:10.854606    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50d6e3f2-1fe7-4697-87ef-cbb615ae4aae-util" (OuterVolumeSpecName: "util") pod "50d6e3f2-1fe7-4697-87ef-cbb615ae4aae" (UID: "50d6e3f2-1fe7-4697-87ef-cbb615ae4aae"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:01:10.930250 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:10.930223    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa0a535c-3226-43e7-96df-0d9372908774-util" (OuterVolumeSpecName: "util") pod "fa0a535c-3226-43e7-96df-0d9372908774" (UID: "fa0a535c-3226-43e7-96df-0d9372908774"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:01:10.947168 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:10.947149    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qpb24\" (UniqueName: \"kubernetes.io/projected/50d6e3f2-1fe7-4697-87ef-cbb615ae4aae-kube-api-access-qpb24\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:01:10.947233 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:10.947170    2576 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fa0a535c-3226-43e7-96df-0d9372908774-bundle\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:01:10.947233 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:10.947179    2576 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fa0a535c-3226-43e7-96df-0d9372908774-util\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:01:10.947233 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:10.947186    2576 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/50d6e3f2-1fe7-4697-87ef-cbb615ae4aae-bundle\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:01:10.947233 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:10.947194    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s88b8\" (UniqueName: \"kubernetes.io/projected/771c548e-b359-4ec8-af21-b68f97a6a1fa-kube-api-access-s88b8\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:01:10.947233 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:10.947202    2576 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/50d6e3f2-1fe7-4697-87ef-cbb615ae4aae-util\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:01:10.947233 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:10.947211    2576 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/771c548e-b359-4ec8-af21-b68f97a6a1fa-util\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:01:10.947233 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:10.947218    2576 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/771c548e-b359-4ec8-af21-b68f97a6a1fa-bundle\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:01:10.947233 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:10.947225    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rk6wd\" (UniqueName: \"kubernetes.io/projected/fa0a535c-3226-43e7-96df-0d9372908774-kube-api-access-rk6wd\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:01:11.605526 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:11.605498    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpvgmk"
Sep 09 21:01:11.605526 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:11.605505    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpvgmk" event={"ID":"fa0a535c-3226-43e7-96df-0d9372908774","Type":"ContainerDied","Data":"5584f3a5022c5ae87a1c4c791335b3e85417506c5608f1e026c15e2ecdc63bf7"}
Sep 09 21:01:11.605962 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:11.605554    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5584f3a5022c5ae87a1c4c791335b3e85417506c5608f1e026c15e2ecdc63bf7"
Sep 09 21:01:11.607647 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:11.607625    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6t2glg"
Sep 09 21:01:11.607647 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:11.607636    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6t2glg" event={"ID":"50d6e3f2-1fe7-4697-87ef-cbb615ae4aae","Type":"ContainerDied","Data":"acbd7e1eb424ec2e497fe95278a742ad60c8bc89924f0d02f5b803f72187c1d8"}
Sep 09 21:01:11.607848 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:11.607674    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="acbd7e1eb424ec2e497fe95278a742ad60c8bc89924f0d02f5b803f72187c1d8"
Sep 09 21:01:11.609903 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:11.609883    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5mnw5" event={"ID":"771c548e-b359-4ec8-af21-b68f97a6a1fa","Type":"ContainerDied","Data":"07df11626da71cc1300e7f416af703d7461e6aaf034079ee65b84dec22d53bc3"}
Sep 09 21:01:11.610008 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:11.609916    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="07df11626da71cc1300e7f416af703d7461e6aaf034079ee65b84dec22d53bc3"
Sep 09 21:01:11.610008 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:11.609926    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5mnw5"
Sep 09 21:01:14.724767 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:14.724722    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:01:17.298543 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:17.298504    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:01:17.392633 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:17.392596    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:01:18.953983 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:18.953945    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:01:19.957578 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:19.957539    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:01:20.060692 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:20.060649    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:01:21.965212 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:21.965178    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:01:23.135720 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:23.135691    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:01:23.476298 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:23.476199    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:01:24.469413 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:24.469354    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:01:28.980784 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:28.980697    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:01:29.075123 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:29.075084    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:01:34.284142 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:34.284109    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-xkpgz"]
Sep 09 21:01:34.284515 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:34.284444    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="771c548e-b359-4ec8-af21-b68f97a6a1fa" containerName="util"
Sep 09 21:01:34.284515 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:34.284456    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="771c548e-b359-4ec8-af21-b68f97a6a1fa" containerName="util"
Sep 09 21:01:34.284515 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:34.284466    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa0a535c-3226-43e7-96df-0d9372908774" containerName="util"
Sep 09 21:01:34.284515 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:34.284472    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa0a535c-3226-43e7-96df-0d9372908774" containerName="util"
Sep 09 21:01:34.284515 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:34.284477    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07652310-10fd-4d42-a8ff-e70294b9037d" containerName="util"
Sep 09 21:01:34.284515 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:34.284482    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="07652310-10fd-4d42-a8ff-e70294b9037d" containerName="util"
Sep 09 21:01:34.284515 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:34.284489    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="771c548e-b359-4ec8-af21-b68f97a6a1fa" containerName="pull"
Sep 09 21:01:34.284515 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:34.284494    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="771c548e-b359-4ec8-af21-b68f97a6a1fa" containerName="pull"
Sep 09 21:01:34.284515 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:34.284501    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07652310-10fd-4d42-a8ff-e70294b9037d" containerName="pull"
Sep 09 21:01:34.284515 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:34.284506    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="07652310-10fd-4d42-a8ff-e70294b9037d" containerName="pull"
Sep 09 21:01:34.284515 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:34.284511    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50d6e3f2-1fe7-4697-87ef-cbb615ae4aae" containerName="extract"
Sep 09 21:01:34.284515 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:34.284516    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="50d6e3f2-1fe7-4697-87ef-cbb615ae4aae" containerName="extract"
Sep 09 21:01:34.284877 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:34.284524    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50d6e3f2-1fe7-4697-87ef-cbb615ae4aae" containerName="util"
Sep 09 21:01:34.284877 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:34.284529    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="50d6e3f2-1fe7-4697-87ef-cbb615ae4aae" containerName="util"
Sep 09 21:01:34.284877 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:34.284538    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50d6e3f2-1fe7-4697-87ef-cbb615ae4aae" containerName="pull"
Sep 09 21:01:34.284877 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:34.284543    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="50d6e3f2-1fe7-4697-87ef-cbb615ae4aae" containerName="pull"
Sep 09 21:01:34.284877 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:34.284552    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07652310-10fd-4d42-a8ff-e70294b9037d" containerName="extract"
Sep 09 21:01:34.284877 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:34.284556    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="07652310-10fd-4d42-a8ff-e70294b9037d" containerName="extract"
Sep 09 21:01:34.284877 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:34.284562    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="771c548e-b359-4ec8-af21-b68f97a6a1fa" containerName="extract"
Sep 09 21:01:34.284877 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:34.284567    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="771c548e-b359-4ec8-af21-b68f97a6a1fa" containerName="extract"
Sep 09 21:01:34.284877 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:34.284573    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa0a535c-3226-43e7-96df-0d9372908774" containerName="extract"
Sep 09 21:01:34.284877 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:34.284578    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa0a535c-3226-43e7-96df-0d9372908774" containerName="extract"
Sep 09 21:01:34.284877 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:34.284586    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa0a535c-3226-43e7-96df-0d9372908774" containerName="pull"
Sep 09 21:01:34.284877 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:34.284590    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa0a535c-3226-43e7-96df-0d9372908774" containerName="pull"
Sep 09 21:01:34.284877 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:34.284651    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="07652310-10fd-4d42-a8ff-e70294b9037d" containerName="extract"
Sep 09 21:01:34.284877 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:34.284661    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="771c548e-b359-4ec8-af21-b68f97a6a1fa" containerName="extract"
Sep 09 21:01:34.284877 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:34.284668    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa0a535c-3226-43e7-96df-0d9372908774" containerName="extract"
Sep 09 21:01:34.284877 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:34.284675    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="50d6e3f2-1fe7-4697-87ef-cbb615ae4aae" containerName="extract"
Sep 09 21:01:34.287556 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:34.287540    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-xkpgz"
Sep 09 21:01:34.290546 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:34.290515    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 09 21:01:34.291433 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:34.291412    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Sep 09 21:01:34.291433 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:34.291427    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 09 21:01:34.291576 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:34.291425    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Sep 09 21:01:34.291576 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:34.291430    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-bm2b9\""
Sep 09 21:01:34.297103 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:34.297078    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-xkpgz"]
Sep 09 21:01:34.313301 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:34.313274    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:01:34.339329 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:34.339282    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m28x2\" (UniqueName: \"kubernetes.io/projected/91b92fac-cbad-43d1-87f5-8d93369b092c-kube-api-access-m28x2\") pod \"kuadrant-console-plugin-fbdd4f445-xkpgz\" (UID: \"91b92fac-cbad-43d1-87f5-8d93369b092c\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-xkpgz"
Sep 09 21:01:34.339473 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:34.339357    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/91b92fac-cbad-43d1-87f5-8d93369b092c-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-xkpgz\" (UID: \"91b92fac-cbad-43d1-87f5-8d93369b092c\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-xkpgz"
Sep 09 21:01:34.339473 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:34.339427    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/91b92fac-cbad-43d1-87f5-8d93369b092c-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-xkpgz\" (UID: \"91b92fac-cbad-43d1-87f5-8d93369b092c\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-xkpgz"
Sep 09 21:01:34.432142 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:34.432107    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:01:34.440124 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:34.440097    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m28x2\" (UniqueName: \"kubernetes.io/projected/91b92fac-cbad-43d1-87f5-8d93369b092c-kube-api-access-m28x2\") pod \"kuadrant-console-plugin-fbdd4f445-xkpgz\" (UID: \"91b92fac-cbad-43d1-87f5-8d93369b092c\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-xkpgz"
Sep 09 21:01:34.440224 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:34.440141    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/91b92fac-cbad-43d1-87f5-8d93369b092c-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-xkpgz\" (UID: \"91b92fac-cbad-43d1-87f5-8d93369b092c\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-xkpgz"
Sep 09 21:01:34.440224 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:34.440165    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/91b92fac-cbad-43d1-87f5-8d93369b092c-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-xkpgz\" (UID: \"91b92fac-cbad-43d1-87f5-8d93369b092c\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-xkpgz"
Sep 09 21:01:34.440779 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:34.440756    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/91b92fac-cbad-43d1-87f5-8d93369b092c-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-xkpgz\" (UID: \"91b92fac-cbad-43d1-87f5-8d93369b092c\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-xkpgz"
Sep 09 21:01:34.442735 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:34.442706    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/91b92fac-cbad-43d1-87f5-8d93369b092c-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-xkpgz\" (UID: \"91b92fac-cbad-43d1-87f5-8d93369b092c\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-xkpgz"
Sep 09 21:01:34.448433 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:34.448408    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m28x2\" (UniqueName: \"kubernetes.io/projected/91b92fac-cbad-43d1-87f5-8d93369b092c-kube-api-access-m28x2\") pod \"kuadrant-console-plugin-fbdd4f445-xkpgz\" (UID: \"91b92fac-cbad-43d1-87f5-8d93369b092c\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-xkpgz"
Sep 09 21:01:34.558878 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:34.558800    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:01:34.597698 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:34.597675    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-xkpgz"
Sep 09 21:01:34.662257 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:34.662218    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:01:34.870957 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:34.870934    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-xkpgz"]
Sep 09 21:01:34.872747 ip-10-0-134-72 kubenswrapper[2576]: W0909 21:01:34.872706    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod91b92fac_cbad_43d1_87f5_8d93369b092c.slice/crio-d2dfe09601a0610a1a35314b8e47bd86146a1fcddca6501ee28bf0b4169713f5 WatchSource:0}: Error finding container d2dfe09601a0610a1a35314b8e47bd86146a1fcddca6501ee28bf0b4169713f5: Status 404 returned error can't find the container with id d2dfe09601a0610a1a35314b8e47bd86146a1fcddca6501ee28bf0b4169713f5
Sep 09 21:01:34.992191 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:34.992160    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:01:35.078360 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:35.078326    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:01:35.722292 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:35.722257    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-xkpgz" event={"ID":"91b92fac-cbad-43d1-87f5-8d93369b092c","Type":"ContainerStarted","Data":"d2dfe09601a0610a1a35314b8e47bd86146a1fcddca6501ee28bf0b4169713f5"}
Sep 09 21:01:36.594320 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:36.594285    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:01:36.702803 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:36.702755    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:01:40.751377 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:40.751308    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-xkpgz" event={"ID":"91b92fac-cbad-43d1-87f5-8d93369b092c","Type":"ContainerStarted","Data":"32dfe3be86cb6a65b7fe0297af922790cf63d19f69c742ab9c3dafff54fe9126"}
Sep 09 21:01:40.772523 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:40.772474    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-xkpgz" podStartSLOduration=1.734007318 podStartE2EDuration="6.772458343s" totalImagesPullingTime="5.038451025s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:01:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:01:40.771980131 +0000 UTC m=+543.885666187" watchObservedRunningTime="2026-09-09 21:01:40.772458343 +0000 UTC m=+543.886144256"
Sep 09 21:01:41.460020 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:41.459986    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:01:41.561112 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:41.561082    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:01:41.661429 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:41.661394    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:01:43.721080 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:43.721014    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:01:43.828725 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:43.828680    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:01:48.312438 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:48.312405    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:01:48.411577 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:48.411546    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:01:53.490311 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:53.490271    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:01:53.558663 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:53.558627    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:01:53.928018 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:53.927985    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:01:54.010113 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:54.010044    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:01:57.292389 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:57.292342    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:01:57.390799 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:01:57.390766    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:02:01.071275 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:02:01.071243    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:02:01.169483 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:02:01.169442    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:02:01.269185 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:02:01.269154    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:02:12.306119 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:02:12.306086    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:02:12.369808 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:02:12.369774    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:02:19.195705 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:02:19.195665    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:02:19.299805 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:02:19.299768    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:02:19.398666 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:02:19.398636    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:02:19.999238 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:02:19.999204    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-gxbkn"]
Sep 09 21:02:20.017317 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:02:20.017284    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-gxbkn"]
Sep 09 21:02:20.017483 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:02:20.017294    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-gxbkn"
Sep 09 21:02:20.019952 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:02:20.019926    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-82rc5\""
Sep 09 21:02:20.034805 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:02:20.034775    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:02:20.099944 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:02:20.099913    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9246w\" (UniqueName: \"kubernetes.io/projected/8f77d835-0464-4eac-8069-6dbbffcec937-kube-api-access-9246w\") pod \"authorino-7bcfd8f9cd-gxbkn\" (UID: \"8f77d835-0464-4eac-8069-6dbbffcec937\") " pod="kuadrant-system/authorino-7bcfd8f9cd-gxbkn"
Sep 09 21:02:20.122401 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:02:20.122347    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:02:20.126260 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:02:20.126238    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-24vcs"]
Sep 09 21:02:20.129149 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:02:20.129135    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-24vcs"
Sep 09 21:02:20.138273 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:02:20.138252    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-24vcs"]
Sep 09 21:02:20.201040 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:02:20.201014    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgcbg\" (UniqueName: \"kubernetes.io/projected/757019d5-6281-4e21-a8b1-a2fc64046091-kube-api-access-wgcbg\") pod \"authorino-59c974bd48-24vcs\" (UID: \"757019d5-6281-4e21-a8b1-a2fc64046091\") " pod="kuadrant-system/authorino-59c974bd48-24vcs"
Sep 09 21:02:20.201342 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:02:20.201071    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9246w\" (UniqueName: \"kubernetes.io/projected/8f77d835-0464-4eac-8069-6dbbffcec937-kube-api-access-9246w\") pod \"authorino-7bcfd8f9cd-gxbkn\" (UID: \"8f77d835-0464-4eac-8069-6dbbffcec937\") " pod="kuadrant-system/authorino-7bcfd8f9cd-gxbkn"
Sep 09 21:02:20.209696 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:02:20.209670    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9246w\" (UniqueName: \"kubernetes.io/projected/8f77d835-0464-4eac-8069-6dbbffcec937-kube-api-access-9246w\") pod \"authorino-7bcfd8f9cd-gxbkn\" (UID: \"8f77d835-0464-4eac-8069-6dbbffcec937\") " pod="kuadrant-system/authorino-7bcfd8f9cd-gxbkn"
Sep 09 21:02:20.301866 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:02:20.301789    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgcbg\" (UniqueName: \"kubernetes.io/projected/757019d5-6281-4e21-a8b1-a2fc64046091-kube-api-access-wgcbg\") pod \"authorino-59c974bd48-24vcs\" (UID: \"757019d5-6281-4e21-a8b1-a2fc64046091\") " pod="kuadrant-system/authorino-59c974bd48-24vcs"
Sep 09 21:02:20.310664 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:02:20.310643    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgcbg\" (UniqueName: \"kubernetes.io/projected/757019d5-6281-4e21-a8b1-a2fc64046091-kube-api-access-wgcbg\") pod \"authorino-59c974bd48-24vcs\" (UID: \"757019d5-6281-4e21-a8b1-a2fc64046091\") " pod="kuadrant-system/authorino-59c974bd48-24vcs"
Sep 09 21:02:20.325819 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:02:20.325798    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-gxbkn"
Sep 09 21:02:20.438152 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:02:20.438122    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-24vcs"
Sep 09 21:02:20.597162 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:02:20.597131    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-gxbkn"]
Sep 09 21:02:20.600342 ip-10-0-134-72 kubenswrapper[2576]: W0909 21:02:20.600305    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8f77d835_0464_4eac_8069_6dbbffcec937.slice/crio-745fda2d48d7f6b936b25596eb3e054c34d0a8b26d34d52d84b4e50687bf567a WatchSource:0}: Error finding container 745fda2d48d7f6b936b25596eb3e054c34d0a8b26d34d52d84b4e50687bf567a: Status 404 returned error can't find the container with id 745fda2d48d7f6b936b25596eb3e054c34d0a8b26d34d52d84b4e50687bf567a
Sep 09 21:02:20.706521 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:02:20.706498    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-24vcs"]
Sep 09 21:02:20.708204 ip-10-0-134-72 kubenswrapper[2576]: W0909 21:02:20.708176    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod757019d5_6281_4e21_a8b1_a2fc64046091.slice/crio-63893db3236e8553147f21364afdf24c36e98d07d0f3d31f7ad59a65442f1eb0 WatchSource:0}: Error finding container 63893db3236e8553147f21364afdf24c36e98d07d0f3d31f7ad59a65442f1eb0: Status 404 returned error can't find the container with id 63893db3236e8553147f21364afdf24c36e98d07d0f3d31f7ad59a65442f1eb0
Sep 09 21:02:20.948854 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:02:20.948766    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-gxbkn" event={"ID":"8f77d835-0464-4eac-8069-6dbbffcec937","Type":"ContainerStarted","Data":"745fda2d48d7f6b936b25596eb3e054c34d0a8b26d34d52d84b4e50687bf567a"}
Sep 09 21:02:20.949931 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:02:20.949906    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-24vcs" event={"ID":"757019d5-6281-4e21-a8b1-a2fc64046091","Type":"ContainerStarted","Data":"63893db3236e8553147f21364afdf24c36e98d07d0f3d31f7ad59a65442f1eb0"}
Sep 09 21:02:22.962276 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:02:22.962240    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-gxbkn" event={"ID":"8f77d835-0464-4eac-8069-6dbbffcec937","Type":"ContainerStarted","Data":"ef7e467268ea929495742ff9e6b951263fc1d6e2df78c5b7daa76437d9363948"}
Sep 09 21:02:22.978132 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:02:22.978091    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-gxbkn" podStartSLOduration=2.226566194 podStartE2EDuration="3.978076196s" totalImagesPullingTime="1.751510002s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:02:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:02:22.977248928 +0000 UTC m=+586.090934798" watchObservedRunningTime="2026-09-09 21:02:22.978076196 +0000 UTC m=+586.091762054"
Sep 09 21:02:24.973206 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:02:24.973167    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-24vcs" event={"ID":"757019d5-6281-4e21-a8b1-a2fc64046091","Type":"ContainerStarted","Data":"2bfa596c6eb40dc0312f038f10bbbcdca7b32a40eabd39cea6ddafe4fa9c3c81"}
Sep 09 21:02:24.990181 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:02:24.989789    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-24vcs" podStartSLOduration=1.459192893 podStartE2EDuration="4.98977183s" totalImagesPullingTime="3.530578937s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:02:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:02:24.989400011 +0000 UTC m=+588.103085875" watchObservedRunningTime="2026-09-09 21:02:24.98977183 +0000 UTC m=+588.103457692"
Sep 09 21:02:25.018663 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:02:25.018633    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-gxbkn"]
Sep 09 21:02:25.018809 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:02:25.018784    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-gxbkn" podUID="8f77d835-0464-4eac-8069-6dbbffcec937" containerName="authorino" containerID="cri-o://ef7e467268ea929495742ff9e6b951263fc1d6e2df78c5b7daa76437d9363948" gracePeriod=30
Sep 09 21:02:25.262443 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:02:25.262414    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-gxbkn"
Sep 09 21:02:25.344111 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:02:25.344085    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9246w\" (UniqueName: \"kubernetes.io/projected/8f77d835-0464-4eac-8069-6dbbffcec937-kube-api-access-9246w\") pod \"8f77d835-0464-4eac-8069-6dbbffcec937\" (UID: \"8f77d835-0464-4eac-8069-6dbbffcec937\") "
Sep 09 21:02:25.346330 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:02:25.346302    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f77d835-0464-4eac-8069-6dbbffcec937-kube-api-access-9246w" (OuterVolumeSpecName: "kube-api-access-9246w") pod "8f77d835-0464-4eac-8069-6dbbffcec937" (UID: "8f77d835-0464-4eac-8069-6dbbffcec937"). InnerVolumeSpecName "kube-api-access-9246w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:02:25.445624 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:02:25.445597    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9246w\" (UniqueName: \"kubernetes.io/projected/8f77d835-0464-4eac-8069-6dbbffcec937-kube-api-access-9246w\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:02:25.979417 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:02:25.979384    2576 generic.go:356] "Generic (PLEG): container finished" podID="8f77d835-0464-4eac-8069-6dbbffcec937" containerID="ef7e467268ea929495742ff9e6b951263fc1d6e2df78c5b7daa76437d9363948" exitCode=0
Sep 09 21:02:25.979806 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:02:25.979420    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-gxbkn"
Sep 09 21:02:25.979806 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:02:25.979467    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-gxbkn" event={"ID":"8f77d835-0464-4eac-8069-6dbbffcec937","Type":"ContainerDied","Data":"ef7e467268ea929495742ff9e6b951263fc1d6e2df78c5b7daa76437d9363948"}
Sep 09 21:02:25.979806 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:02:25.979517    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-gxbkn" event={"ID":"8f77d835-0464-4eac-8069-6dbbffcec937","Type":"ContainerDied","Data":"745fda2d48d7f6b936b25596eb3e054c34d0a8b26d34d52d84b4e50687bf567a"}
Sep 09 21:02:25.979806 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:02:25.979541    2576 scope.go:117] "RemoveContainer" containerID="ef7e467268ea929495742ff9e6b951263fc1d6e2df78c5b7daa76437d9363948"
Sep 09 21:02:25.988465 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:02:25.988435    2576 scope.go:117] "RemoveContainer" containerID="ef7e467268ea929495742ff9e6b951263fc1d6e2df78c5b7daa76437d9363948"
Sep 09 21:02:25.988738 ip-10-0-134-72 kubenswrapper[2576]: E0909 21:02:25.988719    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef7e467268ea929495742ff9e6b951263fc1d6e2df78c5b7daa76437d9363948\": container with ID starting with ef7e467268ea929495742ff9e6b951263fc1d6e2df78c5b7daa76437d9363948 not found: ID does not exist" containerID="ef7e467268ea929495742ff9e6b951263fc1d6e2df78c5b7daa76437d9363948"
Sep 09 21:02:25.988790 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:02:25.988746    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef7e467268ea929495742ff9e6b951263fc1d6e2df78c5b7daa76437d9363948"} err="failed to get container status \"ef7e467268ea929495742ff9e6b951263fc1d6e2df78c5b7daa76437d9363948\": rpc error: code = NotFound desc = could not find container \"ef7e467268ea929495742ff9e6b951263fc1d6e2df78c5b7daa76437d9363948\": container with ID starting with ef7e467268ea929495742ff9e6b951263fc1d6e2df78c5b7daa76437d9363948 not found: ID does not exist"
Sep 09 21:02:25.996866 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:02:25.996845    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-gxbkn"]
Sep 09 21:02:26.002334 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:02:26.002315    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-gxbkn"]
Sep 09 21:02:27.409696 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:02:27.409663    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f77d835-0464-4eac-8069-6dbbffcec937" path="/var/lib/kubelet/pods/8f77d835-0464-4eac-8069-6dbbffcec937/volumes"
Sep 09 21:02:27.764607 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:02:27.764518    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:02:32.932681 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:02:32.932646    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:02:38.439899 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:02:38.439849    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:02:38.611812 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:02:38.611778    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:02:41.672657 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:02:41.672624    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d9898bd99-s8t9x"]
Sep 09 21:02:41.673100 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:02:41.673087    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f77d835-0464-4eac-8069-6dbbffcec937" containerName="authorino"
Sep 09 21:02:41.673168 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:02:41.673107    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f77d835-0464-4eac-8069-6dbbffcec937" containerName="authorino"
Sep 09 21:02:41.673223 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:02:41.673212    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f77d835-0464-4eac-8069-6dbbffcec937" containerName="authorino"
Sep 09 21:02:41.677168 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:02:41.677147    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d9898bd99-s8t9x"
Sep 09 21:02:41.679703 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:02:41.679685    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 09 21:02:41.685099 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:02:41.685078    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d9898bd99-s8t9x"]
Sep 09 21:02:41.789599 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:02:41.789570    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f4780d21-1e4d-447c-82d5-8bd54a4c7ab6-tls-cert\") pod \"authorino-6d9898bd99-s8t9x\" (UID: \"f4780d21-1e4d-447c-82d5-8bd54a4c7ab6\") " pod="kuadrant-system/authorino-6d9898bd99-s8t9x"
Sep 09 21:02:41.789599 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:02:41.789600    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thwrt\" (UniqueName: \"kubernetes.io/projected/f4780d21-1e4d-447c-82d5-8bd54a4c7ab6-kube-api-access-thwrt\") pod \"authorino-6d9898bd99-s8t9x\" (UID: \"f4780d21-1e4d-447c-82d5-8bd54a4c7ab6\") " pod="kuadrant-system/authorino-6d9898bd99-s8t9x"
Sep 09 21:02:41.890699 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:02:41.890668    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f4780d21-1e4d-447c-82d5-8bd54a4c7ab6-tls-cert\") pod \"authorino-6d9898bd99-s8t9x\" (UID: \"f4780d21-1e4d-447c-82d5-8bd54a4c7ab6\") " pod="kuadrant-system/authorino-6d9898bd99-s8t9x"
Sep 09 21:02:41.890699 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:02:41.890699    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thwrt\" (UniqueName: \"kubernetes.io/projected/f4780d21-1e4d-447c-82d5-8bd54a4c7ab6-kube-api-access-thwrt\") pod \"authorino-6d9898bd99-s8t9x\" (UID: \"f4780d21-1e4d-447c-82d5-8bd54a4c7ab6\") " pod="kuadrant-system/authorino-6d9898bd99-s8t9x"
Sep 09 21:02:41.893232 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:02:41.893214    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f4780d21-1e4d-447c-82d5-8bd54a4c7ab6-tls-cert\") pod \"authorino-6d9898bd99-s8t9x\" (UID: \"f4780d21-1e4d-447c-82d5-8bd54a4c7ab6\") " pod="kuadrant-system/authorino-6d9898bd99-s8t9x"
Sep 09 21:02:41.898640 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:02:41.898623    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-thwrt\" (UniqueName: \"kubernetes.io/projected/f4780d21-1e4d-447c-82d5-8bd54a4c7ab6-kube-api-access-thwrt\") pod \"authorino-6d9898bd99-s8t9x\" (UID: \"f4780d21-1e4d-447c-82d5-8bd54a4c7ab6\") " pod="kuadrant-system/authorino-6d9898bd99-s8t9x"
Sep 09 21:02:41.987025 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:02:41.986963    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d9898bd99-s8t9x"
Sep 09 21:02:42.257553 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:02:42.257530    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d9898bd99-s8t9x"]
Sep 09 21:02:42.259466 ip-10-0-134-72 kubenswrapper[2576]: W0909 21:02:42.259425    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf4780d21_1e4d_447c_82d5_8bd54a4c7ab6.slice/crio-d33363266d59a6e60d55f6b2f56e1e0651a5389c47b47f47c48a763e02dc8a20 WatchSource:0}: Error finding container d33363266d59a6e60d55f6b2f56e1e0651a5389c47b47f47c48a763e02dc8a20: Status 404 returned error can't find the container with id d33363266d59a6e60d55f6b2f56e1e0651a5389c47b47f47c48a763e02dc8a20
Sep 09 21:02:43.067580 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:02:43.067550    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6d9898bd99-s8t9x" event={"ID":"f4780d21-1e4d-447c-82d5-8bd54a4c7ab6","Type":"ContainerStarted","Data":"d33363266d59a6e60d55f6b2f56e1e0651a5389c47b47f47c48a763e02dc8a20"}
Sep 09 21:02:44.073786 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:02:44.073746    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6d9898bd99-s8t9x" event={"ID":"f4780d21-1e4d-447c-82d5-8bd54a4c7ab6","Type":"ContainerStarted","Data":"24b3db76c286e6924b3b5c3c39a79a2c6320af6c760a65fef34f11ab26be8ed8"}
Sep 09 21:02:44.092149 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:02:44.092099    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-6d9898bd99-s8t9x" podStartSLOduration=2.291826672 podStartE2EDuration="3.092084473s" totalImagesPullingTime="800.257801ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:02:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:02:44.090265697 +0000 UTC m=+607.203951557" watchObservedRunningTime="2026-09-09 21:02:44.092084473 +0000 UTC m=+607.205770332"
Sep 09 21:02:44.122457 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:02:44.122429    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-24vcs"]
Sep 09 21:02:44.122622 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:02:44.122602    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-24vcs" podUID="757019d5-6281-4e21-a8b1-a2fc64046091" containerName="authorino" containerID="cri-o://2bfa596c6eb40dc0312f038f10bbbcdca7b32a40eabd39cea6ddafe4fa9c3c81" gracePeriod=30
Sep 09 21:02:44.364182 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:02:44.364154    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-24vcs"
Sep 09 21:02:44.511615 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:02:44.511582    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wgcbg\" (UniqueName: \"kubernetes.io/projected/757019d5-6281-4e21-a8b1-a2fc64046091-kube-api-access-wgcbg\") pod \"757019d5-6281-4e21-a8b1-a2fc64046091\" (UID: \"757019d5-6281-4e21-a8b1-a2fc64046091\") "
Sep 09 21:02:44.513739 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:02:44.513710    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/757019d5-6281-4e21-a8b1-a2fc64046091-kube-api-access-wgcbg" (OuterVolumeSpecName: "kube-api-access-wgcbg") pod "757019d5-6281-4e21-a8b1-a2fc64046091" (UID: "757019d5-6281-4e21-a8b1-a2fc64046091"). InnerVolumeSpecName "kube-api-access-wgcbg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:02:44.612678 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:02:44.612608    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wgcbg\" (UniqueName: \"kubernetes.io/projected/757019d5-6281-4e21-a8b1-a2fc64046091-kube-api-access-wgcbg\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:02:45.080270 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:02:45.080180    2576 generic.go:356] "Generic (PLEG): container finished" podID="757019d5-6281-4e21-a8b1-a2fc64046091" containerID="2bfa596c6eb40dc0312f038f10bbbcdca7b32a40eabd39cea6ddafe4fa9c3c81" exitCode=0
Sep 09 21:02:45.080270 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:02:45.080224    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-24vcs"
Sep 09 21:02:45.080737 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:02:45.080261    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-24vcs" event={"ID":"757019d5-6281-4e21-a8b1-a2fc64046091","Type":"ContainerDied","Data":"2bfa596c6eb40dc0312f038f10bbbcdca7b32a40eabd39cea6ddafe4fa9c3c81"}
Sep 09 21:02:45.080737 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:02:45.080308    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-24vcs" event={"ID":"757019d5-6281-4e21-a8b1-a2fc64046091","Type":"ContainerDied","Data":"63893db3236e8553147f21364afdf24c36e98d07d0f3d31f7ad59a65442f1eb0"}
Sep 09 21:02:45.080737 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:02:45.080326    2576 scope.go:117] "RemoveContainer" containerID="2bfa596c6eb40dc0312f038f10bbbcdca7b32a40eabd39cea6ddafe4fa9c3c81"
Sep 09 21:02:45.089218 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:02:45.089199    2576 scope.go:117] "RemoveContainer" containerID="2bfa596c6eb40dc0312f038f10bbbcdca7b32a40eabd39cea6ddafe4fa9c3c81"
Sep 09 21:02:45.089510 ip-10-0-134-72 kubenswrapper[2576]: E0909 21:02:45.089491    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bfa596c6eb40dc0312f038f10bbbcdca7b32a40eabd39cea6ddafe4fa9c3c81\": container with ID starting with 2bfa596c6eb40dc0312f038f10bbbcdca7b32a40eabd39cea6ddafe4fa9c3c81 not found: ID does not exist" containerID="2bfa596c6eb40dc0312f038f10bbbcdca7b32a40eabd39cea6ddafe4fa9c3c81"
Sep 09 21:02:45.089562 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:02:45.089518    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bfa596c6eb40dc0312f038f10bbbcdca7b32a40eabd39cea6ddafe4fa9c3c81"} err="failed to get container status \"2bfa596c6eb40dc0312f038f10bbbcdca7b32a40eabd39cea6ddafe4fa9c3c81\": rpc error: code = NotFound desc = could not find container \"2bfa596c6eb40dc0312f038f10bbbcdca7b32a40eabd39cea6ddafe4fa9c3c81\": container with ID starting with 2bfa596c6eb40dc0312f038f10bbbcdca7b32a40eabd39cea6ddafe4fa9c3c81 not found: ID does not exist"
Sep 09 21:02:45.101330 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:02:45.101306    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-24vcs"]
Sep 09 21:02:45.105171 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:02:45.105154    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-24vcs"]
Sep 09 21:02:45.408882 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:02:45.408848    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="757019d5-6281-4e21-a8b1-a2fc64046091" path="/var/lib/kubelet/pods/757019d5-6281-4e21-a8b1-a2fc64046091/volumes"
Sep 09 21:02:47.337791 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:02:47.337758    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:02:47.645996 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:02:47.645909    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:03:17.377875 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:03:17.377835    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:03:17.461010 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:03:17.460975    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:03:17.559459 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:03:17.559423    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:03:17.662145 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:03:17.662060    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:03:18.003521 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:03:18.003425    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:03:18.108286 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:03:18.108248    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:03:18.237158 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:03:18.237123    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:03:52.948136 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:03:52.948098    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:04:37.574752 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:04:37.574561    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:04:39.308533 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:04:39.308497    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:04:54.327733 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:04:54.327700    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-dlz7m"]
Sep 09 21:04:54.328166 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:04:54.328043    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="757019d5-6281-4e21-a8b1-a2fc64046091" containerName="authorino"
Sep 09 21:04:54.328166 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:04:54.328055    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="757019d5-6281-4e21-a8b1-a2fc64046091" containerName="authorino"
Sep 09 21:04:54.328166 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:04:54.328122    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="757019d5-6281-4e21-a8b1-a2fc64046091" containerName="authorino"
Sep 09 21:04:54.331124 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:04:54.331107    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-dlz7m"
Sep 09 21:04:54.334194 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:04:54.334164    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Sep 09 21:04:54.334194 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:04:54.334186    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-hcvd6\""
Sep 09 21:04:54.334359 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:04:54.334203    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 09 21:04:54.335197 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:04:54.335161    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 09 21:04:54.335197 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:04:54.335178    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 09 21:04:54.342685 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:04:54.342648    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-dlz7m"]
Sep 09 21:04:54.399470 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:04:54.399432    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b56feb4a-6f9a-40c7-be0d-391097259ccf-model-cache\") pod \"s3-init-dlz7m\" (UID: \"b56feb4a-6f9a-40c7-be0d-391097259ccf\") " pod="kserve/s3-init-dlz7m"
Sep 09 21:04:54.399633 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:04:54.399504    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhht5\" (UniqueName: \"kubernetes.io/projected/b56feb4a-6f9a-40c7-be0d-391097259ccf-kube-api-access-qhht5\") pod \"s3-init-dlz7m\" (UID: \"b56feb4a-6f9a-40c7-be0d-391097259ccf\") " pod="kserve/s3-init-dlz7m"
Sep 09 21:04:54.499874 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:04:54.499837    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhht5\" (UniqueName: \"kubernetes.io/projected/b56feb4a-6f9a-40c7-be0d-391097259ccf-kube-api-access-qhht5\") pod \"s3-init-dlz7m\" (UID: \"b56feb4a-6f9a-40c7-be0d-391097259ccf\") " pod="kserve/s3-init-dlz7m"
Sep 09 21:04:54.500067 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:04:54.499910    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b56feb4a-6f9a-40c7-be0d-391097259ccf-model-cache\") pod \"s3-init-dlz7m\" (UID: \"b56feb4a-6f9a-40c7-be0d-391097259ccf\") " pod="kserve/s3-init-dlz7m"
Sep 09 21:04:54.500287 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:04:54.500263    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b56feb4a-6f9a-40c7-be0d-391097259ccf-model-cache\") pod \"s3-init-dlz7m\" (UID: \"b56feb4a-6f9a-40c7-be0d-391097259ccf\") " pod="kserve/s3-init-dlz7m"
Sep 09 21:04:54.509428 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:04:54.509406    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhht5\" (UniqueName: \"kubernetes.io/projected/b56feb4a-6f9a-40c7-be0d-391097259ccf-kube-api-access-qhht5\") pod \"s3-init-dlz7m\" (UID: \"b56feb4a-6f9a-40c7-be0d-391097259ccf\") " pod="kserve/s3-init-dlz7m"
Sep 09 21:04:54.639674 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:04:54.639646    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-dlz7m"
Sep 09 21:04:54.911217 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:04:54.910993    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-dlz7m"]
Sep 09 21:04:54.913453 ip-10-0-134-72 kubenswrapper[2576]: W0909 21:04:54.913424    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb56feb4a_6f9a_40c7_be0d_391097259ccf.slice/crio-2e9994f4cdcfceaff014ae5385f0a01074a2cbc3803dcf2bf28b5a91de3ff858 WatchSource:0}: Error finding container 2e9994f4cdcfceaff014ae5385f0a01074a2cbc3803dcf2bf28b5a91de3ff858: Status 404 returned error can't find the container with id 2e9994f4cdcfceaff014ae5385f0a01074a2cbc3803dcf2bf28b5a91de3ff858
Sep 09 21:04:54.915407 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:04:54.915358    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 21:04:55.727215 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:04:55.727182    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-dlz7m" event={"ID":"b56feb4a-6f9a-40c7-be0d-391097259ccf","Type":"ContainerStarted","Data":"2e9994f4cdcfceaff014ae5385f0a01074a2cbc3803dcf2bf28b5a91de3ff858"}
Sep 09 21:04:58.746845 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:04:58.746757    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-dlz7m" event={"ID":"b56feb4a-6f9a-40c7-be0d-391097259ccf","Type":"ContainerStarted","Data":"747fe3a9cd61b2bd17d99ddaa7ab5aa247a21c7eb3540f10c31724d2d7d700cd"}
Sep 09 21:05:00.759212 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:05:00.759122    2576 generic.go:356] "Generic (PLEG): container finished" podID="b56feb4a-6f9a-40c7-be0d-391097259ccf" containerID="747fe3a9cd61b2bd17d99ddaa7ab5aa247a21c7eb3540f10c31724d2d7d700cd" exitCode=0
Sep 09 21:05:00.759212 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:05:00.759195    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-dlz7m" event={"ID":"b56feb4a-6f9a-40c7-be0d-391097259ccf","Type":"ContainerDied","Data":"747fe3a9cd61b2bd17d99ddaa7ab5aa247a21c7eb3540f10c31724d2d7d700cd"}
Sep 09 21:05:05.789080 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:05:05.789047    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-dlz7m" event={"ID":"b56feb4a-6f9a-40c7-be0d-391097259ccf","Type":"ContainerStarted","Data":"ee7acc79a5cb0f0d52fe8459f563954e5663c86f57d82ed0a6dc1a158767a1e9"}
Sep 09 21:05:05.809769 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:05:05.809723    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-dlz7m" podStartSLOduration=1.823716554 podStartE2EDuration="11.809708437s" totalImagesPullingTime="8.059979397s" totalInitContainerRuntime="1.926012486s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:04:54 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:05:05.807748248 +0000 UTC m=+748.921434106" watchObservedRunningTime="2026-09-09 21:05:05.809708437 +0000 UTC m=+748.923394296"
Sep 09 21:05:12.281381 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:05:12.281328    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:05:29.890920 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:05:29.890885    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:05:29.963894 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:05:29.963855    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:06:11.124900 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:06:11.124863    2576 generic.go:356] "Generic (PLEG): container finished" podID="b56feb4a-6f9a-40c7-be0d-391097259ccf" containerID="ee7acc79a5cb0f0d52fe8459f563954e5663c86f57d82ed0a6dc1a158767a1e9" exitCode=0
Sep 09 21:06:11.125297 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:06:11.124933    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-dlz7m" event={"ID":"b56feb4a-6f9a-40c7-be0d-391097259ccf","Type":"ContainerDied","Data":"ee7acc79a5cb0f0d52fe8459f563954e5663c86f57d82ed0a6dc1a158767a1e9"}
Sep 09 21:06:12.255463 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:06:12.255441    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-dlz7m"
Sep 09 21:06:12.364901 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:06:12.364870    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b56feb4a-6f9a-40c7-be0d-391097259ccf-model-cache\") pod \"b56feb4a-6f9a-40c7-be0d-391097259ccf\" (UID: \"b56feb4a-6f9a-40c7-be0d-391097259ccf\") "
Sep 09 21:06:12.365075 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:06:12.364954    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qhht5\" (UniqueName: \"kubernetes.io/projected/b56feb4a-6f9a-40c7-be0d-391097259ccf-kube-api-access-qhht5\") pod \"b56feb4a-6f9a-40c7-be0d-391097259ccf\" (UID: \"b56feb4a-6f9a-40c7-be0d-391097259ccf\") "
Sep 09 21:06:12.367189 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:06:12.367159    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b56feb4a-6f9a-40c7-be0d-391097259ccf-kube-api-access-qhht5" (OuterVolumeSpecName: "kube-api-access-qhht5") pod "b56feb4a-6f9a-40c7-be0d-391097259ccf" (UID: "b56feb4a-6f9a-40c7-be0d-391097259ccf"). InnerVolumeSpecName "kube-api-access-qhht5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:06:12.383702 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:06:12.383647    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b56feb4a-6f9a-40c7-be0d-391097259ccf-model-cache" (OuterVolumeSpecName: "model-cache") pod "b56feb4a-6f9a-40c7-be0d-391097259ccf" (UID: "b56feb4a-6f9a-40c7-be0d-391097259ccf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:06:12.465679 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:06:12.465650    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhht5\" (UniqueName: \"kubernetes.io/projected/b56feb4a-6f9a-40c7-be0d-391097259ccf-kube-api-access-qhht5\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:06:12.465679 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:06:12.465677    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b56feb4a-6f9a-40c7-be0d-391097259ccf-model-cache\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:06:13.135937 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:06:13.135910    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-dlz7m"
Sep 09 21:06:13.136127 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:06:13.135949    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-dlz7m" event={"ID":"b56feb4a-6f9a-40c7-be0d-391097259ccf","Type":"ContainerDied","Data":"2e9994f4cdcfceaff014ae5385f0a01074a2cbc3803dcf2bf28b5a91de3ff858"}
Sep 09 21:06:13.136127 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:06:13.135983    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2e9994f4cdcfceaff014ae5385f0a01074a2cbc3803dcf2bf28b5a91de3ff858"
Sep 09 21:06:13.515832 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:06:13.515745    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:06:13.611638 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:06:13.611610    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:06:26.725972 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:06:26.725937    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:06:26.817271 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:06:26.817227    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:06:36.224552 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:06:36.224521    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6dc886bc8bvrdcp"]
Sep 09 21:06:36.224898 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:06:36.224844    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b56feb4a-6f9a-40c7-be0d-391097259ccf" containerName="download-hf-model"
Sep 09 21:06:36.224898 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:06:36.224856    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="b56feb4a-6f9a-40c7-be0d-391097259ccf" containerName="download-hf-model"
Sep 09 21:06:36.224898 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:06:36.224867    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b56feb4a-6f9a-40c7-be0d-391097259ccf" containerName="s3-init"
Sep 09 21:06:36.224898 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:06:36.224873    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="b56feb4a-6f9a-40c7-be0d-391097259ccf" containerName="s3-init"
Sep 09 21:06:36.225032 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:06:36.224934    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="b56feb4a-6f9a-40c7-be0d-391097259ccf" containerName="s3-init"
Sep 09 21:06:36.227889 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:06:36.227873    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6dc886bc8bvrdcp"
Sep 09 21:06:36.232304 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:06:36.231083    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"seaweedfs-s3-creds\""
Sep 09 21:06:36.232304 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:06:36.231441    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"openshift-service-ca.crt\""
Sep 09 21:06:36.232304 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:06:36.231693    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"default-dockercfg-ptdkc\""
Sep 09 21:06:36.232304 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:06:36.231725    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"odh-kserve-custom-ca-bundle\""
Sep 09 21:06:36.232304 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:06:36.232105    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"kube-root-ca.crt\""
Sep 09 21:06:36.232304 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:06:36.232219    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"llmisvd10b93c8eba12f87c3a350a5cae2ee0c-kserve-self-signed-certs\""
Sep 09 21:06:36.242619 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:06:36.242599    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6dc886bc8bvrdcp"]
Sep 09 21:06:36.270015 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:06:36.269987    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de677c34-d375-42a3-9346-7a4bc1f82f44-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6dc886bc8bvrdcp\" (UID: \"de677c34-d375-42a3-9346-7a4bc1f82f44\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6dc886bc8bvrdcp"
Sep 09 21:06:36.270146 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:06:36.270031    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de677c34-d375-42a3-9346-7a4bc1f82f44-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6dc886bc8bvrdcp\" (UID: \"de677c34-d375-42a3-9346-7a4bc1f82f44\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6dc886bc8bvrdcp"
Sep 09 21:06:36.270146 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:06:36.270055    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de677c34-d375-42a3-9346-7a4bc1f82f44-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6dc886bc8bvrdcp\" (UID: \"de677c34-d375-42a3-9346-7a4bc1f82f44\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6dc886bc8bvrdcp"
Sep 09 21:06:36.270146 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:06:36.270074    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/de677c34-d375-42a3-9346-7a4bc1f82f44-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6dc886bc8bvrdcp\" (UID: \"de677c34-d375-42a3-9346-7a4bc1f82f44\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6dc886bc8bvrdcp"
Sep 09 21:06:36.270146 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:06:36.270089    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2h6l5\" (UniqueName: \"kubernetes.io/projected/de677c34-d375-42a3-9346-7a4bc1f82f44-kube-api-access-2h6l5\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6dc886bc8bvrdcp\" (UID: \"de677c34-d375-42a3-9346-7a4bc1f82f44\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6dc886bc8bvrdcp"
Sep 09 21:06:36.270146 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:06:36.270114    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/de677c34-d375-42a3-9346-7a4bc1f82f44-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6dc886bc8bvrdcp\" (UID: \"de677c34-d375-42a3-9346-7a4bc1f82f44\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6dc886bc8bvrdcp"
Sep 09 21:06:36.270146 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:06:36.270130    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de677c34-d375-42a3-9346-7a4bc1f82f44-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6dc886bc8bvrdcp\" (UID: \"de677c34-d375-42a3-9346-7a4bc1f82f44\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6dc886bc8bvrdcp"
Sep 09 21:06:36.270357 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:06:36.270222    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de677c34-d375-42a3-9346-7a4bc1f82f44-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6dc886bc8bvrdcp\" (UID: \"de677c34-d375-42a3-9346-7a4bc1f82f44\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6dc886bc8bvrdcp"
Sep 09 21:06:36.287938 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:06:36.287912    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:06:36.370704 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:06:36.370678    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de677c34-d375-42a3-9346-7a4bc1f82f44-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6dc886bc8bvrdcp\" (UID: \"de677c34-d375-42a3-9346-7a4bc1f82f44\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6dc886bc8bvrdcp"
Sep 09 21:06:36.370811 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:06:36.370708    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/de677c34-d375-42a3-9346-7a4bc1f82f44-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6dc886bc8bvrdcp\" (UID: \"de677c34-d375-42a3-9346-7a4bc1f82f44\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6dc886bc8bvrdcp"
Sep 09 21:06:36.370811 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:06:36.370726    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2h6l5\" (UniqueName: \"kubernetes.io/projected/de677c34-d375-42a3-9346-7a4bc1f82f44-kube-api-access-2h6l5\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6dc886bc8bvrdcp\" (UID: \"de677c34-d375-42a3-9346-7a4bc1f82f44\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6dc886bc8bvrdcp"
Sep 09 21:06:36.370811 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:06:36.370754    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/de677c34-d375-42a3-9346-7a4bc1f82f44-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6dc886bc8bvrdcp\" (UID: \"de677c34-d375-42a3-9346-7a4bc1f82f44\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6dc886bc8bvrdcp"
Sep 09 21:06:36.370811 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:06:36.370771    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de677c34-d375-42a3-9346-7a4bc1f82f44-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6dc886bc8bvrdcp\" (UID: \"de677c34-d375-42a3-9346-7a4bc1f82f44\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6dc886bc8bvrdcp"
Sep 09 21:06:36.371014 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:06:36.370815    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de677c34-d375-42a3-9346-7a4bc1f82f44-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6dc886bc8bvrdcp\" (UID: \"de677c34-d375-42a3-9346-7a4bc1f82f44\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6dc886bc8bvrdcp"
Sep 09 21:06:36.371014 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:06:36.370837    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de677c34-d375-42a3-9346-7a4bc1f82f44-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6dc886bc8bvrdcp\" (UID: \"de677c34-d375-42a3-9346-7a4bc1f82f44\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6dc886bc8bvrdcp"
Sep 09 21:06:36.371014 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:06:36.370880    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de677c34-d375-42a3-9346-7a4bc1f82f44-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6dc886bc8bvrdcp\" (UID: \"de677c34-d375-42a3-9346-7a4bc1f82f44\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6dc886bc8bvrdcp"
Sep 09 21:06:36.371180 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:06:36.371128    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/de677c34-d375-42a3-9346-7a4bc1f82f44-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6dc886bc8bvrdcp\" (UID: \"de677c34-d375-42a3-9346-7a4bc1f82f44\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6dc886bc8bvrdcp"
Sep 09 21:06:36.371301 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:06:36.371187    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de677c34-d375-42a3-9346-7a4bc1f82f44-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6dc886bc8bvrdcp\" (UID: \"de677c34-d375-42a3-9346-7a4bc1f82f44\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6dc886bc8bvrdcp"
Sep 09 21:06:36.371301 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:06:36.371211    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de677c34-d375-42a3-9346-7a4bc1f82f44-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6dc886bc8bvrdcp\" (UID: \"de677c34-d375-42a3-9346-7a4bc1f82f44\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6dc886bc8bvrdcp"
Sep 09 21:06:36.371301 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:06:36.371291    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de677c34-d375-42a3-9346-7a4bc1f82f44-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6dc886bc8bvrdcp\" (UID: \"de677c34-d375-42a3-9346-7a4bc1f82f44\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6dc886bc8bvrdcp"
Sep 09 21:06:36.371508 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:06:36.371491    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/de677c34-d375-42a3-9346-7a4bc1f82f44-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6dc886bc8bvrdcp\" (UID: \"de677c34-d375-42a3-9346-7a4bc1f82f44\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6dc886bc8bvrdcp"
Sep 09 21:06:36.373329 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:06:36.373314    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de677c34-d375-42a3-9346-7a4bc1f82f44-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6dc886bc8bvrdcp\" (UID: \"de677c34-d375-42a3-9346-7a4bc1f82f44\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6dc886bc8bvrdcp"
Sep 09 21:06:36.373434 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:06:36.373416    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de677c34-d375-42a3-9346-7a4bc1f82f44-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6dc886bc8bvrdcp\" (UID: \"de677c34-d375-42a3-9346-7a4bc1f82f44\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6dc886bc8bvrdcp"
Sep 09 21:06:36.379298 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:06:36.379278    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2h6l5\" (UniqueName: \"kubernetes.io/projected/de677c34-d375-42a3-9346-7a4bc1f82f44-kube-api-access-2h6l5\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6dc886bc8bvrdcp\" (UID: \"de677c34-d375-42a3-9346-7a4bc1f82f44\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6dc886bc8bvrdcp"
Sep 09 21:06:36.542020 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:06:36.541929    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6dc886bc8bvrdcp"
Sep 09 21:06:36.827763 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:06:36.827737    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6dc886bc8bvrdcp"]
Sep 09 21:06:36.829787 ip-10-0-134-72 kubenswrapper[2576]: W0909 21:06:36.829754    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde677c34_d375_42a3_9346_7a4bc1f82f44.slice/crio-2034d7ca35933a3fe54bf9e39061c762b8b3a083e69ddea57747a72c27935bc1 WatchSource:0}: Error finding container 2034d7ca35933a3fe54bf9e39061c762b8b3a083e69ddea57747a72c27935bc1: Status 404 returned error can't find the container with id 2034d7ca35933a3fe54bf9e39061c762b8b3a083e69ddea57747a72c27935bc1
Sep 09 21:06:37.254143 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:06:37.254105    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6dc886bc8bvrdcp" event={"ID":"de677c34-d375-42a3-9346-7a4bc1f82f44","Type":"ContainerStarted","Data":"cefe101fc04436cb8235b368fef48b2b51d021e5592ee2c9b3057672128601a6"}
Sep 09 21:06:37.254630 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:06:37.254174    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6dc886bc8bvrdcp" event={"ID":"de677c34-d375-42a3-9346-7a4bc1f82f44","Type":"ContainerStarted","Data":"2034d7ca35933a3fe54bf9e39061c762b8b3a083e69ddea57747a72c27935bc1"}
Sep 09 21:06:37.987805 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:06:37.987770    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:06:39.267115 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:06:39.267028    2576 generic.go:356] "Generic (PLEG): container finished" podID="de677c34-d375-42a3-9346-7a4bc1f82f44" containerID="cefe101fc04436cb8235b368fef48b2b51d021e5592ee2c9b3057672128601a6" exitCode=0
Sep 09 21:06:39.267115 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:06:39.267104    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6dc886bc8bvrdcp" event={"ID":"de677c34-d375-42a3-9346-7a4bc1f82f44","Type":"ContainerDied","Data":"cefe101fc04436cb8235b368fef48b2b51d021e5592ee2c9b3057672128601a6"}
Sep 09 21:06:40.320777 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:06:40.320627    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:06:40.665772 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:06:40.665709    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:06:57.312334 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:06:57.312283    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:06:57.506223 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:06:57.506184    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:07:08.453606 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:07:08.453570    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6dc886bc8bvrdcp" event={"ID":"de677c34-d375-42a3-9346-7a4bc1f82f44","Type":"ContainerStarted","Data":"f4d01076973d719eb0882b61b79f70598a1a9a8a5b7bc2b9d965b4a61f5b6afd"}
Sep 09 21:07:08.492184 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:07:08.492118    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6dc886bc8bvrdcp" podStartSLOduration=1.671964448 podStartE2EDuration="32.492096633s" totalImagesPullingTime="28.731002376s" totalInitContainerRuntime="2.089129809s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:06:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:07:08.489707651 +0000 UTC m=+871.603393510" watchObservedRunningTime="2026-09-09 21:07:08.492096633 +0000 UTC m=+871.605782500"
Sep 09 21:07:08.542350 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:07:08.542286    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6dc886bc8bvrdcp"
Sep 09 21:07:16.542354 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:07:16.542318    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6dc886bc8bvrdcp"
Sep 09 21:07:16.543591 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:07:16.543558    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6dc886bc8bvrdcp" podUID="de677c34-d375-42a3-9346-7a4bc1f82f44" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/health\": dial tcp 10.132.0.43:8000: connect: connection refused"
Sep 09 21:07:21.680939 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:07:21.680882    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:07:21.765024 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:07:21.764986    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:07:26.542877 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:07:26.542828    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6dc886bc8bvrdcp" podUID="de677c34-d375-42a3-9346-7a4bc1f82f44" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/health\": dial tcp 10.132.0.43:8000: connect: connection refused"
Sep 09 21:07:36.273320 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:07:36.273280    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:07:36.543443 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:07:36.543298    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6dc886bc8bvrdcp" podUID="de677c34-d375-42a3-9346-7a4bc1f82f44" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/health\": dial tcp 10.132.0.43:8000: connect: connection refused"
Sep 09 21:07:46.542991 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:07:46.542941    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6dc886bc8bvrdcp" podUID="de677c34-d375-42a3-9346-7a4bc1f82f44" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/health\": dial tcp 10.132.0.43:8000: connect: connection refused"
Sep 09 21:07:56.542427 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:07:56.542348    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6dc886bc8bvrdcp" podUID="de677c34-d375-42a3-9346-7a4bc1f82f44" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/health\": dial tcp 10.132.0.43:8000: connect: connection refused"
Sep 09 21:08:06.543020 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:08:06.542978    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6dc886bc8bvrdcp" podUID="de677c34-d375-42a3-9346-7a4bc1f82f44" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/health\": dial tcp 10.132.0.43:8000: connect: connection refused"
Sep 09 21:08:08.161470 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:08:08.161431    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:08:08.263034 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:08:08.263000    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:08:11.707950 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:08:11.707904    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:08:11.805438 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:08:11.805399    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:08:16.542601 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:08:16.542555    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6dc886bc8bvrdcp" podUID="de677c34-d375-42a3-9346-7a4bc1f82f44" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/health\": dial tcp 10.132.0.43:8000: connect: connection refused"
Sep 09 21:08:18.968565 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:08:18.968520    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:08:19.448816 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:08:19.448774    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:08:19.751922 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:08:19.751836    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:08:25.195513 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:08:25.195468    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:08:25.295437 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:08:25.295401    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:08:26.096304 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:08:26.096270    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:08:26.543394 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:08:26.543284    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6dc886bc8bvrdcp" podUID="de677c34-d375-42a3-9346-7a4bc1f82f44" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/health\": dial tcp 10.132.0.43:8000: connect: connection refused"
Sep 09 21:08:36.542764 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:08:36.542728    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6dc886bc8bvrdcp" podUID="de677c34-d375-42a3-9346-7a4bc1f82f44" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/health\": dial tcp 10.132.0.43:8000: connect: connection refused"
Sep 09 21:08:46.549493 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:08:46.549446    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6dc886bc8bvrdcp" podUID="de677c34-d375-42a3-9346-7a4bc1f82f44" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/health\": dial tcp 10.132.0.43:8000: connect: connection refused"
Sep 09 21:08:56.552251 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:08:56.552221    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6dc886bc8bvrdcp"
Sep 09 21:08:56.560456 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:08:56.560430    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6dc886bc8bvrdcp"
Sep 09 21:08:57.709359 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:08:57.709316    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:08:57.809215 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:08:57.809178    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:09:00.554422 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:09:00.554325    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6dc886bc8bvrdcp"]
Sep 09 21:09:00.595756 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:09:00.595724    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:09:15.577408 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:09:15.577337    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6dc886bc8bvrdcp" podUID="de677c34-d375-42a3-9346-7a4bc1f82f44" containerName="main" containerID="cri-o://f4d01076973d719eb0882b61b79f70598a1a9a8a5b7bc2b9d965b4a61f5b6afd" gracePeriod=45
Sep 09 21:09:16.314586 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:09:16.314563    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6dc886bc8bvrdcp"
Sep 09 21:09:16.410215 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:09:16.410185    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/de677c34-d375-42a3-9346-7a4bc1f82f44-kserve-provision-location\") pod \"de677c34-d375-42a3-9346-7a4bc1f82f44\" (UID: \"de677c34-d375-42a3-9346-7a4bc1f82f44\") "
Sep 09 21:09:16.410215 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:09:16.410215    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2h6l5\" (UniqueName: \"kubernetes.io/projected/de677c34-d375-42a3-9346-7a4bc1f82f44-kube-api-access-2h6l5\") pod \"de677c34-d375-42a3-9346-7a4bc1f82f44\" (UID: \"de677c34-d375-42a3-9346-7a4bc1f82f44\") "
Sep 09 21:09:16.410456 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:09:16.410236    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de677c34-d375-42a3-9346-7a4bc1f82f44-dshm\") pod \"de677c34-d375-42a3-9346-7a4bc1f82f44\" (UID: \"de677c34-d375-42a3-9346-7a4bc1f82f44\") "
Sep 09 21:09:16.410456 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:09:16.410313    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/de677c34-d375-42a3-9346-7a4bc1f82f44-cabundle-cert\") pod \"de677c34-d375-42a3-9346-7a4bc1f82f44\" (UID: \"de677c34-d375-42a3-9346-7a4bc1f82f44\") "
Sep 09 21:09:16.410456 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:09:16.410342    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de677c34-d375-42a3-9346-7a4bc1f82f44-home\") pod \"de677c34-d375-42a3-9346-7a4bc1f82f44\" (UID: \"de677c34-d375-42a3-9346-7a4bc1f82f44\") "
Sep 09 21:09:16.410456 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:09:16.410410    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de677c34-d375-42a3-9346-7a4bc1f82f44-model-cache\") pod \"de677c34-d375-42a3-9346-7a4bc1f82f44\" (UID: \"de677c34-d375-42a3-9346-7a4bc1f82f44\") "
Sep 09 21:09:16.410456 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:09:16.410455    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de677c34-d375-42a3-9346-7a4bc1f82f44-tls-certs\") pod \"de677c34-d375-42a3-9346-7a4bc1f82f44\" (UID: \"de677c34-d375-42a3-9346-7a4bc1f82f44\") "
Sep 09 21:09:16.410719 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:09:16.410483    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de677c34-d375-42a3-9346-7a4bc1f82f44-tmp-dir\") pod \"de677c34-d375-42a3-9346-7a4bc1f82f44\" (UID: \"de677c34-d375-42a3-9346-7a4bc1f82f44\") "
Sep 09 21:09:16.410719 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:09:16.410692    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de677c34-d375-42a3-9346-7a4bc1f82f44-model-cache" (OuterVolumeSpecName: "model-cache") pod "de677c34-d375-42a3-9346-7a4bc1f82f44" (UID: "de677c34-d375-42a3-9346-7a4bc1f82f44"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:09:16.410832 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:09:16.410781    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de677c34-d375-42a3-9346-7a4bc1f82f44-model-cache\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:09:16.411029 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:09:16.411001    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de677c34-d375-42a3-9346-7a4bc1f82f44-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "de677c34-d375-42a3-9346-7a4bc1f82f44" (UID: "de677c34-d375-42a3-9346-7a4bc1f82f44"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:09:16.411119 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:09:16.411090    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de677c34-d375-42a3-9346-7a4bc1f82f44-home" (OuterVolumeSpecName: "home") pod "de677c34-d375-42a3-9346-7a4bc1f82f44" (UID: "de677c34-d375-42a3-9346-7a4bc1f82f44"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:09:16.412833 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:09:16.412807    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de677c34-d375-42a3-9346-7a4bc1f82f44-dshm" (OuterVolumeSpecName: "dshm") pod "de677c34-d375-42a3-9346-7a4bc1f82f44" (UID: "de677c34-d375-42a3-9346-7a4bc1f82f44"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:09:16.412970 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:09:16.412924    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de677c34-d375-42a3-9346-7a4bc1f82f44-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "de677c34-d375-42a3-9346-7a4bc1f82f44" (UID: "de677c34-d375-42a3-9346-7a4bc1f82f44"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:09:16.413039 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:09:16.412996    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de677c34-d375-42a3-9346-7a4bc1f82f44-kube-api-access-2h6l5" (OuterVolumeSpecName: "kube-api-access-2h6l5") pod "de677c34-d375-42a3-9346-7a4bc1f82f44" (UID: "de677c34-d375-42a3-9346-7a4bc1f82f44"). InnerVolumeSpecName "kube-api-access-2h6l5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:09:16.423282 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:09:16.423258    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de677c34-d375-42a3-9346-7a4bc1f82f44-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "de677c34-d375-42a3-9346-7a4bc1f82f44" (UID: "de677c34-d375-42a3-9346-7a4bc1f82f44"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:09:16.430074 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:09:16.430054    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de677c34-d375-42a3-9346-7a4bc1f82f44-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "de677c34-d375-42a3-9346-7a4bc1f82f44" (UID: "de677c34-d375-42a3-9346-7a4bc1f82f44"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:09:16.511899 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:09:16.511876    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/de677c34-d375-42a3-9346-7a4bc1f82f44-kserve-provision-location\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:09:16.511899 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:09:16.511898    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2h6l5\" (UniqueName: \"kubernetes.io/projected/de677c34-d375-42a3-9346-7a4bc1f82f44-kube-api-access-2h6l5\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:09:16.512028 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:09:16.511908    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de677c34-d375-42a3-9346-7a4bc1f82f44-dshm\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:09:16.512028 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:09:16.511917    2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/de677c34-d375-42a3-9346-7a4bc1f82f44-cabundle-cert\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:09:16.512028 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:09:16.511926    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de677c34-d375-42a3-9346-7a4bc1f82f44-home\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:09:16.512028 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:09:16.511933    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de677c34-d375-42a3-9346-7a4bc1f82f44-tls-certs\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:09:16.512028 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:09:16.511940    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de677c34-d375-42a3-9346-7a4bc1f82f44-tmp-dir\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:09:17.130620 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:09:17.130577    2576 generic.go:356] "Generic (PLEG): container finished" podID="de677c34-d375-42a3-9346-7a4bc1f82f44" containerID="f4d01076973d719eb0882b61b79f70598a1a9a8a5b7bc2b9d965b4a61f5b6afd" exitCode=0
Sep 09 21:09:17.131032 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:09:17.130635    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6dc886bc8bvrdcp"
Sep 09 21:09:17.131032 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:09:17.130655    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6dc886bc8bvrdcp" event={"ID":"de677c34-d375-42a3-9346-7a4bc1f82f44","Type":"ContainerDied","Data":"f4d01076973d719eb0882b61b79f70598a1a9a8a5b7bc2b9d965b4a61f5b6afd"}
Sep 09 21:09:17.131032 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:09:17.130709    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6dc886bc8bvrdcp" event={"ID":"de677c34-d375-42a3-9346-7a4bc1f82f44","Type":"ContainerDied","Data":"2034d7ca35933a3fe54bf9e39061c762b8b3a083e69ddea57747a72c27935bc1"}
Sep 09 21:09:17.131032 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:09:17.130731    2576 scope.go:117] "RemoveContainer" containerID="f4d01076973d719eb0882b61b79f70598a1a9a8a5b7bc2b9d965b4a61f5b6afd"
Sep 09 21:09:17.142336 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:09:17.142315    2576 scope.go:117] "RemoveContainer" containerID="cefe101fc04436cb8235b368fef48b2b51d021e5592ee2c9b3057672128601a6"
Sep 09 21:09:17.150931 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:09:17.150910    2576 scope.go:117] "RemoveContainer" containerID="f4d01076973d719eb0882b61b79f70598a1a9a8a5b7bc2b9d965b4a61f5b6afd"
Sep 09 21:09:17.151215 ip-10-0-134-72 kubenswrapper[2576]: E0909 21:09:17.151196    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4d01076973d719eb0882b61b79f70598a1a9a8a5b7bc2b9d965b4a61f5b6afd\": container with ID starting with f4d01076973d719eb0882b61b79f70598a1a9a8a5b7bc2b9d965b4a61f5b6afd not found: ID does not exist" containerID="f4d01076973d719eb0882b61b79f70598a1a9a8a5b7bc2b9d965b4a61f5b6afd"
Sep 09 21:09:17.151294 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:09:17.151228    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4d01076973d719eb0882b61b79f70598a1a9a8a5b7bc2b9d965b4a61f5b6afd"} err="failed to get container status \"f4d01076973d719eb0882b61b79f70598a1a9a8a5b7bc2b9d965b4a61f5b6afd\": rpc error: code = NotFound desc = could not find container \"f4d01076973d719eb0882b61b79f70598a1a9a8a5b7bc2b9d965b4a61f5b6afd\": container with ID starting with f4d01076973d719eb0882b61b79f70598a1a9a8a5b7bc2b9d965b4a61f5b6afd not found: ID does not exist"
Sep 09 21:09:17.151294 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:09:17.151252    2576 scope.go:117] "RemoveContainer" containerID="cefe101fc04436cb8235b368fef48b2b51d021e5592ee2c9b3057672128601a6"
Sep 09 21:09:17.151512 ip-10-0-134-72 kubenswrapper[2576]: E0909 21:09:17.151491    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cefe101fc04436cb8235b368fef48b2b51d021e5592ee2c9b3057672128601a6\": container with ID starting with cefe101fc04436cb8235b368fef48b2b51d021e5592ee2c9b3057672128601a6 not found: ID does not exist" containerID="cefe101fc04436cb8235b368fef48b2b51d021e5592ee2c9b3057672128601a6"
Sep 09 21:09:17.151554 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:09:17.151520    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cefe101fc04436cb8235b368fef48b2b51d021e5592ee2c9b3057672128601a6"} err="failed to get container status \"cefe101fc04436cb8235b368fef48b2b51d021e5592ee2c9b3057672128601a6\": rpc error: code = NotFound desc = could not find container \"cefe101fc04436cb8235b368fef48b2b51d021e5592ee2c9b3057672128601a6\": container with ID starting with cefe101fc04436cb8235b368fef48b2b51d021e5592ee2c9b3057672128601a6 not found: ID does not exist"
Sep 09 21:09:17.168688 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:09:17.168654    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6dc886bc8bvrdcp"]
Sep 09 21:09:17.181134 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:09:17.181113    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6dc886bc8bvrdcp"]
Sep 09 21:09:17.410377 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:09:17.410292    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="de677c34-d375-42a3-9346-7a4bc1f82f44" path="/var/lib/kubelet/pods/de677c34-d375-42a3-9346-7a4bc1f82f44/volumes"
Sep 09 21:09:20.635104 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:09:20.635068    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:09:20.731279 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:09:20.731242    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:09:27.277093 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:09:27.277056    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:09:31.835518 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:09:31.835483    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:09:31.990616 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:09:31.990582    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:09:32.119554 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:09:32.119520    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:09:32.163310 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:09:32.163282    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 09 21:09:32.163661 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:09:32.163647    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de677c34-d375-42a3-9346-7a4bc1f82f44" containerName="main"
Sep 09 21:09:32.163722 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:09:32.163663    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="de677c34-d375-42a3-9346-7a4bc1f82f44" containerName="main"
Sep 09 21:09:32.163722 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:09:32.163675    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de677c34-d375-42a3-9346-7a4bc1f82f44" containerName="storage-initializer"
Sep 09 21:09:32.163722 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:09:32.163681    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="de677c34-d375-42a3-9346-7a4bc1f82f44" containerName="storage-initializer"
Sep 09 21:09:32.163828 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:09:32.163734    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="de677c34-d375-42a3-9346-7a4bc1f82f44" containerName="main"
Sep 09 21:09:32.168241 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:09:32.168225    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 09 21:09:32.171049 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:09:32.171016    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"kube-root-ca.crt\""
Sep 09 21:09:32.171268 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:09:32.171254    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"seaweedfs-s3-creds\""
Sep 09 21:09:32.172108 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:09:32.172083    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisvc-model-fb-opt-125m-route-dc21cb-57cf7736-dockercfg-vf5nm\""
Sep 09 21:09:32.172194 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:09:32.172103    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs\""
Sep 09 21:09:32.172250 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:09:32.172197    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"default-dockercfg-vvgkn\""
Sep 09 21:09:32.172250 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:09:32.172202    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"odh-kserve-custom-ca-bundle\""
Sep 09 21:09:32.172344 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:09:32.172299    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"openshift-service-ca.crt\""
Sep 09 21:09:32.182648 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:09:32.182627    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 09 21:09:32.345259 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:09:32.345228    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db7ecaa9-1f5f-4957-9145-e02b5c9c1b43-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"db7ecaa9-1f5f-4957-9145-e02b5c9c1b43\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 09 21:09:32.345508 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:09:32.345267    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db7ecaa9-1f5f-4957-9145-e02b5c9c1b43-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"db7ecaa9-1f5f-4957-9145-e02b5c9c1b43\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 09 21:09:32.345508 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:09:32.345287    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db7ecaa9-1f5f-4957-9145-e02b5c9c1b43-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"db7ecaa9-1f5f-4957-9145-e02b5c9c1b43\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 09 21:09:32.345508 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:09:32.345352    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db7ecaa9-1f5f-4957-9145-e02b5c9c1b43-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"db7ecaa9-1f5f-4957-9145-e02b5c9c1b43\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 09 21:09:32.345508 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:09:32.345409    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db7ecaa9-1f5f-4957-9145-e02b5c9c1b43-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"db7ecaa9-1f5f-4957-9145-e02b5c9c1b43\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 09 21:09:32.345508 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:09:32.345439    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7mk4\" (UniqueName: \"kubernetes.io/projected/db7ecaa9-1f5f-4957-9145-e02b5c9c1b43-kube-api-access-z7mk4\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"db7ecaa9-1f5f-4957-9145-e02b5c9c1b43\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 09 21:09:32.345508 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:09:32.345460    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/db7ecaa9-1f5f-4957-9145-e02b5c9c1b43-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"db7ecaa9-1f5f-4957-9145-e02b5c9c1b43\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 09 21:09:32.345508 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:09:32.345503    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/db7ecaa9-1f5f-4957-9145-e02b5c9c1b43-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"db7ecaa9-1f5f-4957-9145-e02b5c9c1b43\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 09 21:09:32.378522 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:09:32.378447    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:09:32.446913 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:09:32.446877    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db7ecaa9-1f5f-4957-9145-e02b5c9c1b43-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"db7ecaa9-1f5f-4957-9145-e02b5c9c1b43\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 09 21:09:32.446913 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:09:32.446915    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db7ecaa9-1f5f-4957-9145-e02b5c9c1b43-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"db7ecaa9-1f5f-4957-9145-e02b5c9c1b43\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 09 21:09:32.447169 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:09:32.446935    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db7ecaa9-1f5f-4957-9145-e02b5c9c1b43-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"db7ecaa9-1f5f-4957-9145-e02b5c9c1b43\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 09 21:09:32.447169 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:09:32.446975    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db7ecaa9-1f5f-4957-9145-e02b5c9c1b43-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"db7ecaa9-1f5f-4957-9145-e02b5c9c1b43\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 09 21:09:32.447169 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:09:32.447001    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db7ecaa9-1f5f-4957-9145-e02b5c9c1b43-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"db7ecaa9-1f5f-4957-9145-e02b5c9c1b43\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 09 21:09:32.447169 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:09:32.447041    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7mk4\" (UniqueName: \"kubernetes.io/projected/db7ecaa9-1f5f-4957-9145-e02b5c9c1b43-kube-api-access-z7mk4\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"db7ecaa9-1f5f-4957-9145-e02b5c9c1b43\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 09 21:09:32.447169 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:09:32.447069    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/db7ecaa9-1f5f-4957-9145-e02b5c9c1b43-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"db7ecaa9-1f5f-4957-9145-e02b5c9c1b43\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 09 21:09:32.447169 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:09:32.447096    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/db7ecaa9-1f5f-4957-9145-e02b5c9c1b43-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"db7ecaa9-1f5f-4957-9145-e02b5c9c1b43\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 09 21:09:32.447436 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:09:32.447251    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db7ecaa9-1f5f-4957-9145-e02b5c9c1b43-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"db7ecaa9-1f5f-4957-9145-e02b5c9c1b43\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 09 21:09:32.447436 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:09:32.447294    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db7ecaa9-1f5f-4957-9145-e02b5c9c1b43-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"db7ecaa9-1f5f-4957-9145-e02b5c9c1b43\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 09 21:09:32.447516 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:09:32.447448    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/db7ecaa9-1f5f-4957-9145-e02b5c9c1b43-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"db7ecaa9-1f5f-4957-9145-e02b5c9c1b43\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 09 21:09:32.447610 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:09:32.447591    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db7ecaa9-1f5f-4957-9145-e02b5c9c1b43-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"db7ecaa9-1f5f-4957-9145-e02b5c9c1b43\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 09 21:09:32.447658 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:09:32.447644    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/db7ecaa9-1f5f-4957-9145-e02b5c9c1b43-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"db7ecaa9-1f5f-4957-9145-e02b5c9c1b43\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 09 21:09:32.449424 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:09:32.449404    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db7ecaa9-1f5f-4957-9145-e02b5c9c1b43-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"db7ecaa9-1f5f-4957-9145-e02b5c9c1b43\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 09 21:09:32.449609 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:09:32.449594    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db7ecaa9-1f5f-4957-9145-e02b5c9c1b43-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"db7ecaa9-1f5f-4957-9145-e02b5c9c1b43\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 09 21:09:32.460126 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:09:32.460105    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7mk4\" (UniqueName: \"kubernetes.io/projected/db7ecaa9-1f5f-4957-9145-e02b5c9c1b43-kube-api-access-z7mk4\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"db7ecaa9-1f5f-4957-9145-e02b5c9c1b43\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 09 21:09:32.478769 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:09:32.478748    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 09 21:09:32.763837 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:09:32.763812    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 09 21:09:32.766377 ip-10-0-134-72 kubenswrapper[2576]: W0909 21:09:32.766326    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb7ecaa9_1f5f_4957_9145_e02b5c9c1b43.slice/crio-41c6103bd469b3274e4f814c2f78189b7d326f848852550180555217b35ec74a WatchSource:0}: Error finding container 41c6103bd469b3274e4f814c2f78189b7d326f848852550180555217b35ec74a: Status 404 returned error can't find the container with id 41c6103bd469b3274e4f814c2f78189b7d326f848852550180555217b35ec74a
Sep 09 21:09:33.215261 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:09:33.215224    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"db7ecaa9-1f5f-4957-9145-e02b5c9c1b43","Type":"ContainerStarted","Data":"797794d815bfc6cb1e5d9354ef9e78b37ad589dc07f4f3dfae0b2284afeeac78"}
Sep 09 21:09:33.215654 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:09:33.215296    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"db7ecaa9-1f5f-4957-9145-e02b5c9c1b43","Type":"ContainerStarted","Data":"41c6103bd469b3274e4f814c2f78189b7d326f848852550180555217b35ec74a"}
Sep 09 21:09:34.946680 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:09:34.946623    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:09:35.023907 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:09:35.023871    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:09:35.226790 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:09:35.226708    2576 generic.go:356] "Generic (PLEG): container finished" podID="db7ecaa9-1f5f-4957-9145-e02b5c9c1b43" containerID="797794d815bfc6cb1e5d9354ef9e78b37ad589dc07f4f3dfae0b2284afeeac78" exitCode=0
Sep 09 21:09:35.226790 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:09:35.226782    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"db7ecaa9-1f5f-4957-9145-e02b5c9c1b43","Type":"ContainerDied","Data":"797794d815bfc6cb1e5d9354ef9e78b37ad589dc07f4f3dfae0b2284afeeac78"}
Sep 09 21:09:36.233723 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:09:36.233684    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"db7ecaa9-1f5f-4957-9145-e02b5c9c1b43","Type":"ContainerStarted","Data":"a58d7f7408fcd2ec1adced9a078741bac6b4e918931182b3cfa61bb01f833023"}
Sep 09 21:09:36.273160 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:09:36.273104    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podStartSLOduration=2.183469199 podStartE2EDuration="4.273087012s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.089617813s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:09:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:09:36.272710844 +0000 UTC m=+1019.386396704" watchObservedRunningTime="2026-09-09 21:09:36.273087012 +0000 UTC m=+1019.386772895"
Sep 09 21:09:36.479134 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:09:36.479094    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 09 21:09:42.479227 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:09:42.479180    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 09 21:09:42.480685 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:09:42.480660    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="db7ecaa9-1f5f-4957-9145-e02b5c9c1b43" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/health\": dial tcp 10.132.0.44:8000: connect: connection refused"
Sep 09 21:09:47.688712 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:09:47.688673    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:09:48.025310 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:09:48.025275    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-zmrmc"]
Sep 09 21:09:48.030154 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:09:48.030121    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-zmrmc"
Sep 09 21:09:48.036321 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:09:48.036297    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"router-gateway-1-openshift-default-dockercfg-tk9l6\""
Sep 09 21:09:48.037071 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:09:48.037052    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"kube-root-ca.crt\""
Sep 09 21:09:48.037393 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:09:48.037355    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"openshift-service-ca.crt\""
Sep 09 21:09:48.037484 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:09:48.037403    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"openshift-gw-ca-root-cert\""
Sep 09 21:09:48.044490 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:09:48.044470    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-zmrmc"]
Sep 09 21:09:48.086264 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:09:48.086235    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12af7609-714f-4b2c-a4b0-cbf4870d02d2-workload-socket\") pod \"router-gateway-1-openshift-default-5b4d9458bd-zmrmc\" (UID: \"12af7609-714f-4b2c-a4b0-cbf4870d02d2\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-zmrmc"
Sep 09 21:09:48.086419 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:09:48.086276    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12af7609-714f-4b2c-a4b0-cbf4870d02d2-workload-certs\") pod \"router-gateway-1-openshift-default-5b4d9458bd-zmrmc\" (UID: \"12af7609-714f-4b2c-a4b0-cbf4870d02d2\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-zmrmc"
Sep 09 21:09:48.086419 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:09:48.086399    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12af7609-714f-4b2c-a4b0-cbf4870d02d2-istio-envoy\") pod \"router-gateway-1-openshift-default-5b4d9458bd-zmrmc\" (UID: \"12af7609-714f-4b2c-a4b0-cbf4870d02d2\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-zmrmc"
Sep 09 21:09:48.086506 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:09:48.086456    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12af7609-714f-4b2c-a4b0-cbf4870d02d2-istio-podinfo\") pod \"router-gateway-1-openshift-default-5b4d9458bd-zmrmc\" (UID: \"12af7609-714f-4b2c-a4b0-cbf4870d02d2\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-zmrmc"
Sep 09 21:09:48.086543 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:09:48.086512    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12af7609-714f-4b2c-a4b0-cbf4870d02d2-istio-token\") pod \"router-gateway-1-openshift-default-5b4d9458bd-zmrmc\" (UID: \"12af7609-714f-4b2c-a4b0-cbf4870d02d2\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-zmrmc"
Sep 09 21:09:48.086577 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:09:48.086545    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12af7609-714f-4b2c-a4b0-cbf4870d02d2-istio-data\") pod \"router-gateway-1-openshift-default-5b4d9458bd-zmrmc\" (UID: \"12af7609-714f-4b2c-a4b0-cbf4870d02d2\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-zmrmc"
Sep 09 21:09:48.086612 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:09:48.086596    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12af7609-714f-4b2c-a4b0-cbf4870d02d2-credential-socket\") pod \"router-gateway-1-openshift-default-5b4d9458bd-zmrmc\" (UID: \"12af7609-714f-4b2c-a4b0-cbf4870d02d2\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-zmrmc"
Sep 09 21:09:48.086646 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:09:48.086620    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12af7609-714f-4b2c-a4b0-cbf4870d02d2-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-5b4d9458bd-zmrmc\" (UID: \"12af7609-714f-4b2c-a4b0-cbf4870d02d2\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-zmrmc"
Sep 09 21:09:48.086682 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:09:48.086652    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctngs\" (UniqueName: \"kubernetes.io/projected/12af7609-714f-4b2c-a4b0-cbf4870d02d2-kube-api-access-ctngs\") pod \"router-gateway-1-openshift-default-5b4d9458bd-zmrmc\" (UID: \"12af7609-714f-4b2c-a4b0-cbf4870d02d2\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-zmrmc"
Sep 09 21:09:48.188001 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:09:48.187964    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12af7609-714f-4b2c-a4b0-cbf4870d02d2-workload-socket\") pod \"router-gateway-1-openshift-default-5b4d9458bd-zmrmc\" (UID: \"12af7609-714f-4b2c-a4b0-cbf4870d02d2\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-zmrmc"
Sep 09 21:09:48.188001 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:09:48.188007    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12af7609-714f-4b2c-a4b0-cbf4870d02d2-workload-certs\") pod \"router-gateway-1-openshift-default-5b4d9458bd-zmrmc\" (UID: \"12af7609-714f-4b2c-a4b0-cbf4870d02d2\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-zmrmc"
Sep 09 21:09:48.188219 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:09:48.188040    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12af7609-714f-4b2c-a4b0-cbf4870d02d2-istio-envoy\") pod \"router-gateway-1-openshift-default-5b4d9458bd-zmrmc\" (UID: \"12af7609-714f-4b2c-a4b0-cbf4870d02d2\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-zmrmc"
Sep 09 21:09:48.188219 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:09:48.188074    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12af7609-714f-4b2c-a4b0-cbf4870d02d2-istio-podinfo\") pod \"router-gateway-1-openshift-default-5b4d9458bd-zmrmc\" (UID: \"12af7609-714f-4b2c-a4b0-cbf4870d02d2\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-zmrmc"
Sep 09 21:09:48.188219 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:09:48.188104    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12af7609-714f-4b2c-a4b0-cbf4870d02d2-istio-token\") pod \"router-gateway-1-openshift-default-5b4d9458bd-zmrmc\" (UID: \"12af7609-714f-4b2c-a4b0-cbf4870d02d2\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-zmrmc"
Sep 09 21:09:48.188219 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:09:48.188144    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12af7609-714f-4b2c-a4b0-cbf4870d02d2-istio-data\") pod \"router-gateway-1-openshift-default-5b4d9458bd-zmrmc\" (UID: \"12af7609-714f-4b2c-a4b0-cbf4870d02d2\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-zmrmc"
Sep 09 21:09:48.188219 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:09:48.188172    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12af7609-714f-4b2c-a4b0-cbf4870d02d2-credential-socket\") pod \"router-gateway-1-openshift-default-5b4d9458bd-zmrmc\" (UID: \"12af7609-714f-4b2c-a4b0-cbf4870d02d2\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-zmrmc"
Sep 09 21:09:48.188522 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:09:48.188456    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12af7609-714f-4b2c-a4b0-cbf4870d02d2-workload-certs\") pod \"router-gateway-1-openshift-default-5b4d9458bd-zmrmc\" (UID: \"12af7609-714f-4b2c-a4b0-cbf4870d02d2\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-zmrmc"
Sep 09 21:09:48.188580 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:09:48.188557    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12af7609-714f-4b2c-a4b0-cbf4870d02d2-workload-socket\") pod \"router-gateway-1-openshift-default-5b4d9458bd-zmrmc\" (UID: \"12af7609-714f-4b2c-a4b0-cbf4870d02d2\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-zmrmc"
Sep 09 21:09:48.188638 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:09:48.188609    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12af7609-714f-4b2c-a4b0-cbf4870d02d2-credential-socket\") pod \"router-gateway-1-openshift-default-5b4d9458bd-zmrmc\" (UID: \"12af7609-714f-4b2c-a4b0-cbf4870d02d2\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-zmrmc"
Sep 09 21:09:48.188694 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:09:48.188674    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12af7609-714f-4b2c-a4b0-cbf4870d02d2-istio-data\") pod \"router-gateway-1-openshift-default-5b4d9458bd-zmrmc\" (UID: \"12af7609-714f-4b2c-a4b0-cbf4870d02d2\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-zmrmc"
Sep 09 21:09:48.188749 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:09:48.188426    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12af7609-714f-4b2c-a4b0-cbf4870d02d2-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-5b4d9458bd-zmrmc\" (UID: \"12af7609-714f-4b2c-a4b0-cbf4870d02d2\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-zmrmc"
Sep 09 21:09:48.188803 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:09:48.188765    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctngs\" (UniqueName: \"kubernetes.io/projected/12af7609-714f-4b2c-a4b0-cbf4870d02d2-kube-api-access-ctngs\") pod \"router-gateway-1-openshift-default-5b4d9458bd-zmrmc\" (UID: \"12af7609-714f-4b2c-a4b0-cbf4870d02d2\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-zmrmc"
Sep 09 21:09:48.189117 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:09:48.189095    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12af7609-714f-4b2c-a4b0-cbf4870d02d2-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-5b4d9458bd-zmrmc\" (UID: \"12af7609-714f-4b2c-a4b0-cbf4870d02d2\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-zmrmc"
Sep 09 21:09:48.190557 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:09:48.190533    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12af7609-714f-4b2c-a4b0-cbf4870d02d2-istio-envoy\") pod \"router-gateway-1-openshift-default-5b4d9458bd-zmrmc\" (UID: \"12af7609-714f-4b2c-a4b0-cbf4870d02d2\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-zmrmc"
Sep 09 21:09:48.191197 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:09:48.191180    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12af7609-714f-4b2c-a4b0-cbf4870d02d2-istio-podinfo\") pod \"router-gateway-1-openshift-default-5b4d9458bd-zmrmc\" (UID: \"12af7609-714f-4b2c-a4b0-cbf4870d02d2\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-zmrmc"
Sep 09 21:09:48.201248 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:09:48.201228    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12af7609-714f-4b2c-a4b0-cbf4870d02d2-istio-token\") pod \"router-gateway-1-openshift-default-5b4d9458bd-zmrmc\" (UID: \"12af7609-714f-4b2c-a4b0-cbf4870d02d2\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-zmrmc"
Sep 09 21:09:48.203223 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:09:48.203199    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctngs\" (UniqueName: \"kubernetes.io/projected/12af7609-714f-4b2c-a4b0-cbf4870d02d2-kube-api-access-ctngs\") pod \"router-gateway-1-openshift-default-5b4d9458bd-zmrmc\" (UID: \"12af7609-714f-4b2c-a4b0-cbf4870d02d2\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-zmrmc"
Sep 09 21:09:48.344318 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:09:48.344279    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-zmrmc"
Sep 09 21:09:48.626894 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:09:48.626871    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-zmrmc"]
Sep 09 21:09:48.629638 ip-10-0-134-72 kubenswrapper[2576]: W0909 21:09:48.629607    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12af7609_714f_4b2c_a4b0_cbf4870d02d2.slice/crio-2e67bdab2056932d7a8af7d3c89165d146393ff29f1379ede76a37d5601a7e98 WatchSource:0}: Error finding container 2e67bdab2056932d7a8af7d3c89165d146393ff29f1379ede76a37d5601a7e98: Status 404 returned error can't find the container with id 2e67bdab2056932d7a8af7d3c89165d146393ff29f1379ede76a37d5601a7e98
Sep 09 21:09:49.015212 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:09:49.015122    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:09:49.303495 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:09:49.303398    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-zmrmc" event={"ID":"12af7609-714f-4b2c-a4b0-cbf4870d02d2","Type":"ContainerStarted","Data":"2e67bdab2056932d7a8af7d3c89165d146393ff29f1379ede76a37d5601a7e98"}
Sep 09 21:09:51.521284 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:09:51.521246    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:09:51.622382 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:09:51.622323    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 09 21:09:51.622505 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:09:51.622426    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 09 21:09:51.622505 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:09:51.622470    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 09 21:09:52.325099 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:09:52.325063    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-zmrmc" event={"ID":"12af7609-714f-4b2c-a4b0-cbf4870d02d2","Type":"ContainerStarted","Data":"dabb13112bce54332b342edaf09915e10ce93f94a182cd36bcecdbeb877f7f4b"}
Sep 09 21:09:52.344973 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:09:52.344941    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-zmrmc"
Sep 09 21:09:52.350164 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:09:52.350136    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-zmrmc"
Sep 09 21:09:52.361299 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:09:52.361243    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-zmrmc" podStartSLOduration=2.370855476 podStartE2EDuration="5.36122328s" totalImagesPullingTime="2.990367804s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:09:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:09:52.357804591 +0000 UTC m=+1035.471490461" watchObservedRunningTime="2026-09-09 21:09:52.36122328 +0000 UTC m=+1035.474909140"
Sep 09 21:09:52.479525 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:09:52.479484    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="db7ecaa9-1f5f-4957-9145-e02b5c9c1b43" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/health\": dial tcp 10.132.0.44:8000: connect: connection refused"
Sep 09 21:09:52.949284 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:09:52.949247    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:09:53.330403 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:09:53.330306    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-zmrmc"
Sep 09 21:09:53.331279 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:09:53.331257    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-zmrmc"
Sep 09 21:09:55.163108 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:09:55.163069    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:10:01.289046 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:01.289007    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:10:01.383300 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:01.383249    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:10:02.480055 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:02.480005    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="db7ecaa9-1f5f-4957-9145-e02b5c9c1b43" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/health\": dial tcp 10.132.0.44:8000: connect: connection refused"
Sep 09 21:10:07.195414 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:07.195382    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-88vvz"]
Sep 09 21:10:07.198656 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:07.198639    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-88vvz"
Sep 09 21:10:07.202317 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:07.202288    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"default-dockercfg-7qzb7\""
Sep 09 21:10:07.202495 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:07.202331    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"gw-sec0c69dceeb48768325d1a53a749e65786-kserve-self-signed-certs\""
Sep 09 21:10:07.211347 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:07.211322    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-88vvz"]
Sep 09 21:10:07.226497 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:07.226470    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:10:07.256728 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:07.256698    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e03347f5-3fb3-4a81-874a-488839b35ebf-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-88vvz\" (UID: \"e03347f5-3fb3-4a81-874a-488839b35ebf\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-88vvz"
Sep 09 21:10:07.256728 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:07.256728    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e03347f5-3fb3-4a81-874a-488839b35ebf-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-88vvz\" (UID: \"e03347f5-3fb3-4a81-874a-488839b35ebf\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-88vvz"
Sep 09 21:10:07.256899 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:07.256818    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e03347f5-3fb3-4a81-874a-488839b35ebf-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-88vvz\" (UID: \"e03347f5-3fb3-4a81-874a-488839b35ebf\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-88vvz"
Sep 09 21:10:07.256899 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:07.256847    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e03347f5-3fb3-4a81-874a-488839b35ebf-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-88vvz\" (UID: \"e03347f5-3fb3-4a81-874a-488839b35ebf\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-88vvz"
Sep 09 21:10:07.256899 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:07.256884    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e03347f5-3fb3-4a81-874a-488839b35ebf-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-88vvz\" (UID: \"e03347f5-3fb3-4a81-874a-488839b35ebf\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-88vvz"
Sep 09 21:10:07.257001 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:07.256936    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdzcs\" (UniqueName: \"kubernetes.io/projected/e03347f5-3fb3-4a81-874a-488839b35ebf-kube-api-access-cdzcs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-88vvz\" (UID: \"e03347f5-3fb3-4a81-874a-488839b35ebf\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-88vvz"
Sep 09 21:10:07.358017 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:07.357983    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e03347f5-3fb3-4a81-874a-488839b35ebf-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-88vvz\" (UID: \"e03347f5-3fb3-4a81-874a-488839b35ebf\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-88vvz"
Sep 09 21:10:07.358017 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:07.358016    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e03347f5-3fb3-4a81-874a-488839b35ebf-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-88vvz\" (UID: \"e03347f5-3fb3-4a81-874a-488839b35ebf\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-88vvz"
Sep 09 21:10:07.358256 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:07.358046    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e03347f5-3fb3-4a81-874a-488839b35ebf-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-88vvz\" (UID: \"e03347f5-3fb3-4a81-874a-488839b35ebf\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-88vvz"
Sep 09 21:10:07.358256 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:07.358083    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdzcs\" (UniqueName: \"kubernetes.io/projected/e03347f5-3fb3-4a81-874a-488839b35ebf-kube-api-access-cdzcs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-88vvz\" (UID: \"e03347f5-3fb3-4a81-874a-488839b35ebf\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-88vvz"
Sep 09 21:10:07.358256 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:07.358105    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e03347f5-3fb3-4a81-874a-488839b35ebf-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-88vvz\" (UID: \"e03347f5-3fb3-4a81-874a-488839b35ebf\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-88vvz"
Sep 09 21:10:07.358256 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:07.358120    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e03347f5-3fb3-4a81-874a-488839b35ebf-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-88vvz\" (UID: \"e03347f5-3fb3-4a81-874a-488839b35ebf\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-88vvz"
Sep 09 21:10:07.358485 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:07.358465    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e03347f5-3fb3-4a81-874a-488839b35ebf-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-88vvz\" (UID: \"e03347f5-3fb3-4a81-874a-488839b35ebf\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-88vvz"
Sep 09 21:10:07.358551 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:07.358507    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e03347f5-3fb3-4a81-874a-488839b35ebf-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-88vvz\" (UID: \"e03347f5-3fb3-4a81-874a-488839b35ebf\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-88vvz"
Sep 09 21:10:07.358608 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:07.358570    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e03347f5-3fb3-4a81-874a-488839b35ebf-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-88vvz\" (UID: \"e03347f5-3fb3-4a81-874a-488839b35ebf\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-88vvz"
Sep 09 21:10:07.360663 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:07.360637    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e03347f5-3fb3-4a81-874a-488839b35ebf-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-88vvz\" (UID: \"e03347f5-3fb3-4a81-874a-488839b35ebf\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-88vvz"
Sep 09 21:10:07.360787 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:07.360769    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e03347f5-3fb3-4a81-874a-488839b35ebf-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-88vvz\" (UID: \"e03347f5-3fb3-4a81-874a-488839b35ebf\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-88vvz"
Sep 09 21:10:07.367149 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:07.367125    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdzcs\" (UniqueName: \"kubernetes.io/projected/e03347f5-3fb3-4a81-874a-488839b35ebf-kube-api-access-cdzcs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-88vvz\" (UID: \"e03347f5-3fb3-4a81-874a-488839b35ebf\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-88vvz"
Sep 09 21:10:07.509168 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:07.509079    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-88vvz"
Sep 09 21:10:07.799224 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:07.799192    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-88vvz"]
Sep 09 21:10:07.801186 ip-10-0-134-72 kubenswrapper[2576]: W0909 21:10:07.801160    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode03347f5_3fb3_4a81_874a_488839b35ebf.slice/crio-217323f3dd99340cf35e1409d150e8c2fcc80d353304e323a0c65ab41c3ecf47 WatchSource:0}: Error finding container 217323f3dd99340cf35e1409d150e8c2fcc80d353304e323a0c65ab41c3ecf47: Status 404 returned error can't find the container with id 217323f3dd99340cf35e1409d150e8c2fcc80d353304e323a0c65ab41c3ecf47
Sep 09 21:10:07.803728 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:07.803708    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 21:10:08.417791 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:08.417754    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-88vvz" event={"ID":"e03347f5-3fb3-4a81-874a-488839b35ebf","Type":"ContainerStarted","Data":"217323f3dd99340cf35e1409d150e8c2fcc80d353304e323a0c65ab41c3ecf47"}
Sep 09 21:10:09.700475 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:09.700428    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:10:09.947869 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:09.947834    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:10:10.036900 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:10.036784    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:10:10.159397 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:10.157862    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:10:10.431974 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:10.431938    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-88vvz" event={"ID":"e03347f5-3fb3-4a81-874a-488839b35ebf","Type":"ContainerStarted","Data":"a5dc0ff68bdab72d5542db5135d45e269a24403793b0495263393b91a2d2517b"}
Sep 09 21:10:10.467655 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:10.467574    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-88vvz" podStartSLOduration=1.017021787 podStartE2EDuration="3.467552402s" totalImagesPullingTime="2.450530615s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:10:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:10:10.462889194 +0000 UTC m=+1053.576575073" watchObservedRunningTime="2026-09-09 21:10:10.467552402 +0000 UTC m=+1053.581238264"
Sep 09 21:10:10.509203 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:10.509167    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-88vvz"
Sep 09 21:10:12.479582 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:12.479530    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="db7ecaa9-1f5f-4957-9145-e02b5c9c1b43" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/health\": dial tcp 10.132.0.44:8000: connect: connection refused"
Sep 09 21:10:12.677999 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:12.677960    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:10:14.687763 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:14.687727    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-88vvz"]
Sep 09 21:10:14.692720 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:14.692671    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-zmrmc"]
Sep 09 21:10:14.692932 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:14.692901    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-zmrmc" podUID="12af7609-714f-4b2c-a4b0-cbf4870d02d2" containerName="istio-proxy" containerID="cri-o://dabb13112bce54332b342edaf09915e10ce93f94a182cd36bcecdbeb877f7f4b" gracePeriod=30
Sep 09 21:10:14.699695 ip-10-0-134-72 kubenswrapper[2576]: E0909 21:10:14.699660    2576 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 09 21:10:14.699695 ip-10-0-134-72 kubenswrapper[2576]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 21:10:14.699695 ip-10-0-134-72 kubenswrapper[2576]:         , stdout: , stderr: , exit code -1
Sep 09 21:10:14.699695 ip-10-0-134-72 kubenswrapper[2576]:  > containerID="a5dc0ff68bdab72d5542db5135d45e269a24403793b0495263393b91a2d2517b" cmd=["/bin/sleep","15"]
Sep 09 21:10:14.699973 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:14.699723    2576 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-88vvz" message="" err=<
Sep 09 21:10:14.699973 ip-10-0-134-72 kubenswrapper[2576]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 21:10:14.699973 ip-10-0-134-72 kubenswrapper[2576]:         , stdout: , stderr: , exit code -1
Sep 09 21:10:14.699973 ip-10-0-134-72 kubenswrapper[2576]:  >
Sep 09 21:10:14.699973 ip-10-0-134-72 kubenswrapper[2576]: E0909 21:10:14.699740    2576 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 09 21:10:14.699973 ip-10-0-134-72 kubenswrapper[2576]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 21:10:14.699973 ip-10-0-134-72 kubenswrapper[2576]:         , stdout: , stderr: , exit code -1
Sep 09 21:10:14.699973 ip-10-0-134-72 kubenswrapper[2576]:  > pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-88vvz" podUID="e03347f5-3fb3-4a81-874a-488839b35ebf" containerName="main" containerID="cri-o://a5dc0ff68bdab72d5542db5135d45e269a24403793b0495263393b91a2d2517b"
Sep 09 21:10:14.699973 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:14.699789    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-88vvz" podUID="e03347f5-3fb3-4a81-874a-488839b35ebf" containerName="main" containerID="cri-o://a5dc0ff68bdab72d5542db5135d45e269a24403793b0495263393b91a2d2517b" gracePeriod=60
Sep 09 21:10:14.784070 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:14.784032    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:10:14.953554 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:14.953478    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-88vvz"
Sep 09 21:10:15.031230 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:15.031196    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e03347f5-3fb3-4a81-874a-488839b35ebf-model-cache\") pod \"e03347f5-3fb3-4a81-874a-488839b35ebf\" (UID: \"e03347f5-3fb3-4a81-874a-488839b35ebf\") "
Sep 09 21:10:15.031438 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:15.031239    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e03347f5-3fb3-4a81-874a-488839b35ebf-tls-certs\") pod \"e03347f5-3fb3-4a81-874a-488839b35ebf\" (UID: \"e03347f5-3fb3-4a81-874a-488839b35ebf\") "
Sep 09 21:10:15.031438 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:15.031289    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e03347f5-3fb3-4a81-874a-488839b35ebf-home\") pod \"e03347f5-3fb3-4a81-874a-488839b35ebf\" (UID: \"e03347f5-3fb3-4a81-874a-488839b35ebf\") "
Sep 09 21:10:15.031438 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:15.031306    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e03347f5-3fb3-4a81-874a-488839b35ebf-tmp-dir\") pod \"e03347f5-3fb3-4a81-874a-488839b35ebf\" (UID: \"e03347f5-3fb3-4a81-874a-488839b35ebf\") "
Sep 09 21:10:15.031438 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:15.031328    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e03347f5-3fb3-4a81-874a-488839b35ebf-dshm\") pod \"e03347f5-3fb3-4a81-874a-488839b35ebf\" (UID: \"e03347f5-3fb3-4a81-874a-488839b35ebf\") "
Sep 09 21:10:15.031438 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:15.031424    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cdzcs\" (UniqueName: \"kubernetes.io/projected/e03347f5-3fb3-4a81-874a-488839b35ebf-kube-api-access-cdzcs\") pod \"e03347f5-3fb3-4a81-874a-488839b35ebf\" (UID: \"e03347f5-3fb3-4a81-874a-488839b35ebf\") "
Sep 09 21:10:15.031723 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:15.031571    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e03347f5-3fb3-4a81-874a-488839b35ebf-model-cache" (OuterVolumeSpecName: "model-cache") pod "e03347f5-3fb3-4a81-874a-488839b35ebf" (UID: "e03347f5-3fb3-4a81-874a-488839b35ebf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:10:15.031723 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:15.031585    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e03347f5-3fb3-4a81-874a-488839b35ebf-home" (OuterVolumeSpecName: "home") pod "e03347f5-3fb3-4a81-874a-488839b35ebf" (UID: "e03347f5-3fb3-4a81-874a-488839b35ebf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:10:15.031723 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:15.031689    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e03347f5-3fb3-4a81-874a-488839b35ebf-model-cache\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:10:15.031723 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:15.031705    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e03347f5-3fb3-4a81-874a-488839b35ebf-home\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:10:15.031935 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:15.031911    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e03347f5-3fb3-4a81-874a-488839b35ebf-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e03347f5-3fb3-4a81-874a-488839b35ebf" (UID: "e03347f5-3fb3-4a81-874a-488839b35ebf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:10:15.034401 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:15.034314    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e03347f5-3fb3-4a81-874a-488839b35ebf-kube-api-access-cdzcs" (OuterVolumeSpecName: "kube-api-access-cdzcs") pod "e03347f5-3fb3-4a81-874a-488839b35ebf" (UID: "e03347f5-3fb3-4a81-874a-488839b35ebf"). InnerVolumeSpecName "kube-api-access-cdzcs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:10:15.034401 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:15.034353    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e03347f5-3fb3-4a81-874a-488839b35ebf-dshm" (OuterVolumeSpecName: "dshm") pod "e03347f5-3fb3-4a81-874a-488839b35ebf" (UID: "e03347f5-3fb3-4a81-874a-488839b35ebf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:10:15.034598 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:15.034434    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e03347f5-3fb3-4a81-874a-488839b35ebf-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e03347f5-3fb3-4a81-874a-488839b35ebf" (UID: "e03347f5-3fb3-4a81-874a-488839b35ebf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:10:15.132809 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:15.132764    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e03347f5-3fb3-4a81-874a-488839b35ebf-tls-certs\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:10:15.132809 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:15.132795    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e03347f5-3fb3-4a81-874a-488839b35ebf-tmp-dir\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:10:15.132809 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:15.132804    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e03347f5-3fb3-4a81-874a-488839b35ebf-dshm\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:10:15.132809 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:15.132813    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cdzcs\" (UniqueName: \"kubernetes.io/projected/e03347f5-3fb3-4a81-874a-488839b35ebf-kube-api-access-cdzcs\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:10:15.460140 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:15.460100    2576 generic.go:356] "Generic (PLEG): container finished" podID="e03347f5-3fb3-4a81-874a-488839b35ebf" containerID="a5dc0ff68bdab72d5542db5135d45e269a24403793b0495263393b91a2d2517b" exitCode=0
Sep 09 21:10:15.460293 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:15.460154    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-88vvz"
Sep 09 21:10:15.460293 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:15.460168    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-88vvz" event={"ID":"e03347f5-3fb3-4a81-874a-488839b35ebf","Type":"ContainerDied","Data":"a5dc0ff68bdab72d5542db5135d45e269a24403793b0495263393b91a2d2517b"}
Sep 09 21:10:15.460293 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:15.460220    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-88vvz" event={"ID":"e03347f5-3fb3-4a81-874a-488839b35ebf","Type":"ContainerDied","Data":"217323f3dd99340cf35e1409d150e8c2fcc80d353304e323a0c65ab41c3ecf47"}
Sep 09 21:10:15.460293 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:15.460250    2576 scope.go:117] "RemoveContainer" containerID="a5dc0ff68bdab72d5542db5135d45e269a24403793b0495263393b91a2d2517b"
Sep 09 21:10:15.469811 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:15.469790    2576 scope.go:117] "RemoveContainer" containerID="a5dc0ff68bdab72d5542db5135d45e269a24403793b0495263393b91a2d2517b"
Sep 09 21:10:15.470135 ip-10-0-134-72 kubenswrapper[2576]: E0909 21:10:15.470106    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5dc0ff68bdab72d5542db5135d45e269a24403793b0495263393b91a2d2517b\": container with ID starting with a5dc0ff68bdab72d5542db5135d45e269a24403793b0495263393b91a2d2517b not found: ID does not exist" containerID="a5dc0ff68bdab72d5542db5135d45e269a24403793b0495263393b91a2d2517b"
Sep 09 21:10:15.470253 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:15.470133    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5dc0ff68bdab72d5542db5135d45e269a24403793b0495263393b91a2d2517b"} err="failed to get container status \"a5dc0ff68bdab72d5542db5135d45e269a24403793b0495263393b91a2d2517b\": rpc error: code = NotFound desc = could not find container \"a5dc0ff68bdab72d5542db5135d45e269a24403793b0495263393b91a2d2517b\": container with ID starting with a5dc0ff68bdab72d5542db5135d45e269a24403793b0495263393b91a2d2517b not found: ID does not exist"
Sep 09 21:10:15.485882 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:15.485855    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-88vvz"]
Sep 09 21:10:15.488776 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:15.488753    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-88vvz"]
Sep 09 21:10:17.042791 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:17.042764    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-zmrmc"
Sep 09 21:10:17.150841 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:17.150805    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12af7609-714f-4b2c-a4b0-cbf4870d02d2-workload-certs\") pod \"12af7609-714f-4b2c-a4b0-cbf4870d02d2\" (UID: \"12af7609-714f-4b2c-a4b0-cbf4870d02d2\") "
Sep 09 21:10:17.150841 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:17.150844    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12af7609-714f-4b2c-a4b0-cbf4870d02d2-istio-podinfo\") pod \"12af7609-714f-4b2c-a4b0-cbf4870d02d2\" (UID: \"12af7609-714f-4b2c-a4b0-cbf4870d02d2\") "
Sep 09 21:10:17.151080 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:17.150874    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12af7609-714f-4b2c-a4b0-cbf4870d02d2-istio-token\") pod \"12af7609-714f-4b2c-a4b0-cbf4870d02d2\" (UID: \"12af7609-714f-4b2c-a4b0-cbf4870d02d2\") "
Sep 09 21:10:17.151080 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:17.150903    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ctngs\" (UniqueName: \"kubernetes.io/projected/12af7609-714f-4b2c-a4b0-cbf4870d02d2-kube-api-access-ctngs\") pod \"12af7609-714f-4b2c-a4b0-cbf4870d02d2\" (UID: \"12af7609-714f-4b2c-a4b0-cbf4870d02d2\") "
Sep 09 21:10:17.151080 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:17.150937    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12af7609-714f-4b2c-a4b0-cbf4870d02d2-credential-socket\") pod \"12af7609-714f-4b2c-a4b0-cbf4870d02d2\" (UID: \"12af7609-714f-4b2c-a4b0-cbf4870d02d2\") "
Sep 09 21:10:17.151080 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:17.150960    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12af7609-714f-4b2c-a4b0-cbf4870d02d2-istio-envoy\") pod \"12af7609-714f-4b2c-a4b0-cbf4870d02d2\" (UID: \"12af7609-714f-4b2c-a4b0-cbf4870d02d2\") "
Sep 09 21:10:17.151080 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:17.150986    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12af7609-714f-4b2c-a4b0-cbf4870d02d2-istiod-ca-cert\") pod \"12af7609-714f-4b2c-a4b0-cbf4870d02d2\" (UID: \"12af7609-714f-4b2c-a4b0-cbf4870d02d2\") "
Sep 09 21:10:17.151080 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:17.151029    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12af7609-714f-4b2c-a4b0-cbf4870d02d2-workload-socket\") pod \"12af7609-714f-4b2c-a4b0-cbf4870d02d2\" (UID: \"12af7609-714f-4b2c-a4b0-cbf4870d02d2\") "
Sep 09 21:10:17.151080 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:17.151072    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12af7609-714f-4b2c-a4b0-cbf4870d02d2-istio-data\") pod \"12af7609-714f-4b2c-a4b0-cbf4870d02d2\" (UID: \"12af7609-714f-4b2c-a4b0-cbf4870d02d2\") "
Sep 09 21:10:17.151470 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:17.151140    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12af7609-714f-4b2c-a4b0-cbf4870d02d2-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "12af7609-714f-4b2c-a4b0-cbf4870d02d2" (UID: "12af7609-714f-4b2c-a4b0-cbf4870d02d2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:10:17.151470 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:17.151323    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12af7609-714f-4b2c-a4b0-cbf4870d02d2-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "12af7609-714f-4b2c-a4b0-cbf4870d02d2" (UID: "12af7609-714f-4b2c-a4b0-cbf4870d02d2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:10:17.151470 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:17.151384    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12af7609-714f-4b2c-a4b0-cbf4870d02d2-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "12af7609-714f-4b2c-a4b0-cbf4870d02d2" (UID: "12af7609-714f-4b2c-a4b0-cbf4870d02d2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:10:17.151629 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:17.151545    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12af7609-714f-4b2c-a4b0-cbf4870d02d2-istio-data" (OuterVolumeSpecName: "istio-data") pod "12af7609-714f-4b2c-a4b0-cbf4870d02d2" (UID: "12af7609-714f-4b2c-a4b0-cbf4870d02d2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:10:17.151629 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:17.151502    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12af7609-714f-4b2c-a4b0-cbf4870d02d2-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "12af7609-714f-4b2c-a4b0-cbf4870d02d2" (UID: "12af7609-714f-4b2c-a4b0-cbf4870d02d2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:10:17.151806 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:17.151790    2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12af7609-714f-4b2c-a4b0-cbf4870d02d2-workload-certs\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:10:17.151891 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:17.151813    2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12af7609-714f-4b2c-a4b0-cbf4870d02d2-credential-socket\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:10:17.151891 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:17.151826    2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12af7609-714f-4b2c-a4b0-cbf4870d02d2-istiod-ca-cert\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:10:17.151891 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:17.151855    2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12af7609-714f-4b2c-a4b0-cbf4870d02d2-workload-socket\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:10:17.151891 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:17.151868    2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12af7609-714f-4b2c-a4b0-cbf4870d02d2-istio-data\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:10:17.153766 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:17.153710    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12af7609-714f-4b2c-a4b0-cbf4870d02d2-kube-api-access-ctngs" (OuterVolumeSpecName: "kube-api-access-ctngs") pod "12af7609-714f-4b2c-a4b0-cbf4870d02d2" (UID: "12af7609-714f-4b2c-a4b0-cbf4870d02d2"). InnerVolumeSpecName "kube-api-access-ctngs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:10:17.153987 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:17.153951    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12af7609-714f-4b2c-a4b0-cbf4870d02d2-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "12af7609-714f-4b2c-a4b0-cbf4870d02d2" (UID: "12af7609-714f-4b2c-a4b0-cbf4870d02d2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:10:17.154080 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:17.154037    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/12af7609-714f-4b2c-a4b0-cbf4870d02d2-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "12af7609-714f-4b2c-a4b0-cbf4870d02d2" (UID: "12af7609-714f-4b2c-a4b0-cbf4870d02d2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 21:10:17.154119 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:17.154070    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12af7609-714f-4b2c-a4b0-cbf4870d02d2-istio-token" (OuterVolumeSpecName: "istio-token") pod "12af7609-714f-4b2c-a4b0-cbf4870d02d2" (UID: "12af7609-714f-4b2c-a4b0-cbf4870d02d2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:10:17.253172 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:17.253140    2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12af7609-714f-4b2c-a4b0-cbf4870d02d2-istio-podinfo\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:10:17.253172 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:17.253169    2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12af7609-714f-4b2c-a4b0-cbf4870d02d2-istio-token\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:10:17.253172 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:17.253178    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctngs\" (UniqueName: \"kubernetes.io/projected/12af7609-714f-4b2c-a4b0-cbf4870d02d2-kube-api-access-ctngs\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:10:17.253425 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:17.253188    2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12af7609-714f-4b2c-a4b0-cbf4870d02d2-istio-envoy\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:10:17.410213 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:17.410137    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e03347f5-3fb3-4a81-874a-488839b35ebf" path="/var/lib/kubelet/pods/e03347f5-3fb3-4a81-874a-488839b35ebf/volumes"
Sep 09 21:10:17.473873 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:17.473834    2576 generic.go:356] "Generic (PLEG): container finished" podID="12af7609-714f-4b2c-a4b0-cbf4870d02d2" containerID="dabb13112bce54332b342edaf09915e10ce93f94a182cd36bcecdbeb877f7f4b" exitCode=0
Sep 09 21:10:17.474036 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:17.473890    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-zmrmc" event={"ID":"12af7609-714f-4b2c-a4b0-cbf4870d02d2","Type":"ContainerDied","Data":"dabb13112bce54332b342edaf09915e10ce93f94a182cd36bcecdbeb877f7f4b"}
Sep 09 21:10:17.474036 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:17.473929    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-zmrmc"
Sep 09 21:10:17.474036 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:17.473947    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-zmrmc" event={"ID":"12af7609-714f-4b2c-a4b0-cbf4870d02d2","Type":"ContainerDied","Data":"2e67bdab2056932d7a8af7d3c89165d146393ff29f1379ede76a37d5601a7e98"}
Sep 09 21:10:17.474036 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:17.473978    2576 scope.go:117] "RemoveContainer" containerID="dabb13112bce54332b342edaf09915e10ce93f94a182cd36bcecdbeb877f7f4b"
Sep 09 21:10:17.484855 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:17.484839    2576 scope.go:117] "RemoveContainer" containerID="dabb13112bce54332b342edaf09915e10ce93f94a182cd36bcecdbeb877f7f4b"
Sep 09 21:10:17.485128 ip-10-0-134-72 kubenswrapper[2576]: E0909 21:10:17.485108    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dabb13112bce54332b342edaf09915e10ce93f94a182cd36bcecdbeb877f7f4b\": container with ID starting with dabb13112bce54332b342edaf09915e10ce93f94a182cd36bcecdbeb877f7f4b not found: ID does not exist" containerID="dabb13112bce54332b342edaf09915e10ce93f94a182cd36bcecdbeb877f7f4b"
Sep 09 21:10:17.485207 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:17.485137    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dabb13112bce54332b342edaf09915e10ce93f94a182cd36bcecdbeb877f7f4b"} err="failed to get container status \"dabb13112bce54332b342edaf09915e10ce93f94a182cd36bcecdbeb877f7f4b\": rpc error: code = NotFound desc = could not find container \"dabb13112bce54332b342edaf09915e10ce93f94a182cd36bcecdbeb877f7f4b\": container with ID starting with dabb13112bce54332b342edaf09915e10ce93f94a182cd36bcecdbeb877f7f4b not found: ID does not exist"
Sep 09 21:10:17.495147 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:17.495122    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-zmrmc"]
Sep 09 21:10:17.503094 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:17.503071    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-zmrmc"]
Sep 09 21:10:18.836918 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:18.836879    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:10:18.899472 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:18.899444    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:10:19.409828 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:19.409792    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="12af7609-714f-4b2c-a4b0-cbf4870d02d2" path="/var/lib/kubelet/pods/12af7609-714f-4b2c-a4b0-cbf4870d02d2/volumes"
Sep 09 21:10:20.395812 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:20.395778    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:10:20.506655 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:20.506620    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:10:22.479405 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:22.479287    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="db7ecaa9-1f5f-4957-9145-e02b5c9c1b43" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/health\": dial tcp 10.132.0.44:8000: connect: connection refused"
Sep 09 21:10:22.757063 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:22.756981    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:10:26.707323 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:26.707280    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:10:28.933274 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:28.933183    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:10:32.479536 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:32.479492    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="db7ecaa9-1f5f-4957-9145-e02b5c9c1b43" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/health\": dial tcp 10.132.0.44:8000: connect: connection refused"
Sep 09 21:10:42.479725 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:42.479686    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="db7ecaa9-1f5f-4957-9145-e02b5c9c1b43" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/health\": dial tcp 10.132.0.44:8000: connect: connection refused"
Sep 09 21:10:44.479167 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:44.479132    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:10:44.578803 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:44.578767    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:10:45.985664 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:45.985633    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbffr45x"]
Sep 09 21:10:45.986023 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:45.985963    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e03347f5-3fb3-4a81-874a-488839b35ebf" containerName="main"
Sep 09 21:10:45.986023 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:45.985973    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="e03347f5-3fb3-4a81-874a-488839b35ebf" containerName="main"
Sep 09 21:10:45.986023 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:45.985984    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12af7609-714f-4b2c-a4b0-cbf4870d02d2" containerName="istio-proxy"
Sep 09 21:10:45.986023 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:45.985989    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="12af7609-714f-4b2c-a4b0-cbf4870d02d2" containerName="istio-proxy"
Sep 09 21:10:45.986152 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:45.986042    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="e03347f5-3fb3-4a81-874a-488839b35ebf" containerName="main"
Sep 09 21:10:45.986152 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:45.986050    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="12af7609-714f-4b2c-a4b0-cbf4870d02d2" containerName="istio-proxy"
Sep 09 21:10:45.990302 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:45.990278    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbffr45x"
Sep 09 21:10:45.992861 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:45.992836    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"openshift-service-ca.crt\""
Sep 09 21:10:45.993832 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:45.993813    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"kube-root-ca.crt\""
Sep 09 21:10:45.993952 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:45.993872    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs\""
Sep 09 21:10:45.994766 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:45.994746    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"default-dockercfg-dv4cb\""
Sep 09 21:10:46.003683 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:46.003656    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbffr45x"]
Sep 09 21:10:46.025639 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:46.025611    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:10:46.099394 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:46.099334    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01508b68-2bf0-4b68-9b0a-d38744b1a00e-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbffr45x\" (UID: \"01508b68-2bf0-4b68-9b0a-d38744b1a00e\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbffr45x"
Sep 09 21:10:46.099578 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:46.099402    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01508b68-2bf0-4b68-9b0a-d38744b1a00e-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbffr45x\" (UID: \"01508b68-2bf0-4b68-9b0a-d38744b1a00e\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbffr45x"
Sep 09 21:10:46.099578 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:46.099435    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01508b68-2bf0-4b68-9b0a-d38744b1a00e-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbffr45x\" (UID: \"01508b68-2bf0-4b68-9b0a-d38744b1a00e\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbffr45x"
Sep 09 21:10:46.099578 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:46.099488    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01508b68-2bf0-4b68-9b0a-d38744b1a00e-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbffr45x\" (UID: \"01508b68-2bf0-4b68-9b0a-d38744b1a00e\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbffr45x"
Sep 09 21:10:46.099578 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:46.099518    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01508b68-2bf0-4b68-9b0a-d38744b1a00e-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbffr45x\" (UID: \"01508b68-2bf0-4b68-9b0a-d38744b1a00e\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbffr45x"
Sep 09 21:10:46.099732 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:46.099594    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nlc9\" (UniqueName: \"kubernetes.io/projected/01508b68-2bf0-4b68-9b0a-d38744b1a00e-kube-api-access-4nlc9\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbffr45x\" (UID: \"01508b68-2bf0-4b68-9b0a-d38744b1a00e\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbffr45x"
Sep 09 21:10:46.200156 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:46.200124    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01508b68-2bf0-4b68-9b0a-d38744b1a00e-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbffr45x\" (UID: \"01508b68-2bf0-4b68-9b0a-d38744b1a00e\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbffr45x"
Sep 09 21:10:46.200156 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:46.200158    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01508b68-2bf0-4b68-9b0a-d38744b1a00e-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbffr45x\" (UID: \"01508b68-2bf0-4b68-9b0a-d38744b1a00e\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbffr45x"
Sep 09 21:10:46.200442 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:46.200178    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01508b68-2bf0-4b68-9b0a-d38744b1a00e-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbffr45x\" (UID: \"01508b68-2bf0-4b68-9b0a-d38744b1a00e\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbffr45x"
Sep 09 21:10:46.200442 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:46.200204    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01508b68-2bf0-4b68-9b0a-d38744b1a00e-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbffr45x\" (UID: \"01508b68-2bf0-4b68-9b0a-d38744b1a00e\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbffr45x"
Sep 09 21:10:46.200442 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:46.200229    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01508b68-2bf0-4b68-9b0a-d38744b1a00e-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbffr45x\" (UID: \"01508b68-2bf0-4b68-9b0a-d38744b1a00e\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbffr45x"
Sep 09 21:10:46.200616 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:46.200477    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nlc9\" (UniqueName: \"kubernetes.io/projected/01508b68-2bf0-4b68-9b0a-d38744b1a00e-kube-api-access-4nlc9\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbffr45x\" (UID: \"01508b68-2bf0-4b68-9b0a-d38744b1a00e\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbffr45x"
Sep 09 21:10:46.200616 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:46.200584    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01508b68-2bf0-4b68-9b0a-d38744b1a00e-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbffr45x\" (UID: \"01508b68-2bf0-4b68-9b0a-d38744b1a00e\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbffr45x"
Sep 09 21:10:46.200693 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:46.200656    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01508b68-2bf0-4b68-9b0a-d38744b1a00e-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbffr45x\" (UID: \"01508b68-2bf0-4b68-9b0a-d38744b1a00e\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbffr45x"
Sep 09 21:10:46.200870 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:46.200844    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01508b68-2bf0-4b68-9b0a-d38744b1a00e-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbffr45x\" (UID: \"01508b68-2bf0-4b68-9b0a-d38744b1a00e\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbffr45x"
Sep 09 21:10:46.202859 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:46.202830    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01508b68-2bf0-4b68-9b0a-d38744b1a00e-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbffr45x\" (UID: \"01508b68-2bf0-4b68-9b0a-d38744b1a00e\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbffr45x"
Sep 09 21:10:46.202974 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:46.202863    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01508b68-2bf0-4b68-9b0a-d38744b1a00e-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbffr45x\" (UID: \"01508b68-2bf0-4b68-9b0a-d38744b1a00e\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbffr45x"
Sep 09 21:10:46.209063 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:46.209039    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nlc9\" (UniqueName: \"kubernetes.io/projected/01508b68-2bf0-4b68-9b0a-d38744b1a00e-kube-api-access-4nlc9\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbffr45x\" (UID: \"01508b68-2bf0-4b68-9b0a-d38744b1a00e\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbffr45x"
Sep 09 21:10:46.301664 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:46.301570    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbffr45x"
Sep 09 21:10:46.586010 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:46.585984    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbffr45x"]
Sep 09 21:10:46.588108 ip-10-0-134-72 kubenswrapper[2576]: W0909 21:10:46.588067    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01508b68_2bf0_4b68_9b0a_d38744b1a00e.slice/crio-236ffc26eec645f7831a8debf45a557165ff74987cc656cbb438d38a66a8ca34 WatchSource:0}: Error finding container 236ffc26eec645f7831a8debf45a557165ff74987cc656cbb438d38a66a8ca34: Status 404 returned error can't find the container with id 236ffc26eec645f7831a8debf45a557165ff74987cc656cbb438d38a66a8ca34
Sep 09 21:10:46.644620 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:46.644579    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbffr45x" event={"ID":"01508b68-2bf0-4b68-9b0a-d38744b1a00e","Type":"ContainerStarted","Data":"236ffc26eec645f7831a8debf45a557165ff74987cc656cbb438d38a66a8ca34"}
Sep 09 21:10:47.651337 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:47.651295    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbffr45x" event={"ID":"01508b68-2bf0-4b68-9b0a-d38744b1a00e","Type":"ContainerStarted","Data":"460739b8c2973b08001cf51895449f02390a706ff50e918f04c2a1eb0d8e52e6"}
Sep 09 21:10:47.680642 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:47.680590    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbffr45x" podStartSLOduration=2.680573711 podStartE2EDuration="2.680573711s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:10:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:10:47.679107528 +0000 UTC m=+1090.792793387" watchObservedRunningTime="2026-09-09 21:10:47.680573711 +0000 UTC m=+1090.794259570"
Sep 09 21:10:48.060253 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:48.060166    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:10:48.175017 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:48.174974    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:10:48.302624 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:48.302589    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbffr45x"
Sep 09 21:10:49.652759 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:49.652731    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbffr45x"]
Sep 09 21:10:49.666670 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:49.666642    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:10:49.733236 ip-10-0-134-72 kubenswrapper[2576]: E0909 21:10:49.733203    2576 secret.go:189] Couldn't get secret e2e-test-gateway-section-name-propagation-94799d44/gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs: secret "gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs" not found
Sep 09 21:10:49.733422 ip-10-0-134-72 kubenswrapper[2576]: E0909 21:10:49.733279    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/01508b68-2bf0-4b68-9b0a-d38744b1a00e-tls-certs podName:01508b68-2bf0-4b68-9b0a-d38744b1a00e nodeName:}" failed. No retries permitted until 2026-09-09 21:10:50.233257357 +0000 UTC m=+1093.346943194 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/01508b68-2bf0-4b68-9b0a-d38744b1a00e-tls-certs") pod "gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbffr45x" (UID: "01508b68-2bf0-4b68-9b0a-d38744b1a00e") : secret "gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs" not found
Sep 09 21:10:50.237806 ip-10-0-134-72 kubenswrapper[2576]: E0909 21:10:50.237771    2576 secret.go:189] Couldn't get secret e2e-test-gateway-section-name-propagation-94799d44/gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs: secret "gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs" not found
Sep 09 21:10:50.238002 ip-10-0-134-72 kubenswrapper[2576]: E0909 21:10:50.237869    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/01508b68-2bf0-4b68-9b0a-d38744b1a00e-tls-certs podName:01508b68-2bf0-4b68-9b0a-d38744b1a00e nodeName:}" failed. No retries permitted until 2026-09-09 21:10:51.23784787 +0000 UTC m=+1094.351533720 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/01508b68-2bf0-4b68-9b0a-d38744b1a00e-tls-certs") pod "gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbffr45x" (UID: "01508b68-2bf0-4b68-9b0a-d38744b1a00e") : secret "gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs" not found
Sep 09 21:10:50.674324 ip-10-0-134-72 kubenswrapper[2576]: E0909 21:10:50.674285    2576 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 09 21:10:50.674324 ip-10-0-134-72 kubenswrapper[2576]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 21:10:50.674324 ip-10-0-134-72 kubenswrapper[2576]:         , stdout: , stderr: , exit code -1
Sep 09 21:10:50.674324 ip-10-0-134-72 kubenswrapper[2576]:  > containerID="460739b8c2973b08001cf51895449f02390a706ff50e918f04c2a1eb0d8e52e6" cmd=["/bin/sleep","15"]
Sep 09 21:10:50.674807 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:50.674334    2576 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbffr45x" message="" err=<
Sep 09 21:10:50.674807 ip-10-0-134-72 kubenswrapper[2576]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 21:10:50.674807 ip-10-0-134-72 kubenswrapper[2576]:         , stdout: , stderr: , exit code -1
Sep 09 21:10:50.674807 ip-10-0-134-72 kubenswrapper[2576]:  >
Sep 09 21:10:50.674807 ip-10-0-134-72 kubenswrapper[2576]: E0909 21:10:50.674347    2576 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 09 21:10:50.674807 ip-10-0-134-72 kubenswrapper[2576]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 21:10:50.674807 ip-10-0-134-72 kubenswrapper[2576]:         , stdout: , stderr: , exit code -1
Sep 09 21:10:50.674807 ip-10-0-134-72 kubenswrapper[2576]:  > pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbffr45x" podUID="01508b68-2bf0-4b68-9b0a-d38744b1a00e" containerName="main" containerID="cri-o://460739b8c2973b08001cf51895449f02390a706ff50e918f04c2a1eb0d8e52e6"
Sep 09 21:10:50.674807 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:50.674402    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbffr45x" podUID="01508b68-2bf0-4b68-9b0a-d38744b1a00e" containerName="main" containerID="cri-o://460739b8c2973b08001cf51895449f02390a706ff50e918f04c2a1eb0d8e52e6" gracePeriod=60
Sep 09 21:10:50.915419 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:50.915393    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbffr45x"
Sep 09 21:10:51.046037 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:51.045943    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01508b68-2bf0-4b68-9b0a-d38744b1a00e-tls-certs\") pod \"01508b68-2bf0-4b68-9b0a-d38744b1a00e\" (UID: \"01508b68-2bf0-4b68-9b0a-d38744b1a00e\") "
Sep 09 21:10:51.046037 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:51.046036    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01508b68-2bf0-4b68-9b0a-d38744b1a00e-dshm\") pod \"01508b68-2bf0-4b68-9b0a-d38744b1a00e\" (UID: \"01508b68-2bf0-4b68-9b0a-d38744b1a00e\") "
Sep 09 21:10:51.046236 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:51.046089    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01508b68-2bf0-4b68-9b0a-d38744b1a00e-home\") pod \"01508b68-2bf0-4b68-9b0a-d38744b1a00e\" (UID: \"01508b68-2bf0-4b68-9b0a-d38744b1a00e\") "
Sep 09 21:10:51.046236 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:51.046139    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01508b68-2bf0-4b68-9b0a-d38744b1a00e-model-cache\") pod \"01508b68-2bf0-4b68-9b0a-d38744b1a00e\" (UID: \"01508b68-2bf0-4b68-9b0a-d38744b1a00e\") "
Sep 09 21:10:51.046236 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:51.046174    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01508b68-2bf0-4b68-9b0a-d38744b1a00e-tmp-dir\") pod \"01508b68-2bf0-4b68-9b0a-d38744b1a00e\" (UID: \"01508b68-2bf0-4b68-9b0a-d38744b1a00e\") "
Sep 09 21:10:51.046236 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:51.046213    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4nlc9\" (UniqueName: \"kubernetes.io/projected/01508b68-2bf0-4b68-9b0a-d38744b1a00e-kube-api-access-4nlc9\") pod \"01508b68-2bf0-4b68-9b0a-d38744b1a00e\" (UID: \"01508b68-2bf0-4b68-9b0a-d38744b1a00e\") "
Sep 09 21:10:51.046478 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:51.046431    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01508b68-2bf0-4b68-9b0a-d38744b1a00e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "01508b68-2bf0-4b68-9b0a-d38744b1a00e" (UID: "01508b68-2bf0-4b68-9b0a-d38744b1a00e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:10:51.046478 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:51.046439    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01508b68-2bf0-4b68-9b0a-d38744b1a00e-home" (OuterVolumeSpecName: "home") pod "01508b68-2bf0-4b68-9b0a-d38744b1a00e" (UID: "01508b68-2bf0-4b68-9b0a-d38744b1a00e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:10:51.046585 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:51.046458    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01508b68-2bf0-4b68-9b0a-d38744b1a00e-model-cache" (OuterVolumeSpecName: "model-cache") pod "01508b68-2bf0-4b68-9b0a-d38744b1a00e" (UID: "01508b68-2bf0-4b68-9b0a-d38744b1a00e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:10:51.046585 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:51.046545    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01508b68-2bf0-4b68-9b0a-d38744b1a00e-home\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:10:51.046585 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:51.046562    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01508b68-2bf0-4b68-9b0a-d38744b1a00e-model-cache\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:10:51.046585 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:51.046577    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01508b68-2bf0-4b68-9b0a-d38744b1a00e-tmp-dir\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:10:51.048530 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:51.048500    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01508b68-2bf0-4b68-9b0a-d38744b1a00e-dshm" (OuterVolumeSpecName: "dshm") pod "01508b68-2bf0-4b68-9b0a-d38744b1a00e" (UID: "01508b68-2bf0-4b68-9b0a-d38744b1a00e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:10:51.048670 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:51.048585    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01508b68-2bf0-4b68-9b0a-d38744b1a00e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "01508b68-2bf0-4b68-9b0a-d38744b1a00e" (UID: "01508b68-2bf0-4b68-9b0a-d38744b1a00e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:10:51.048670 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:51.048613    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01508b68-2bf0-4b68-9b0a-d38744b1a00e-kube-api-access-4nlc9" (OuterVolumeSpecName: "kube-api-access-4nlc9") pod "01508b68-2bf0-4b68-9b0a-d38744b1a00e" (UID: "01508b68-2bf0-4b68-9b0a-d38744b1a00e"). InnerVolumeSpecName "kube-api-access-4nlc9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:10:51.147228 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:51.147192    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4nlc9\" (UniqueName: \"kubernetes.io/projected/01508b68-2bf0-4b68-9b0a-d38744b1a00e-kube-api-access-4nlc9\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:10:51.147228 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:51.147223    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01508b68-2bf0-4b68-9b0a-d38744b1a00e-tls-certs\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:10:51.147228 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:51.147237    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01508b68-2bf0-4b68-9b0a-d38744b1a00e-dshm\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:10:51.671688 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:51.671594    2576 generic.go:356] "Generic (PLEG): container finished" podID="01508b68-2bf0-4b68-9b0a-d38744b1a00e" containerID="460739b8c2973b08001cf51895449f02390a706ff50e918f04c2a1eb0d8e52e6" exitCode=0
Sep 09 21:10:51.671688 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:51.671667    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbffr45x"
Sep 09 21:10:51.671875 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:51.671687    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbffr45x" event={"ID":"01508b68-2bf0-4b68-9b0a-d38744b1a00e","Type":"ContainerDied","Data":"460739b8c2973b08001cf51895449f02390a706ff50e918f04c2a1eb0d8e52e6"}
Sep 09 21:10:51.671875 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:51.671751    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbffr45x" event={"ID":"01508b68-2bf0-4b68-9b0a-d38744b1a00e","Type":"ContainerDied","Data":"236ffc26eec645f7831a8debf45a557165ff74987cc656cbb438d38a66a8ca34"}
Sep 09 21:10:51.671875 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:51.671773    2576 scope.go:117] "RemoveContainer" containerID="460739b8c2973b08001cf51895449f02390a706ff50e918f04c2a1eb0d8e52e6"
Sep 09 21:10:51.681274 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:51.681112    2576 scope.go:117] "RemoveContainer" containerID="460739b8c2973b08001cf51895449f02390a706ff50e918f04c2a1eb0d8e52e6"
Sep 09 21:10:51.681538 ip-10-0-134-72 kubenswrapper[2576]: E0909 21:10:51.681350    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"460739b8c2973b08001cf51895449f02390a706ff50e918f04c2a1eb0d8e52e6\": container with ID starting with 460739b8c2973b08001cf51895449f02390a706ff50e918f04c2a1eb0d8e52e6 not found: ID does not exist" containerID="460739b8c2973b08001cf51895449f02390a706ff50e918f04c2a1eb0d8e52e6"
Sep 09 21:10:51.681538 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:51.681395    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"460739b8c2973b08001cf51895449f02390a706ff50e918f04c2a1eb0d8e52e6"} err="failed to get container status \"460739b8c2973b08001cf51895449f02390a706ff50e918f04c2a1eb0d8e52e6\": rpc error: code = NotFound desc = could not find container \"460739b8c2973b08001cf51895449f02390a706ff50e918f04c2a1eb0d8e52e6\": container with ID starting with 460739b8c2973b08001cf51895449f02390a706ff50e918f04c2a1eb0d8e52e6 not found: ID does not exist"
Sep 09 21:10:51.696383 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:51.693527    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbffr45x"]
Sep 09 21:10:51.700169 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:51.700127    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbffr45x"]
Sep 09 21:10:52.479598 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:52.479552    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="db7ecaa9-1f5f-4957-9145-e02b5c9c1b43" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/health\": dial tcp 10.132.0.44:8000: connect: connection refused"
Sep 09 21:10:53.196472 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:53.196432    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:10:53.409744 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:53.409712    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="01508b68-2bf0-4b68-9b0a-d38744b1a00e" path="/var/lib/kubelet/pods/01508b68-2bf0-4b68-9b0a-d38744b1a00e/volumes"
Sep 09 21:10:59.889487 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:10:59.889439    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:11:00.228951 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:11:00.228848    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:11:01.837855 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:11:01.837822    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:11:02.479577 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:11:02.479530    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="db7ecaa9-1f5f-4957-9145-e02b5c9c1b43" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/health\": dial tcp 10.132.0.44:8000: connect: connection refused"
Sep 09 21:11:07.317419 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:11:07.317383    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5cdbbd5bdc-j9j6w"]
Sep 09 21:11:07.318022 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:11:07.317999    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01508b68-2bf0-4b68-9b0a-d38744b1a00e" containerName="main"
Sep 09 21:11:07.318022 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:11:07.318024    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="01508b68-2bf0-4b68-9b0a-d38744b1a00e" containerName="main"
Sep 09 21:11:07.318192 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:11:07.318118    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="01508b68-2bf0-4b68-9b0a-d38744b1a00e" containerName="main"
Sep 09 21:11:07.321779 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:11:07.321759    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5cdbbd5bdc-j9j6w"
Sep 09 21:11:07.324791 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:11:07.324766    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"kube-root-ca.crt\""
Sep 09 21:11:07.324923 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:11:07.324812    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"seaweedfs-s3-creds\""
Sep 09 21:11:07.324923 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:11:07.324769    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"auth-enabled-test-kserve-self-signed-certs\""
Sep 09 21:11:07.324923 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:11:07.324818    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"default-dockercfg-thq9k\""
Sep 09 21:11:07.324923 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:11:07.324828    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"openshift-service-ca.crt\""
Sep 09 21:11:07.325198 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:11:07.325182    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"odh-kserve-custom-ca-bundle\""
Sep 09 21:11:07.337526 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:11:07.337504    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5cdbbd5bdc-j9j6w"]
Sep 09 21:11:07.347297 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:11:07.347270    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:11:07.497127 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:11:07.497078    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9baaa876-0d93-4580-a0f9-e6cbe1175f5f-dshm\") pod \"auth-enabled-test-kserve-5cdbbd5bdc-j9j6w\" (UID: \"9baaa876-0d93-4580-a0f9-e6cbe1175f5f\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5cdbbd5bdc-j9j6w"
Sep 09 21:11:07.497127 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:11:07.497127    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9baaa876-0d93-4580-a0f9-e6cbe1175f5f-model-cache\") pod \"auth-enabled-test-kserve-5cdbbd5bdc-j9j6w\" (UID: \"9baaa876-0d93-4580-a0f9-e6cbe1175f5f\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5cdbbd5bdc-j9j6w"
Sep 09 21:11:07.497404 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:11:07.497155    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9baaa876-0d93-4580-a0f9-e6cbe1175f5f-tls-certs\") pod \"auth-enabled-test-kserve-5cdbbd5bdc-j9j6w\" (UID: \"9baaa876-0d93-4580-a0f9-e6cbe1175f5f\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5cdbbd5bdc-j9j6w"
Sep 09 21:11:07.497404 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:11:07.497207    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcjhk\" (UniqueName: \"kubernetes.io/projected/9baaa876-0d93-4580-a0f9-e6cbe1175f5f-kube-api-access-fcjhk\") pod \"auth-enabled-test-kserve-5cdbbd5bdc-j9j6w\" (UID: \"9baaa876-0d93-4580-a0f9-e6cbe1175f5f\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5cdbbd5bdc-j9j6w"
Sep 09 21:11:07.497404 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:11:07.497265    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9baaa876-0d93-4580-a0f9-e6cbe1175f5f-tmp-dir\") pod \"auth-enabled-test-kserve-5cdbbd5bdc-j9j6w\" (UID: \"9baaa876-0d93-4580-a0f9-e6cbe1175f5f\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5cdbbd5bdc-j9j6w"
Sep 09 21:11:07.497404 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:11:07.497288    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9baaa876-0d93-4580-a0f9-e6cbe1175f5f-home\") pod \"auth-enabled-test-kserve-5cdbbd5bdc-j9j6w\" (UID: \"9baaa876-0d93-4580-a0f9-e6cbe1175f5f\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5cdbbd5bdc-j9j6w"
Sep 09 21:11:07.497404 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:11:07.497355    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9baaa876-0d93-4580-a0f9-e6cbe1175f5f-cabundle-cert\") pod \"auth-enabled-test-kserve-5cdbbd5bdc-j9j6w\" (UID: \"9baaa876-0d93-4580-a0f9-e6cbe1175f5f\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5cdbbd5bdc-j9j6w"
Sep 09 21:11:07.497404 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:11:07.497398    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9baaa876-0d93-4580-a0f9-e6cbe1175f5f-kserve-provision-location\") pod \"auth-enabled-test-kserve-5cdbbd5bdc-j9j6w\" (UID: \"9baaa876-0d93-4580-a0f9-e6cbe1175f5f\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5cdbbd5bdc-j9j6w"
Sep 09 21:11:07.598223 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:11:07.598187    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9baaa876-0d93-4580-a0f9-e6cbe1175f5f-tmp-dir\") pod \"auth-enabled-test-kserve-5cdbbd5bdc-j9j6w\" (UID: \"9baaa876-0d93-4580-a0f9-e6cbe1175f5f\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5cdbbd5bdc-j9j6w"
Sep 09 21:11:07.598418 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:11:07.598232    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9baaa876-0d93-4580-a0f9-e6cbe1175f5f-home\") pod \"auth-enabled-test-kserve-5cdbbd5bdc-j9j6w\" (UID: \"9baaa876-0d93-4580-a0f9-e6cbe1175f5f\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5cdbbd5bdc-j9j6w"
Sep 09 21:11:07.598418 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:11:07.598289    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9baaa876-0d93-4580-a0f9-e6cbe1175f5f-cabundle-cert\") pod \"auth-enabled-test-kserve-5cdbbd5bdc-j9j6w\" (UID: \"9baaa876-0d93-4580-a0f9-e6cbe1175f5f\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5cdbbd5bdc-j9j6w"
Sep 09 21:11:07.598418 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:11:07.598316    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9baaa876-0d93-4580-a0f9-e6cbe1175f5f-kserve-provision-location\") pod \"auth-enabled-test-kserve-5cdbbd5bdc-j9j6w\" (UID: \"9baaa876-0d93-4580-a0f9-e6cbe1175f5f\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5cdbbd5bdc-j9j6w"
Sep 09 21:11:07.598602 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:11:07.598452    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9baaa876-0d93-4580-a0f9-e6cbe1175f5f-dshm\") pod \"auth-enabled-test-kserve-5cdbbd5bdc-j9j6w\" (UID: \"9baaa876-0d93-4580-a0f9-e6cbe1175f5f\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5cdbbd5bdc-j9j6w"
Sep 09 21:11:07.598602 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:11:07.598491    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9baaa876-0d93-4580-a0f9-e6cbe1175f5f-model-cache\") pod \"auth-enabled-test-kserve-5cdbbd5bdc-j9j6w\" (UID: \"9baaa876-0d93-4580-a0f9-e6cbe1175f5f\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5cdbbd5bdc-j9j6w"
Sep 09 21:11:07.598602 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:11:07.598537    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9baaa876-0d93-4580-a0f9-e6cbe1175f5f-tls-certs\") pod \"auth-enabled-test-kserve-5cdbbd5bdc-j9j6w\" (UID: \"9baaa876-0d93-4580-a0f9-e6cbe1175f5f\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5cdbbd5bdc-j9j6w"
Sep 09 21:11:07.598754 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:11:07.598608    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9baaa876-0d93-4580-a0f9-e6cbe1175f5f-tmp-dir\") pod \"auth-enabled-test-kserve-5cdbbd5bdc-j9j6w\" (UID: \"9baaa876-0d93-4580-a0f9-e6cbe1175f5f\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5cdbbd5bdc-j9j6w"
Sep 09 21:11:07.598754 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:11:07.598637    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcjhk\" (UniqueName: \"kubernetes.io/projected/9baaa876-0d93-4580-a0f9-e6cbe1175f5f-kube-api-access-fcjhk\") pod \"auth-enabled-test-kserve-5cdbbd5bdc-j9j6w\" (UID: \"9baaa876-0d93-4580-a0f9-e6cbe1175f5f\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5cdbbd5bdc-j9j6w"
Sep 09 21:11:07.598754 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:11:07.598716    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9baaa876-0d93-4580-a0f9-e6cbe1175f5f-home\") pod \"auth-enabled-test-kserve-5cdbbd5bdc-j9j6w\" (UID: \"9baaa876-0d93-4580-a0f9-e6cbe1175f5f\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5cdbbd5bdc-j9j6w"
Sep 09 21:11:07.598754 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:11:07.598748    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9baaa876-0d93-4580-a0f9-e6cbe1175f5f-model-cache\") pod \"auth-enabled-test-kserve-5cdbbd5bdc-j9j6w\" (UID: \"9baaa876-0d93-4580-a0f9-e6cbe1175f5f\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5cdbbd5bdc-j9j6w"
Sep 09 21:11:07.599143 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:11:07.599118    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9baaa876-0d93-4580-a0f9-e6cbe1175f5f-cabundle-cert\") pod \"auth-enabled-test-kserve-5cdbbd5bdc-j9j6w\" (UID: \"9baaa876-0d93-4580-a0f9-e6cbe1175f5f\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5cdbbd5bdc-j9j6w"
Sep 09 21:11:07.599143 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:11:07.599132    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9baaa876-0d93-4580-a0f9-e6cbe1175f5f-kserve-provision-location\") pod \"auth-enabled-test-kserve-5cdbbd5bdc-j9j6w\" (UID: \"9baaa876-0d93-4580-a0f9-e6cbe1175f5f\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5cdbbd5bdc-j9j6w"
Sep 09 21:11:07.600848 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:11:07.600824    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9baaa876-0d93-4580-a0f9-e6cbe1175f5f-dshm\") pod \"auth-enabled-test-kserve-5cdbbd5bdc-j9j6w\" (UID: \"9baaa876-0d93-4580-a0f9-e6cbe1175f5f\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5cdbbd5bdc-j9j6w"
Sep 09 21:11:07.601001 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:11:07.600986    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9baaa876-0d93-4580-a0f9-e6cbe1175f5f-tls-certs\") pod \"auth-enabled-test-kserve-5cdbbd5bdc-j9j6w\" (UID: \"9baaa876-0d93-4580-a0f9-e6cbe1175f5f\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5cdbbd5bdc-j9j6w"
Sep 09 21:11:07.608402 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:11:07.608336    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:11:07.610343 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:11:07.610320    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcjhk\" (UniqueName: \"kubernetes.io/projected/9baaa876-0d93-4580-a0f9-e6cbe1175f5f-kube-api-access-fcjhk\") pod \"auth-enabled-test-kserve-5cdbbd5bdc-j9j6w\" (UID: \"9baaa876-0d93-4580-a0f9-e6cbe1175f5f\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5cdbbd5bdc-j9j6w"
Sep 09 21:11:07.632598 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:11:07.632570    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5cdbbd5bdc-j9j6w"
Sep 09 21:11:07.919733 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:11:07.919708    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5cdbbd5bdc-j9j6w"]
Sep 09 21:11:07.919954 ip-10-0-134-72 kubenswrapper[2576]: W0909 21:11:07.919928    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9baaa876_0d93_4580_a0f9_e6cbe1175f5f.slice/crio-546898583e61e66dbad101a5335368fafdef839c7a100db8504e194e2435cceb WatchSource:0}: Error finding container 546898583e61e66dbad101a5335368fafdef839c7a100db8504e194e2435cceb: Status 404 returned error can't find the container with id 546898583e61e66dbad101a5335368fafdef839c7a100db8504e194e2435cceb
Sep 09 21:11:08.767584 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:11:08.767544    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5cdbbd5bdc-j9j6w" event={"ID":"9baaa876-0d93-4580-a0f9-e6cbe1175f5f","Type":"ContainerStarted","Data":"dbeeed1b72b0a615e7dc8b9b45762293b4399969b46c186ca48514775b20601e"}
Sep 09 21:11:08.767977 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:11:08.767616    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5cdbbd5bdc-j9j6w" event={"ID":"9baaa876-0d93-4580-a0f9-e6cbe1175f5f","Type":"ContainerStarted","Data":"546898583e61e66dbad101a5335368fafdef839c7a100db8504e194e2435cceb"}
Sep 09 21:11:10.780086 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:11:10.780052    2576 generic.go:356] "Generic (PLEG): container finished" podID="9baaa876-0d93-4580-a0f9-e6cbe1175f5f" containerID="dbeeed1b72b0a615e7dc8b9b45762293b4399969b46c186ca48514775b20601e" exitCode=0
Sep 09 21:11:10.780086 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:11:10.780092    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5cdbbd5bdc-j9j6w" event={"ID":"9baaa876-0d93-4580-a0f9-e6cbe1175f5f","Type":"ContainerDied","Data":"dbeeed1b72b0a615e7dc8b9b45762293b4399969b46c186ca48514775b20601e"}
Sep 09 21:11:11.789400 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:11:11.789342    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5cdbbd5bdc-j9j6w" event={"ID":"9baaa876-0d93-4580-a0f9-e6cbe1175f5f","Type":"ContainerStarted","Data":"6980653714f46dd3f91f2bddfa789838443ba8031afb9449bb457158fc4c3e0a"}
Sep 09 21:11:11.809389 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:11:11.809335    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:11:11.820433 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:11:11.820388    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5cdbbd5bdc-j9j6w" podStartSLOduration=2.695336413 podStartE2EDuration="4.82035624s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.125019827s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:11:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:11:11.819914488 +0000 UTC m=+1114.933600349" watchObservedRunningTime="2026-09-09 21:11:11.82035624 +0000 UTC m=+1114.934042098"
Sep 09 21:11:12.489722 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:11:12.489688    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 09 21:11:12.497108 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:11:12.497081    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 09 21:11:12.633671 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:11:12.633640    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5cdbbd5bdc-j9j6w"
Sep 09 21:11:14.985272 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:11:14.985233    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:11:15.855055 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:11:15.855020    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:11:15.920572 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:11:15.920540    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:11:17.633549 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:11:17.633511    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5cdbbd5bdc-j9j6w"
Sep 09 21:11:17.635111 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:11:17.635076    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5cdbbd5bdc-j9j6w" podUID="9baaa876-0d93-4580-a0f9-e6cbe1175f5f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/health\": dial tcp 10.132.0.48:8000: connect: connection refused"
Sep 09 21:11:22.118740 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:11:22.118704    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 09 21:11:22.153426 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:11:22.153388    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:11:22.220100 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:11:22.220064    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:11:27.633310 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:11:27.633264    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5cdbbd5bdc-j9j6w" podUID="9baaa876-0d93-4580-a0f9-e6cbe1175f5f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/health\": dial tcp 10.132.0.48:8000: connect: connection refused"
Sep 09 21:11:37.141967 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:11:37.141919    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="db7ecaa9-1f5f-4957-9145-e02b5c9c1b43" containerName="main" containerID="cri-o://a58d7f7408fcd2ec1adced9a078741bac6b4e918931182b3cfa61bb01f833023" gracePeriod=45
Sep 09 21:11:37.491301 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:11:37.491211    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="db7ecaa9-1f5f-4957-9145-e02b5c9c1b43" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/health\": dial tcp 10.132.0.44:8000: connect: connection refused"
Sep 09 21:11:37.633546 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:11:37.633493    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5cdbbd5bdc-j9j6w" podUID="9baaa876-0d93-4580-a0f9-e6cbe1175f5f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/health\": dial tcp 10.132.0.48:8000: connect: connection refused"
Sep 09 21:11:37.901095 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:11:37.901068    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 09 21:11:37.942215 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:11:37.942178    2576 generic.go:356] "Generic (PLEG): container finished" podID="db7ecaa9-1f5f-4957-9145-e02b5c9c1b43" containerID="a58d7f7408fcd2ec1adced9a078741bac6b4e918931182b3cfa61bb01f833023" exitCode=0
Sep 09 21:11:37.942447 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:11:37.942238    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 09 21:11:37.942447 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:11:37.942237    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"db7ecaa9-1f5f-4957-9145-e02b5c9c1b43","Type":"ContainerDied","Data":"a58d7f7408fcd2ec1adced9a078741bac6b4e918931182b3cfa61bb01f833023"}
Sep 09 21:11:37.942447 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:11:37.942350    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"db7ecaa9-1f5f-4957-9145-e02b5c9c1b43","Type":"ContainerDied","Data":"41c6103bd469b3274e4f814c2f78189b7d326f848852550180555217b35ec74a"}
Sep 09 21:11:37.942447 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:11:37.942399    2576 scope.go:117] "RemoveContainer" containerID="a58d7f7408fcd2ec1adced9a078741bac6b4e918931182b3cfa61bb01f833023"
Sep 09 21:11:37.951736 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:11:37.951717    2576 scope.go:117] "RemoveContainer" containerID="797794d815bfc6cb1e5d9354ef9e78b37ad589dc07f4f3dfae0b2284afeeac78"
Sep 09 21:11:37.960032 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:11:37.960013    2576 scope.go:117] "RemoveContainer" containerID="a58d7f7408fcd2ec1adced9a078741bac6b4e918931182b3cfa61bb01f833023"
Sep 09 21:11:37.960345 ip-10-0-134-72 kubenswrapper[2576]: E0909 21:11:37.960317    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a58d7f7408fcd2ec1adced9a078741bac6b4e918931182b3cfa61bb01f833023\": container with ID starting with a58d7f7408fcd2ec1adced9a078741bac6b4e918931182b3cfa61bb01f833023 not found: ID does not exist" containerID="a58d7f7408fcd2ec1adced9a078741bac6b4e918931182b3cfa61bb01f833023"
Sep 09 21:11:37.960466 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:11:37.960353    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a58d7f7408fcd2ec1adced9a078741bac6b4e918931182b3cfa61bb01f833023"} err="failed to get container status \"a58d7f7408fcd2ec1adced9a078741bac6b4e918931182b3cfa61bb01f833023\": rpc error: code = NotFound desc = could not find container \"a58d7f7408fcd2ec1adced9a078741bac6b4e918931182b3cfa61bb01f833023\": container with ID starting with a58d7f7408fcd2ec1adced9a078741bac6b4e918931182b3cfa61bb01f833023 not found: ID does not exist"
Sep 09 21:11:37.960466 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:11:37.960383    2576 scope.go:117] "RemoveContainer" containerID="797794d815bfc6cb1e5d9354ef9e78b37ad589dc07f4f3dfae0b2284afeeac78"
Sep 09 21:11:37.960656 ip-10-0-134-72 kubenswrapper[2576]: E0909 21:11:37.960639    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"797794d815bfc6cb1e5d9354ef9e78b37ad589dc07f4f3dfae0b2284afeeac78\": container with ID starting with 797794d815bfc6cb1e5d9354ef9e78b37ad589dc07f4f3dfae0b2284afeeac78 not found: ID does not exist" containerID="797794d815bfc6cb1e5d9354ef9e78b37ad589dc07f4f3dfae0b2284afeeac78"
Sep 09 21:11:37.960703 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:11:37.960662    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"797794d815bfc6cb1e5d9354ef9e78b37ad589dc07f4f3dfae0b2284afeeac78"} err="failed to get container status \"797794d815bfc6cb1e5d9354ef9e78b37ad589dc07f4f3dfae0b2284afeeac78\": rpc error: code = NotFound desc = could not find container \"797794d815bfc6cb1e5d9354ef9e78b37ad589dc07f4f3dfae0b2284afeeac78\": container with ID starting with 797794d815bfc6cb1e5d9354ef9e78b37ad589dc07f4f3dfae0b2284afeeac78 not found: ID does not exist"
Sep 09 21:11:37.964908 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:11:37.964894    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db7ecaa9-1f5f-4957-9145-e02b5c9c1b43-tmp-dir\") pod \"db7ecaa9-1f5f-4957-9145-e02b5c9c1b43\" (UID: \"db7ecaa9-1f5f-4957-9145-e02b5c9c1b43\") "
Sep 09 21:11:37.964996 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:11:37.964919    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/db7ecaa9-1f5f-4957-9145-e02b5c9c1b43-kserve-provision-location\") pod \"db7ecaa9-1f5f-4957-9145-e02b5c9c1b43\" (UID: \"db7ecaa9-1f5f-4957-9145-e02b5c9c1b43\") "
Sep 09 21:11:37.964996 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:11:37.964935    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/db7ecaa9-1f5f-4957-9145-e02b5c9c1b43-cabundle-cert\") pod \"db7ecaa9-1f5f-4957-9145-e02b5c9c1b43\" (UID: \"db7ecaa9-1f5f-4957-9145-e02b5c9c1b43\") "
Sep 09 21:11:37.964996 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:11:37.964979    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db7ecaa9-1f5f-4957-9145-e02b5c9c1b43-home\") pod \"db7ecaa9-1f5f-4957-9145-e02b5c9c1b43\" (UID: \"db7ecaa9-1f5f-4957-9145-e02b5c9c1b43\") "
Sep 09 21:11:37.965153 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:11:37.965008    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db7ecaa9-1f5f-4957-9145-e02b5c9c1b43-dshm\") pod \"db7ecaa9-1f5f-4957-9145-e02b5c9c1b43\" (UID: \"db7ecaa9-1f5f-4957-9145-e02b5c9c1b43\") "
Sep 09 21:11:37.965153 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:11:37.965076    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db7ecaa9-1f5f-4957-9145-e02b5c9c1b43-tls-certs\") pod \"db7ecaa9-1f5f-4957-9145-e02b5c9c1b43\" (UID: \"db7ecaa9-1f5f-4957-9145-e02b5c9c1b43\") "
Sep 09 21:11:37.965153 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:11:37.965116    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z7mk4\" (UniqueName: \"kubernetes.io/projected/db7ecaa9-1f5f-4957-9145-e02b5c9c1b43-kube-api-access-z7mk4\") pod \"db7ecaa9-1f5f-4957-9145-e02b5c9c1b43\" (UID: \"db7ecaa9-1f5f-4957-9145-e02b5c9c1b43\") "
Sep 09 21:11:37.965153 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:11:37.965143    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db7ecaa9-1f5f-4957-9145-e02b5c9c1b43-model-cache\") pod \"db7ecaa9-1f5f-4957-9145-e02b5c9c1b43\" (UID: \"db7ecaa9-1f5f-4957-9145-e02b5c9c1b43\") "
Sep 09 21:11:37.965356 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:11:37.965320    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db7ecaa9-1f5f-4957-9145-e02b5c9c1b43-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "db7ecaa9-1f5f-4957-9145-e02b5c9c1b43" (UID: "db7ecaa9-1f5f-4957-9145-e02b5c9c1b43"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:11:37.965462 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:11:37.965426    2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/db7ecaa9-1f5f-4957-9145-e02b5c9c1b43-cabundle-cert\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:11:37.965870 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:11:37.965832    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db7ecaa9-1f5f-4957-9145-e02b5c9c1b43-home" (OuterVolumeSpecName: "home") pod "db7ecaa9-1f5f-4957-9145-e02b5c9c1b43" (UID: "db7ecaa9-1f5f-4957-9145-e02b5c9c1b43"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:11:37.965870 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:11:37.965855    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db7ecaa9-1f5f-4957-9145-e02b5c9c1b43-model-cache" (OuterVolumeSpecName: "model-cache") pod "db7ecaa9-1f5f-4957-9145-e02b5c9c1b43" (UID: "db7ecaa9-1f5f-4957-9145-e02b5c9c1b43"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:11:37.967718 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:11:37.967698    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db7ecaa9-1f5f-4957-9145-e02b5c9c1b43-kube-api-access-z7mk4" (OuterVolumeSpecName: "kube-api-access-z7mk4") pod "db7ecaa9-1f5f-4957-9145-e02b5c9c1b43" (UID: "db7ecaa9-1f5f-4957-9145-e02b5c9c1b43"). InnerVolumeSpecName "kube-api-access-z7mk4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:11:37.967838 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:11:37.967817    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db7ecaa9-1f5f-4957-9145-e02b5c9c1b43-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "db7ecaa9-1f5f-4957-9145-e02b5c9c1b43" (UID: "db7ecaa9-1f5f-4957-9145-e02b5c9c1b43"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:11:37.968172 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:11:37.968153    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db7ecaa9-1f5f-4957-9145-e02b5c9c1b43-dshm" (OuterVolumeSpecName: "dshm") pod "db7ecaa9-1f5f-4957-9145-e02b5c9c1b43" (UID: "db7ecaa9-1f5f-4957-9145-e02b5c9c1b43"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:11:37.983946 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:11:37.983920    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db7ecaa9-1f5f-4957-9145-e02b5c9c1b43-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "db7ecaa9-1f5f-4957-9145-e02b5c9c1b43" (UID: "db7ecaa9-1f5f-4957-9145-e02b5c9c1b43"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:11:37.995978 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:11:37.995950    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db7ecaa9-1f5f-4957-9145-e02b5c9c1b43-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "db7ecaa9-1f5f-4957-9145-e02b5c9c1b43" (UID: "db7ecaa9-1f5f-4957-9145-e02b5c9c1b43"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:11:38.065905 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:11:38.065813    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db7ecaa9-1f5f-4957-9145-e02b5c9c1b43-tls-certs\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:11:38.065905 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:11:38.065842    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7mk4\" (UniqueName: \"kubernetes.io/projected/db7ecaa9-1f5f-4957-9145-e02b5c9c1b43-kube-api-access-z7mk4\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:11:38.065905 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:11:38.065851    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db7ecaa9-1f5f-4957-9145-e02b5c9c1b43-model-cache\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:11:38.065905 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:11:38.065861    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db7ecaa9-1f5f-4957-9145-e02b5c9c1b43-tmp-dir\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:11:38.065905 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:11:38.065868    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/db7ecaa9-1f5f-4957-9145-e02b5c9c1b43-kserve-provision-location\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:11:38.065905 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:11:38.065877    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db7ecaa9-1f5f-4957-9145-e02b5c9c1b43-home\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:11:38.065905 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:11:38.065885    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db7ecaa9-1f5f-4957-9145-e02b5c9c1b43-dshm\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:11:38.266210 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:11:38.266182    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 09 21:11:38.273250 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:11:38.273213    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 09 21:11:39.409213 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:11:39.409174    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="db7ecaa9-1f5f-4957-9145-e02b5c9c1b43" path="/var/lib/kubelet/pods/db7ecaa9-1f5f-4957-9145-e02b5c9c1b43/volumes"
Sep 09 21:11:47.633926 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:11:47.633881    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5cdbbd5bdc-j9j6w" podUID="9baaa876-0d93-4580-a0f9-e6cbe1175f5f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/health\": dial tcp 10.132.0.48:8000: connect: connection refused"
Sep 09 21:11:57.633269 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:11:57.633224    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5cdbbd5bdc-j9j6w" podUID="9baaa876-0d93-4580-a0f9-e6cbe1175f5f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/health\": dial tcp 10.132.0.48:8000: connect: connection refused"
Sep 09 21:12:07.633210 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:12:07.633107    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5cdbbd5bdc-j9j6w" podUID="9baaa876-0d93-4580-a0f9-e6cbe1175f5f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/health\": dial tcp 10.132.0.48:8000: connect: connection refused"
Sep 09 21:12:12.238239 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:12:12.238196    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:12:12.336673 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:12:12.336629    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:12:17.633236 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:12:17.633193    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5cdbbd5bdc-j9j6w" podUID="9baaa876-0d93-4580-a0f9-e6cbe1175f5f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/health\": dial tcp 10.132.0.48:8000: connect: connection refused"
Sep 09 21:12:19.154601 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:12:19.154566    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:12:19.435974 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:12:19.435886    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:12:20.071132 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:12:20.071101    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:12:26.273597 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:12:26.273559    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:12:26.347285 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:12:26.347251    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:12:27.633673 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:12:27.633622    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5cdbbd5bdc-j9j6w" podUID="9baaa876-0d93-4580-a0f9-e6cbe1175f5f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/health\": dial tcp 10.132.0.48:8000: connect: connection refused"
Sep 09 21:12:36.034862 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:12:36.034813    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:12:37.633482 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:12:37.633446    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5cdbbd5bdc-j9j6w" podUID="9baaa876-0d93-4580-a0f9-e6cbe1175f5f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/health\": dial tcp 10.132.0.48:8000: connect: connection refused"
Sep 09 21:12:47.633978 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:12:47.633926    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5cdbbd5bdc-j9j6w" podUID="9baaa876-0d93-4580-a0f9-e6cbe1175f5f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/health\": dial tcp 10.132.0.48:8000: connect: connection refused"
Sep 09 21:12:53.169477 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:12:53.169432    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:12:53.256743 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:12:53.256710    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:12:57.633143 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:12:57.633106    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5cdbbd5bdc-j9j6w" podUID="9baaa876-0d93-4580-a0f9-e6cbe1175f5f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/health\": dial tcp 10.132.0.48:8000: connect: connection refused"
Sep 09 21:13:07.643140 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:13:07.643111    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5cdbbd5bdc-j9j6w"
Sep 09 21:13:07.650691 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:13:07.650665    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5cdbbd5bdc-j9j6w"
Sep 09 21:13:12.899684 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:13:12.899653    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5cdbbd5bdc-j9j6w"]
Sep 09 21:13:12.906211 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:13:12.906179    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:13:13.006486 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:13:13.006450    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:13:27.922670 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:13:27.922629    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5cdbbd5bdc-j9j6w" podUID="9baaa876-0d93-4580-a0f9-e6cbe1175f5f" containerName="main" containerID="cri-o://6980653714f46dd3f91f2bddfa789838443ba8031afb9449bb457158fc4c3e0a" gracePeriod=45
Sep 09 21:13:28.543407 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:13:28.543355    2576 generic.go:356] "Generic (PLEG): container finished" podID="9baaa876-0d93-4580-a0f9-e6cbe1175f5f" containerID="6980653714f46dd3f91f2bddfa789838443ba8031afb9449bb457158fc4c3e0a" exitCode=0
Sep 09 21:13:28.543528 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:13:28.543403    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5cdbbd5bdc-j9j6w" event={"ID":"9baaa876-0d93-4580-a0f9-e6cbe1175f5f","Type":"ContainerDied","Data":"6980653714f46dd3f91f2bddfa789838443ba8031afb9449bb457158fc4c3e0a"}
Sep 09 21:13:28.673055 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:13:28.673031    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5cdbbd5bdc-j9j6w"
Sep 09 21:13:28.843776 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:13:28.843747    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9baaa876-0d93-4580-a0f9-e6cbe1175f5f-tmp-dir\") pod \"9baaa876-0d93-4580-a0f9-e6cbe1175f5f\" (UID: \"9baaa876-0d93-4580-a0f9-e6cbe1175f5f\") "
Sep 09 21:13:28.843952 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:13:28.843790    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fcjhk\" (UniqueName: \"kubernetes.io/projected/9baaa876-0d93-4580-a0f9-e6cbe1175f5f-kube-api-access-fcjhk\") pod \"9baaa876-0d93-4580-a0f9-e6cbe1175f5f\" (UID: \"9baaa876-0d93-4580-a0f9-e6cbe1175f5f\") "
Sep 09 21:13:28.843952 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:13:28.843821    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9baaa876-0d93-4580-a0f9-e6cbe1175f5f-home\") pod \"9baaa876-0d93-4580-a0f9-e6cbe1175f5f\" (UID: \"9baaa876-0d93-4580-a0f9-e6cbe1175f5f\") "
Sep 09 21:13:28.843952 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:13:28.843853    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9baaa876-0d93-4580-a0f9-e6cbe1175f5f-cabundle-cert\") pod \"9baaa876-0d93-4580-a0f9-e6cbe1175f5f\" (UID: \"9baaa876-0d93-4580-a0f9-e6cbe1175f5f\") "
Sep 09 21:13:28.843952 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:13:28.843899    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9baaa876-0d93-4580-a0f9-e6cbe1175f5f-model-cache\") pod \"9baaa876-0d93-4580-a0f9-e6cbe1175f5f\" (UID: \"9baaa876-0d93-4580-a0f9-e6cbe1175f5f\") "
Sep 09 21:13:28.843952 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:13:28.843914    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9baaa876-0d93-4580-a0f9-e6cbe1175f5f-tls-certs\") pod \"9baaa876-0d93-4580-a0f9-e6cbe1175f5f\" (UID: \"9baaa876-0d93-4580-a0f9-e6cbe1175f5f\") "
Sep 09 21:13:28.843952 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:13:28.843945    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9baaa876-0d93-4580-a0f9-e6cbe1175f5f-kserve-provision-location\") pod \"9baaa876-0d93-4580-a0f9-e6cbe1175f5f\" (UID: \"9baaa876-0d93-4580-a0f9-e6cbe1175f5f\") "
Sep 09 21:13:28.844267 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:13:28.843974    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9baaa876-0d93-4580-a0f9-e6cbe1175f5f-dshm\") pod \"9baaa876-0d93-4580-a0f9-e6cbe1175f5f\" (UID: \"9baaa876-0d93-4580-a0f9-e6cbe1175f5f\") "
Sep 09 21:13:28.844267 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:13:28.844172    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9baaa876-0d93-4580-a0f9-e6cbe1175f5f-model-cache" (OuterVolumeSpecName: "model-cache") pod "9baaa876-0d93-4580-a0f9-e6cbe1175f5f" (UID: "9baaa876-0d93-4580-a0f9-e6cbe1175f5f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:13:28.844267 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:13:28.844245    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9baaa876-0d93-4580-a0f9-e6cbe1175f5f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "9baaa876-0d93-4580-a0f9-e6cbe1175f5f" (UID: "9baaa876-0d93-4580-a0f9-e6cbe1175f5f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:13:28.844621 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:13:28.844599    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9baaa876-0d93-4580-a0f9-e6cbe1175f5f-home" (OuterVolumeSpecName: "home") pod "9baaa876-0d93-4580-a0f9-e6cbe1175f5f" (UID: "9baaa876-0d93-4580-a0f9-e6cbe1175f5f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:13:28.846871 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:13:28.846846    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9baaa876-0d93-4580-a0f9-e6cbe1175f5f-kube-api-access-fcjhk" (OuterVolumeSpecName: "kube-api-access-fcjhk") pod "9baaa876-0d93-4580-a0f9-e6cbe1175f5f" (UID: "9baaa876-0d93-4580-a0f9-e6cbe1175f5f"). InnerVolumeSpecName "kube-api-access-fcjhk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:13:28.846989 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:13:28.846921    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9baaa876-0d93-4580-a0f9-e6cbe1175f5f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9baaa876-0d93-4580-a0f9-e6cbe1175f5f" (UID: "9baaa876-0d93-4580-a0f9-e6cbe1175f5f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:13:28.847146 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:13:28.847129    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9baaa876-0d93-4580-a0f9-e6cbe1175f5f-dshm" (OuterVolumeSpecName: "dshm") pod "9baaa876-0d93-4580-a0f9-e6cbe1175f5f" (UID: "9baaa876-0d93-4580-a0f9-e6cbe1175f5f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:13:28.857581 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:13:28.857555    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9baaa876-0d93-4580-a0f9-e6cbe1175f5f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9baaa876-0d93-4580-a0f9-e6cbe1175f5f" (UID: "9baaa876-0d93-4580-a0f9-e6cbe1175f5f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:13:28.867652 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:13:28.867629    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9baaa876-0d93-4580-a0f9-e6cbe1175f5f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9baaa876-0d93-4580-a0f9-e6cbe1175f5f" (UID: "9baaa876-0d93-4580-a0f9-e6cbe1175f5f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:13:28.944774 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:13:28.944734    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9baaa876-0d93-4580-a0f9-e6cbe1175f5f-home\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:13:28.944774 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:13:28.944767    2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9baaa876-0d93-4580-a0f9-e6cbe1175f5f-cabundle-cert\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:13:28.944774 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:13:28.944780    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9baaa876-0d93-4580-a0f9-e6cbe1175f5f-model-cache\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:13:28.945197 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:13:28.944792    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9baaa876-0d93-4580-a0f9-e6cbe1175f5f-tls-certs\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:13:28.945197 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:13:28.944804    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9baaa876-0d93-4580-a0f9-e6cbe1175f5f-kserve-provision-location\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:13:28.945197 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:13:28.944815    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9baaa876-0d93-4580-a0f9-e6cbe1175f5f-dshm\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:13:28.945197 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:13:28.944825    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9baaa876-0d93-4580-a0f9-e6cbe1175f5f-tmp-dir\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:13:28.945197 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:13:28.944836    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fcjhk\" (UniqueName: \"kubernetes.io/projected/9baaa876-0d93-4580-a0f9-e6cbe1175f5f-kube-api-access-fcjhk\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:13:29.555022 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:13:29.554987    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5cdbbd5bdc-j9j6w" event={"ID":"9baaa876-0d93-4580-a0f9-e6cbe1175f5f","Type":"ContainerDied","Data":"546898583e61e66dbad101a5335368fafdef839c7a100db8504e194e2435cceb"}
Sep 09 21:13:29.555184 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:13:29.555038    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5cdbbd5bdc-j9j6w"
Sep 09 21:13:29.555184 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:13:29.555069    2576 scope.go:117] "RemoveContainer" containerID="6980653714f46dd3f91f2bddfa789838443ba8031afb9449bb457158fc4c3e0a"
Sep 09 21:13:29.565608 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:13:29.565594    2576 scope.go:117] "RemoveContainer" containerID="dbeeed1b72b0a615e7dc8b9b45762293b4399969b46c186ca48514775b20601e"
Sep 09 21:13:29.586542 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:13:29.586512    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5cdbbd5bdc-j9j6w"]
Sep 09 21:13:29.596659 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:13:29.596636    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5cdbbd5bdc-j9j6w"]
Sep 09 21:13:29.643757 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:13:29.643722    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5cdbbd5bdc-j9j6w" podUID="9baaa876-0d93-4580-a0f9-e6cbe1175f5f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/health\": context deadline exceeded"
Sep 09 21:13:31.409261 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:13:31.409225    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9baaa876-0d93-4580-a0f9-e6cbe1175f5f" path="/var/lib/kubelet/pods/9baaa876-0d93-4580-a0f9-e6cbe1175f5f/volumes"
Sep 09 21:13:43.210311 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:13:43.210260    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:13:43.299041 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:13:43.299010    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:13:50.372230 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:13:50.372194    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:13:51.728568 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:13:51.728531    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fm7ggz"]
Sep 09 21:13:51.728918 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:13:51.728856    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db7ecaa9-1f5f-4957-9145-e02b5c9c1b43" containerName="main"
Sep 09 21:13:51.728918 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:13:51.728867    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="db7ecaa9-1f5f-4957-9145-e02b5c9c1b43" containerName="main"
Sep 09 21:13:51.728918 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:13:51.728886    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db7ecaa9-1f5f-4957-9145-e02b5c9c1b43" containerName="storage-initializer"
Sep 09 21:13:51.728918 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:13:51.728892    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="db7ecaa9-1f5f-4957-9145-e02b5c9c1b43" containerName="storage-initializer"
Sep 09 21:13:51.728918 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:13:51.728898    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9baaa876-0d93-4580-a0f9-e6cbe1175f5f" containerName="main"
Sep 09 21:13:51.728918 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:13:51.728904    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="9baaa876-0d93-4580-a0f9-e6cbe1175f5f" containerName="main"
Sep 09 21:13:51.728918 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:13:51.728921    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9baaa876-0d93-4580-a0f9-e6cbe1175f5f" containerName="storage-initializer"
Sep 09 21:13:51.729137 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:13:51.728927    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="9baaa876-0d93-4580-a0f9-e6cbe1175f5f" containerName="storage-initializer"
Sep 09 21:13:51.729137 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:13:51.728973    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="db7ecaa9-1f5f-4957-9145-e02b5c9c1b43" containerName="main"
Sep 09 21:13:51.729137 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:13:51.728984    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="9baaa876-0d93-4580-a0f9-e6cbe1175f5f" containerName="main"
Sep 09 21:13:51.733196 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:13:51.733179    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fm7ggz"
Sep 09 21:13:51.735796 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:13:51.735776    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"openshift-service-ca.crt\""
Sep 09 21:13:51.736729 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:13:51.736710    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"default-dockercfg-vsjdz\""
Sep 09 21:13:51.736821 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:13:51.736798    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"kube-root-ca.crt\""
Sep 09 21:13:51.736892 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:13:51.736874    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"llmisv431d7e30f13554296364403a5b6365f5-kserve-self-signed-certs\""
Sep 09 21:13:51.751650 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:13:51.751613    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fm7ggz"]
Sep 09 21:13:51.753203 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:13:51.753179    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:13:51.835428 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:13:51.835390    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8428bcfb-b062-4b58-80d2-8a2dd217475d-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fm7ggz\" (UID: \"8428bcfb-b062-4b58-80d2-8a2dd217475d\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fm7ggz"
Sep 09 21:13:51.835582 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:13:51.835444    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8428bcfb-b062-4b58-80d2-8a2dd217475d-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fm7ggz\" (UID: \"8428bcfb-b062-4b58-80d2-8a2dd217475d\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fm7ggz"
Sep 09 21:13:51.835582 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:13:51.835480    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xd87t\" (UniqueName: \"kubernetes.io/projected/8428bcfb-b062-4b58-80d2-8a2dd217475d-kube-api-access-xd87t\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fm7ggz\" (UID: \"8428bcfb-b062-4b58-80d2-8a2dd217475d\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fm7ggz"
Sep 09 21:13:51.835582 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:13:51.835512    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8428bcfb-b062-4b58-80d2-8a2dd217475d-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fm7ggz\" (UID: \"8428bcfb-b062-4b58-80d2-8a2dd217475d\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fm7ggz"
Sep 09 21:13:51.835693 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:13:51.835598    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8428bcfb-b062-4b58-80d2-8a2dd217475d-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fm7ggz\" (UID: \"8428bcfb-b062-4b58-80d2-8a2dd217475d\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fm7ggz"
Sep 09 21:13:51.835693 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:13:51.835629    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8428bcfb-b062-4b58-80d2-8a2dd217475d-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fm7ggz\" (UID: \"8428bcfb-b062-4b58-80d2-8a2dd217475d\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fm7ggz"
Sep 09 21:13:51.936911 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:13:51.936881    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xd87t\" (UniqueName: \"kubernetes.io/projected/8428bcfb-b062-4b58-80d2-8a2dd217475d-kube-api-access-xd87t\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fm7ggz\" (UID: \"8428bcfb-b062-4b58-80d2-8a2dd217475d\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fm7ggz"
Sep 09 21:13:51.937081 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:13:51.936926    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8428bcfb-b062-4b58-80d2-8a2dd217475d-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fm7ggz\" (UID: \"8428bcfb-b062-4b58-80d2-8a2dd217475d\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fm7ggz"
Sep 09 21:13:51.937081 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:13:51.936964    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8428bcfb-b062-4b58-80d2-8a2dd217475d-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fm7ggz\" (UID: \"8428bcfb-b062-4b58-80d2-8a2dd217475d\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fm7ggz"
Sep 09 21:13:51.937081 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:13:51.936990    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8428bcfb-b062-4b58-80d2-8a2dd217475d-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fm7ggz\" (UID: \"8428bcfb-b062-4b58-80d2-8a2dd217475d\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fm7ggz"
Sep 09 21:13:51.937081 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:13:51.937040    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8428bcfb-b062-4b58-80d2-8a2dd217475d-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fm7ggz\" (UID: \"8428bcfb-b062-4b58-80d2-8a2dd217475d\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fm7ggz"
Sep 09 21:13:51.937327 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:13:51.937082    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8428bcfb-b062-4b58-80d2-8a2dd217475d-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fm7ggz\" (UID: \"8428bcfb-b062-4b58-80d2-8a2dd217475d\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fm7ggz"
Sep 09 21:13:51.937433 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:13:51.937409    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8428bcfb-b062-4b58-80d2-8a2dd217475d-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fm7ggz\" (UID: \"8428bcfb-b062-4b58-80d2-8a2dd217475d\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fm7ggz"
Sep 09 21:13:51.937494 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:13:51.937462    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8428bcfb-b062-4b58-80d2-8a2dd217475d-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fm7ggz\" (UID: \"8428bcfb-b062-4b58-80d2-8a2dd217475d\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fm7ggz"
Sep 09 21:13:51.937558 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:13:51.937494    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8428bcfb-b062-4b58-80d2-8a2dd217475d-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fm7ggz\" (UID: \"8428bcfb-b062-4b58-80d2-8a2dd217475d\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fm7ggz"
Sep 09 21:13:51.939408 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:13:51.939387    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8428bcfb-b062-4b58-80d2-8a2dd217475d-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fm7ggz\" (UID: \"8428bcfb-b062-4b58-80d2-8a2dd217475d\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fm7ggz"
Sep 09 21:13:51.939759 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:13:51.939737    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8428bcfb-b062-4b58-80d2-8a2dd217475d-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fm7ggz\" (UID: \"8428bcfb-b062-4b58-80d2-8a2dd217475d\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fm7ggz"
Sep 09 21:13:51.951524 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:13:51.951496    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xd87t\" (UniqueName: \"kubernetes.io/projected/8428bcfb-b062-4b58-80d2-8a2dd217475d-kube-api-access-xd87t\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fm7ggz\" (UID: \"8428bcfb-b062-4b58-80d2-8a2dd217475d\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fm7ggz"
Sep 09 21:13:52.003306 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:13:52.003211    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:13:52.043328 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:13:52.043299    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fm7ggz"
Sep 09 21:13:52.322615 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:13:52.322591    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fm7ggz"]
Sep 09 21:13:52.324773 ip-10-0-134-72 kubenswrapper[2576]: W0909 21:13:52.324745    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8428bcfb_b062_4b58_80d2_8a2dd217475d.slice/crio-600075fdbe6c1e6d88fdf7b8f4325af5f2c18787ad0659d2ecaa361db680318e WatchSource:0}: Error finding container 600075fdbe6c1e6d88fdf7b8f4325af5f2c18787ad0659d2ecaa361db680318e: Status 404 returned error can't find the container with id 600075fdbe6c1e6d88fdf7b8f4325af5f2c18787ad0659d2ecaa361db680318e
Sep 09 21:13:52.676731 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:13:52.676695    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fm7ggz" event={"ID":"8428bcfb-b062-4b58-80d2-8a2dd217475d","Type":"ContainerStarted","Data":"9d015ca582cc481bc2624e7ee38bcfa1b20481fd924acebe247f94a0c7cec287"}
Sep 09 21:13:52.676892 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:13:52.676756    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fm7ggz" event={"ID":"8428bcfb-b062-4b58-80d2-8a2dd217475d","Type":"ContainerStarted","Data":"600075fdbe6c1e6d88fdf7b8f4325af5f2c18787ad0659d2ecaa361db680318e"}
Sep 09 21:13:52.706717 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:13:52.706656    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fm7ggz" podStartSLOduration=1.706640364 podStartE2EDuration="1.706640364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:13:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:13:52.70343317 +0000 UTC m=+1275.817119028" watchObservedRunningTime="2026-09-09 21:13:52.706640364 +0000 UTC m=+1275.820326223"
Sep 09 21:13:53.043507 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:13:53.043413    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fm7ggz"
Sep 09 21:13:56.432930 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:13:56.432902    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:13:56.533808 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:13:56.533779    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:13:59.049114 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:13:59.049078    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:14:02.043773 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:14:02.043735    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fm7ggz"
Sep 09 21:14:02.056436 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:14:02.056408    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fm7ggz"
Sep 09 21:14:02.737710 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:14:02.737677    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fm7ggz"
Sep 09 21:14:02.885063 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:14:02.885032    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:14:03.022120 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:14:03.022033    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:14:03.173656 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:14:03.173620    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:14:07.851706 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:14:07.851677    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:14:07.969805 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:14:07.969771    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:14:09.877944 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:14:09.877911    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:14:11.184243 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:14:11.184216    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:14:11.419733 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:14:11.419693    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:14:16.335651 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:14:16.335603    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:14:16.419637 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:14:16.419514    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:14:20.045922 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:14:20.045892    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:14:25.878063 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:14:25.878007    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:14:25.894672 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:14:25.894642    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fm7ggz"]
Sep 09 21:14:25.903643 ip-10-0-134-72 kubenswrapper[2576]: E0909 21:14:25.903615    2576 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 09 21:14:25.903643 ip-10-0-134-72 kubenswrapper[2576]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 21:14:25.903643 ip-10-0-134-72 kubenswrapper[2576]:         , stdout: , stderr: , exit code -1
Sep 09 21:14:25.903643 ip-10-0-134-72 kubenswrapper[2576]:  > containerID="9d015ca582cc481bc2624e7ee38bcfa1b20481fd924acebe247f94a0c7cec287" cmd=["/bin/sleep","15"]
Sep 09 21:14:25.903835 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:14:25.903659    2576 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fm7ggz" message="" err=<
Sep 09 21:14:25.903835 ip-10-0-134-72 kubenswrapper[2576]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 21:14:25.903835 ip-10-0-134-72 kubenswrapper[2576]:         , stdout: , stderr: , exit code -1
Sep 09 21:14:25.903835 ip-10-0-134-72 kubenswrapper[2576]:  >
Sep 09 21:14:25.903835 ip-10-0-134-72 kubenswrapper[2576]: E0909 21:14:25.903671    2576 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 09 21:14:25.903835 ip-10-0-134-72 kubenswrapper[2576]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 21:14:25.903835 ip-10-0-134-72 kubenswrapper[2576]:         , stdout: , stderr: , exit code -1
Sep 09 21:14:25.903835 ip-10-0-134-72 kubenswrapper[2576]:  > pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fm7ggz" podUID="8428bcfb-b062-4b58-80d2-8a2dd217475d" containerName="main" containerID="cri-o://9d015ca582cc481bc2624e7ee38bcfa1b20481fd924acebe247f94a0c7cec287"
Sep 09 21:14:25.903835 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:14:25.903715    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fm7ggz" podUID="8428bcfb-b062-4b58-80d2-8a2dd217475d" containerName="main" containerID="cri-o://9d015ca582cc481bc2624e7ee38bcfa1b20481fd924acebe247f94a0c7cec287" gracePeriod=60
Sep 09 21:14:26.004667 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:14:26.004624    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:14:26.241206 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:14:26.241182    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fm7ggz"
Sep 09 21:14:26.336665 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:14:26.336634    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xd87t\" (UniqueName: \"kubernetes.io/projected/8428bcfb-b062-4b58-80d2-8a2dd217475d-kube-api-access-xd87t\") pod \"8428bcfb-b062-4b58-80d2-8a2dd217475d\" (UID: \"8428bcfb-b062-4b58-80d2-8a2dd217475d\") "
Sep 09 21:14:26.336827 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:14:26.336686    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8428bcfb-b062-4b58-80d2-8a2dd217475d-tls-certs\") pod \"8428bcfb-b062-4b58-80d2-8a2dd217475d\" (UID: \"8428bcfb-b062-4b58-80d2-8a2dd217475d\") "
Sep 09 21:14:26.336827 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:14:26.336708    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8428bcfb-b062-4b58-80d2-8a2dd217475d-model-cache\") pod \"8428bcfb-b062-4b58-80d2-8a2dd217475d\" (UID: \"8428bcfb-b062-4b58-80d2-8a2dd217475d\") "
Sep 09 21:14:26.336827 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:14:26.336800    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8428bcfb-b062-4b58-80d2-8a2dd217475d-tmp-dir\") pod \"8428bcfb-b062-4b58-80d2-8a2dd217475d\" (UID: \"8428bcfb-b062-4b58-80d2-8a2dd217475d\") "
Sep 09 21:14:26.337003 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:14:26.336862    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8428bcfb-b062-4b58-80d2-8a2dd217475d-home\") pod \"8428bcfb-b062-4b58-80d2-8a2dd217475d\" (UID: \"8428bcfb-b062-4b58-80d2-8a2dd217475d\") "
Sep 09 21:14:26.337003 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:14:26.336888    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8428bcfb-b062-4b58-80d2-8a2dd217475d-dshm\") pod \"8428bcfb-b062-4b58-80d2-8a2dd217475d\" (UID: \"8428bcfb-b062-4b58-80d2-8a2dd217475d\") "
Sep 09 21:14:26.337003 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:14:26.336956    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8428bcfb-b062-4b58-80d2-8a2dd217475d-model-cache" (OuterVolumeSpecName: "model-cache") pod "8428bcfb-b062-4b58-80d2-8a2dd217475d" (UID: "8428bcfb-b062-4b58-80d2-8a2dd217475d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:14:26.337160 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:14:26.337000    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8428bcfb-b062-4b58-80d2-8a2dd217475d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8428bcfb-b062-4b58-80d2-8a2dd217475d" (UID: "8428bcfb-b062-4b58-80d2-8a2dd217475d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:14:26.337160 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:14:26.337120    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8428bcfb-b062-4b58-80d2-8a2dd217475d-home" (OuterVolumeSpecName: "home") pod "8428bcfb-b062-4b58-80d2-8a2dd217475d" (UID: "8428bcfb-b062-4b58-80d2-8a2dd217475d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:14:26.337267 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:14:26.337167    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8428bcfb-b062-4b58-80d2-8a2dd217475d-model-cache\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:14:26.337267 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:14:26.337184    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8428bcfb-b062-4b58-80d2-8a2dd217475d-tmp-dir\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:14:26.339093 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:14:26.339066    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8428bcfb-b062-4b58-80d2-8a2dd217475d-dshm" (OuterVolumeSpecName: "dshm") pod "8428bcfb-b062-4b58-80d2-8a2dd217475d" (UID: "8428bcfb-b062-4b58-80d2-8a2dd217475d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:14:26.339197 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:14:26.339090    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8428bcfb-b062-4b58-80d2-8a2dd217475d-kube-api-access-xd87t" (OuterVolumeSpecName: "kube-api-access-xd87t") pod "8428bcfb-b062-4b58-80d2-8a2dd217475d" (UID: "8428bcfb-b062-4b58-80d2-8a2dd217475d"). InnerVolumeSpecName "kube-api-access-xd87t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:14:26.339197 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:14:26.339096    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8428bcfb-b062-4b58-80d2-8a2dd217475d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8428bcfb-b062-4b58-80d2-8a2dd217475d" (UID: "8428bcfb-b062-4b58-80d2-8a2dd217475d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:14:26.437692 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:14:26.437627    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xd87t\" (UniqueName: \"kubernetes.io/projected/8428bcfb-b062-4b58-80d2-8a2dd217475d-kube-api-access-xd87t\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:14:26.437692 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:14:26.437650    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8428bcfb-b062-4b58-80d2-8a2dd217475d-tls-certs\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:14:26.437692 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:14:26.437660    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8428bcfb-b062-4b58-80d2-8a2dd217475d-home\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:14:26.437692 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:14:26.437668    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8428bcfb-b062-4b58-80d2-8a2dd217475d-dshm\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:14:26.852111 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:14:26.852070    2576 generic.go:356] "Generic (PLEG): container finished" podID="8428bcfb-b062-4b58-80d2-8a2dd217475d" containerID="9d015ca582cc481bc2624e7ee38bcfa1b20481fd924acebe247f94a0c7cec287" exitCode=0
Sep 09 21:14:26.852279 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:14:26.852117    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fm7ggz"
Sep 09 21:14:26.852279 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:14:26.852131    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fm7ggz" event={"ID":"8428bcfb-b062-4b58-80d2-8a2dd217475d","Type":"ContainerDied","Data":"9d015ca582cc481bc2624e7ee38bcfa1b20481fd924acebe247f94a0c7cec287"}
Sep 09 21:14:26.852279 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:14:26.852180    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fm7ggz" event={"ID":"8428bcfb-b062-4b58-80d2-8a2dd217475d","Type":"ContainerDied","Data":"600075fdbe6c1e6d88fdf7b8f4325af5f2c18787ad0659d2ecaa361db680318e"}
Sep 09 21:14:26.852279 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:14:26.852204    2576 scope.go:117] "RemoveContainer" containerID="9d015ca582cc481bc2624e7ee38bcfa1b20481fd924acebe247f94a0c7cec287"
Sep 09 21:14:26.862072 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:14:26.862056    2576 scope.go:117] "RemoveContainer" containerID="9d015ca582cc481bc2624e7ee38bcfa1b20481fd924acebe247f94a0c7cec287"
Sep 09 21:14:26.862320 ip-10-0-134-72 kubenswrapper[2576]: E0909 21:14:26.862302    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d015ca582cc481bc2624e7ee38bcfa1b20481fd924acebe247f94a0c7cec287\": container with ID starting with 9d015ca582cc481bc2624e7ee38bcfa1b20481fd924acebe247f94a0c7cec287 not found: ID does not exist" containerID="9d015ca582cc481bc2624e7ee38bcfa1b20481fd924acebe247f94a0c7cec287"
Sep 09 21:14:26.862406 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:14:26.862327    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d015ca582cc481bc2624e7ee38bcfa1b20481fd924acebe247f94a0c7cec287"} err="failed to get container status \"9d015ca582cc481bc2624e7ee38bcfa1b20481fd924acebe247f94a0c7cec287\": rpc error: code = NotFound desc = could not find container \"9d015ca582cc481bc2624e7ee38bcfa1b20481fd924acebe247f94a0c7cec287\": container with ID starting with 9d015ca582cc481bc2624e7ee38bcfa1b20481fd924acebe247f94a0c7cec287 not found: ID does not exist"
Sep 09 21:14:26.877665 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:14:26.877643    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fm7ggz"]
Sep 09 21:14:26.880973 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:14:26.880949    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fm7ggz"]
Sep 09 21:14:27.409215 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:14:27.409173    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8428bcfb-b062-4b58-80d2-8a2dd217475d" path="/var/lib/kubelet/pods/8428bcfb-b062-4b58-80d2-8a2dd217475d/volumes"
Sep 09 21:15:15.843526 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:15:15.843493    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:15:17.807911 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:15:17.807870    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:15:23.036696 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:15:23.036651    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:15:26.285240 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:15:26.285201    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-4x5nn"]
Sep 09 21:15:26.285719 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:15:26.285703    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8428bcfb-b062-4b58-80d2-8a2dd217475d" containerName="main"
Sep 09 21:15:26.285762 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:15:26.285722    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="8428bcfb-b062-4b58-80d2-8a2dd217475d" containerName="main"
Sep 09 21:15:26.285861 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:15:26.285850    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="8428bcfb-b062-4b58-80d2-8a2dd217475d" containerName="main"
Sep 09 21:15:26.288931 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:15:26.288914    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-4x5nn"
Sep 09 21:15:26.292464 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:15:26.292444    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"default-dockercfg-j4qtt\""
Sep 09 21:15:26.293446 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:15:26.293422    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"kube-root-ca.crt\""
Sep 09 21:15:26.293446 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:15:26.293434    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"openshift-service-ca.crt\""
Sep 09 21:15:26.293608 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:15:26.293435    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"scheduler-configmap-ref-test-kserve-self-signed-certs\""
Sep 09 21:15:26.303322 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:15:26.303301    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-4x5nn"]
Sep 09 21:15:26.325385 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:15:26.325336    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:15:26.335378 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:15:26.335344    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r842v\" (UniqueName: \"kubernetes.io/projected/32d5b9ca-ad2b-4f74-acfc-791e617b65e1-kube-api-access-r842v\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-4x5nn\" (UID: \"32d5b9ca-ad2b-4f74-acfc-791e617b65e1\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-4x5nn"
Sep 09 21:15:26.335479 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:15:26.335400    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32d5b9ca-ad2b-4f74-acfc-791e617b65e1-home\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-4x5nn\" (UID: \"32d5b9ca-ad2b-4f74-acfc-791e617b65e1\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-4x5nn"
Sep 09 21:15:26.335479 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:15:26.335439    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32d5b9ca-ad2b-4f74-acfc-791e617b65e1-model-cache\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-4x5nn\" (UID: \"32d5b9ca-ad2b-4f74-acfc-791e617b65e1\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-4x5nn"
Sep 09 21:15:26.335552 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:15:26.335478    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32d5b9ca-ad2b-4f74-acfc-791e617b65e1-dshm\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-4x5nn\" (UID: \"32d5b9ca-ad2b-4f74-acfc-791e617b65e1\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-4x5nn"
Sep 09 21:15:26.335552 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:15:26.335513    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32d5b9ca-ad2b-4f74-acfc-791e617b65e1-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-4x5nn\" (UID: \"32d5b9ca-ad2b-4f74-acfc-791e617b65e1\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-4x5nn"
Sep 09 21:15:26.335621 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:15:26.335563    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32d5b9ca-ad2b-4f74-acfc-791e617b65e1-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-4x5nn\" (UID: \"32d5b9ca-ad2b-4f74-acfc-791e617b65e1\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-4x5nn"
Sep 09 21:15:26.436219 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:15:26.436189    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32d5b9ca-ad2b-4f74-acfc-791e617b65e1-dshm\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-4x5nn\" (UID: \"32d5b9ca-ad2b-4f74-acfc-791e617b65e1\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-4x5nn"
Sep 09 21:15:26.436383 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:15:26.436231    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32d5b9ca-ad2b-4f74-acfc-791e617b65e1-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-4x5nn\" (UID: \"32d5b9ca-ad2b-4f74-acfc-791e617b65e1\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-4x5nn"
Sep 09 21:15:26.436383 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:15:26.436259    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32d5b9ca-ad2b-4f74-acfc-791e617b65e1-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-4x5nn\" (UID: \"32d5b9ca-ad2b-4f74-acfc-791e617b65e1\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-4x5nn"
Sep 09 21:15:26.436507 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:15:26.436454    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r842v\" (UniqueName: \"kubernetes.io/projected/32d5b9ca-ad2b-4f74-acfc-791e617b65e1-kube-api-access-r842v\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-4x5nn\" (UID: \"32d5b9ca-ad2b-4f74-acfc-791e617b65e1\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-4x5nn"
Sep 09 21:15:26.436571 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:15:26.436508    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32d5b9ca-ad2b-4f74-acfc-791e617b65e1-home\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-4x5nn\" (UID: \"32d5b9ca-ad2b-4f74-acfc-791e617b65e1\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-4x5nn"
Sep 09 21:15:26.436571 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:15:26.436560    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32d5b9ca-ad2b-4f74-acfc-791e617b65e1-model-cache\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-4x5nn\" (UID: \"32d5b9ca-ad2b-4f74-acfc-791e617b65e1\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-4x5nn"
Sep 09 21:15:26.436677 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:15:26.436640    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32d5b9ca-ad2b-4f74-acfc-791e617b65e1-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-4x5nn\" (UID: \"32d5b9ca-ad2b-4f74-acfc-791e617b65e1\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-4x5nn"
Sep 09 21:15:26.436894 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:15:26.436875    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32d5b9ca-ad2b-4f74-acfc-791e617b65e1-model-cache\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-4x5nn\" (UID: \"32d5b9ca-ad2b-4f74-acfc-791e617b65e1\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-4x5nn"
Sep 09 21:15:26.437010 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:15:26.436921    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32d5b9ca-ad2b-4f74-acfc-791e617b65e1-home\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-4x5nn\" (UID: \"32d5b9ca-ad2b-4f74-acfc-791e617b65e1\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-4x5nn"
Sep 09 21:15:26.438774 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:15:26.438741    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32d5b9ca-ad2b-4f74-acfc-791e617b65e1-dshm\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-4x5nn\" (UID: \"32d5b9ca-ad2b-4f74-acfc-791e617b65e1\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-4x5nn"
Sep 09 21:15:26.438951 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:15:26.438937    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32d5b9ca-ad2b-4f74-acfc-791e617b65e1-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-4x5nn\" (UID: \"32d5b9ca-ad2b-4f74-acfc-791e617b65e1\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-4x5nn"
Sep 09 21:15:26.446138 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:15:26.446108    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r842v\" (UniqueName: \"kubernetes.io/projected/32d5b9ca-ad2b-4f74-acfc-791e617b65e1-kube-api-access-r842v\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-4x5nn\" (UID: \"32d5b9ca-ad2b-4f74-acfc-791e617b65e1\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-4x5nn"
Sep 09 21:15:26.600254 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:15:26.600213    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-4x5nn"
Sep 09 21:15:26.622571 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:15:26.622539    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:15:26.877657 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:15:26.877631    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-4x5nn"]
Sep 09 21:15:26.878736 ip-10-0-134-72 kubenswrapper[2576]: W0909 21:15:26.878712    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32d5b9ca_ad2b_4f74_acfc_791e617b65e1.slice/crio-3df70c71a3153bab80984b63c09136b1910d65728cb360e284ef6ea27306d166 WatchSource:0}: Error finding container 3df70c71a3153bab80984b63c09136b1910d65728cb360e284ef6ea27306d166: Status 404 returned error can't find the container with id 3df70c71a3153bab80984b63c09136b1910d65728cb360e284ef6ea27306d166
Sep 09 21:15:27.159264 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:15:27.159129    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-4x5nn" event={"ID":"32d5b9ca-ad2b-4f74-acfc-791e617b65e1","Type":"ContainerStarted","Data":"268a72838b74ba70b4072c407997f247374d70df5383d557eace20d9d73ddfac"}
Sep 09 21:15:27.159264 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:15:27.159226    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-4x5nn" event={"ID":"32d5b9ca-ad2b-4f74-acfc-791e617b65e1","Type":"ContainerStarted","Data":"3df70c71a3153bab80984b63c09136b1910d65728cb360e284ef6ea27306d166"}
Sep 09 21:15:27.182819 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:15:27.182777    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-4x5nn" podStartSLOduration=1.182762334 podStartE2EDuration="1.182762334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:15:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:15:27.18171273 +0000 UTC m=+1370.295398587" watchObservedRunningTime="2026-09-09 21:15:27.182762334 +0000 UTC m=+1370.296448208"
Sep 09 21:15:27.601189 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:15:27.601159    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-4x5nn"
Sep 09 21:15:29.530390 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:15:29.530334    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:15:29.617708 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:15:29.617679    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:15:35.845616 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:15:35.845577    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:15:36.600947 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:15:36.600914    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-4x5nn"
Sep 09 21:15:36.613533 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:15:36.613505    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-4x5nn"
Sep 09 21:15:37.221696 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:15:37.221656    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-4x5nn"
Sep 09 21:15:50.420269 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:15:50.420207    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:15:59.998815 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:15:59.998779    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:16:00.031880 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:16:00.031842    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-4x5nn"]
Sep 09 21:16:00.041167 ip-10-0-134-72 kubenswrapper[2576]: E0909 21:16:00.041128    2576 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 09 21:16:00.041167 ip-10-0-134-72 kubenswrapper[2576]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 21:16:00.041167 ip-10-0-134-72 kubenswrapper[2576]:         , stdout: , stderr: , exit code -1
Sep 09 21:16:00.041167 ip-10-0-134-72 kubenswrapper[2576]:  > containerID="268a72838b74ba70b4072c407997f247374d70df5383d557eace20d9d73ddfac" cmd=["/bin/sleep","15"]
Sep 09 21:16:00.041446 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:16:00.041180    2576 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-4x5nn" message="" err=<
Sep 09 21:16:00.041446 ip-10-0-134-72 kubenswrapper[2576]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 21:16:00.041446 ip-10-0-134-72 kubenswrapper[2576]:         , stdout: , stderr: , exit code -1
Sep 09 21:16:00.041446 ip-10-0-134-72 kubenswrapper[2576]:  >
Sep 09 21:16:00.041446 ip-10-0-134-72 kubenswrapper[2576]: E0909 21:16:00.041193    2576 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 09 21:16:00.041446 ip-10-0-134-72 kubenswrapper[2576]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 21:16:00.041446 ip-10-0-134-72 kubenswrapper[2576]:         , stdout: , stderr: , exit code -1
Sep 09 21:16:00.041446 ip-10-0-134-72 kubenswrapper[2576]:  > pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-4x5nn" podUID="32d5b9ca-ad2b-4f74-acfc-791e617b65e1" containerName="main" containerID="cri-o://268a72838b74ba70b4072c407997f247374d70df5383d557eace20d9d73ddfac"
Sep 09 21:16:00.041446 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:16:00.041229    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-4x5nn" podUID="32d5b9ca-ad2b-4f74-acfc-791e617b65e1" containerName="main" containerID="cri-o://268a72838b74ba70b4072c407997f247374d70df5383d557eace20d9d73ddfac" gracePeriod=60
Sep 09 21:16:00.095696 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:16:00.095662    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:16:00.211083 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:16:00.211044    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-4x5nn" podUID="32d5b9ca-ad2b-4f74-acfc-791e617b65e1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/health\": dial tcp 10.132.0.50:8000: connect: connection refused"
Sep 09 21:16:00.334892 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:16:00.334860    2576 generic.go:356] "Generic (PLEG): container finished" podID="32d5b9ca-ad2b-4f74-acfc-791e617b65e1" containerID="268a72838b74ba70b4072c407997f247374d70df5383d557eace20d9d73ddfac" exitCode=0
Sep 09 21:16:00.335057 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:16:00.334930    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-4x5nn" event={"ID":"32d5b9ca-ad2b-4f74-acfc-791e617b65e1","Type":"ContainerDied","Data":"268a72838b74ba70b4072c407997f247374d70df5383d557eace20d9d73ddfac"}
Sep 09 21:16:00.393769 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:16:00.393746    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-4x5nn"
Sep 09 21:16:00.443430 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:16:00.443391    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32d5b9ca-ad2b-4f74-acfc-791e617b65e1-tmp-dir\") pod \"32d5b9ca-ad2b-4f74-acfc-791e617b65e1\" (UID: \"32d5b9ca-ad2b-4f74-acfc-791e617b65e1\") "
Sep 09 21:16:00.443430 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:16:00.443430    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32d5b9ca-ad2b-4f74-acfc-791e617b65e1-model-cache\") pod \"32d5b9ca-ad2b-4f74-acfc-791e617b65e1\" (UID: \"32d5b9ca-ad2b-4f74-acfc-791e617b65e1\") "
Sep 09 21:16:00.443663 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:16:00.443454    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32d5b9ca-ad2b-4f74-acfc-791e617b65e1-home\") pod \"32d5b9ca-ad2b-4f74-acfc-791e617b65e1\" (UID: \"32d5b9ca-ad2b-4f74-acfc-791e617b65e1\") "
Sep 09 21:16:00.443663 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:16:00.443505    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32d5b9ca-ad2b-4f74-acfc-791e617b65e1-tls-certs\") pod \"32d5b9ca-ad2b-4f74-acfc-791e617b65e1\" (UID: \"32d5b9ca-ad2b-4f74-acfc-791e617b65e1\") "
Sep 09 21:16:00.443663 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:16:00.443542    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32d5b9ca-ad2b-4f74-acfc-791e617b65e1-dshm\") pod \"32d5b9ca-ad2b-4f74-acfc-791e617b65e1\" (UID: \"32d5b9ca-ad2b-4f74-acfc-791e617b65e1\") "
Sep 09 21:16:00.443663 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:16:00.443572    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r842v\" (UniqueName: \"kubernetes.io/projected/32d5b9ca-ad2b-4f74-acfc-791e617b65e1-kube-api-access-r842v\") pod \"32d5b9ca-ad2b-4f74-acfc-791e617b65e1\" (UID: \"32d5b9ca-ad2b-4f74-acfc-791e617b65e1\") "
Sep 09 21:16:00.443663 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:16:00.443641    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32d5b9ca-ad2b-4f74-acfc-791e617b65e1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "32d5b9ca-ad2b-4f74-acfc-791e617b65e1" (UID: "32d5b9ca-ad2b-4f74-acfc-791e617b65e1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:16:00.443913 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:16:00.443728    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32d5b9ca-ad2b-4f74-acfc-791e617b65e1-model-cache" (OuterVolumeSpecName: "model-cache") pod "32d5b9ca-ad2b-4f74-acfc-791e617b65e1" (UID: "32d5b9ca-ad2b-4f74-acfc-791e617b65e1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:16:00.443913 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:16:00.443745    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32d5b9ca-ad2b-4f74-acfc-791e617b65e1-home" (OuterVolumeSpecName: "home") pod "32d5b9ca-ad2b-4f74-acfc-791e617b65e1" (UID: "32d5b9ca-ad2b-4f74-acfc-791e617b65e1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:16:00.443913 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:16:00.443845    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32d5b9ca-ad2b-4f74-acfc-791e617b65e1-tmp-dir\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:16:00.443913 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:16:00.443861    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32d5b9ca-ad2b-4f74-acfc-791e617b65e1-model-cache\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:16:00.443913 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:16:00.443877    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32d5b9ca-ad2b-4f74-acfc-791e617b65e1-home\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:16:00.445842 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:16:00.445806    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32d5b9ca-ad2b-4f74-acfc-791e617b65e1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "32d5b9ca-ad2b-4f74-acfc-791e617b65e1" (UID: "32d5b9ca-ad2b-4f74-acfc-791e617b65e1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:16:00.445993 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:16:00.445867    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32d5b9ca-ad2b-4f74-acfc-791e617b65e1-dshm" (OuterVolumeSpecName: "dshm") pod "32d5b9ca-ad2b-4f74-acfc-791e617b65e1" (UID: "32d5b9ca-ad2b-4f74-acfc-791e617b65e1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:16:00.445993 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:16:00.445886    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32d5b9ca-ad2b-4f74-acfc-791e617b65e1-kube-api-access-r842v" (OuterVolumeSpecName: "kube-api-access-r842v") pod "32d5b9ca-ad2b-4f74-acfc-791e617b65e1" (UID: "32d5b9ca-ad2b-4f74-acfc-791e617b65e1"). InnerVolumeSpecName "kube-api-access-r842v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:16:00.544738 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:16:00.544651    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32d5b9ca-ad2b-4f74-acfc-791e617b65e1-dshm\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:16:00.544738 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:16:00.544684    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r842v\" (UniqueName: \"kubernetes.io/projected/32d5b9ca-ad2b-4f74-acfc-791e617b65e1-kube-api-access-r842v\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:16:00.544738 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:16:00.544693    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32d5b9ca-ad2b-4f74-acfc-791e617b65e1-tls-certs\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:16:01.341235 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:16:01.341198    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-4x5nn" event={"ID":"32d5b9ca-ad2b-4f74-acfc-791e617b65e1","Type":"ContainerDied","Data":"3df70c71a3153bab80984b63c09136b1910d65728cb360e284ef6ea27306d166"}
Sep 09 21:16:01.341235 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:16:01.341216    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-4x5nn"
Sep 09 21:16:01.341785 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:16:01.341266    2576 scope.go:117] "RemoveContainer" containerID="268a72838b74ba70b4072c407997f247374d70df5383d557eace20d9d73ddfac"
Sep 09 21:16:01.367593 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:16:01.367563    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-4x5nn"]
Sep 09 21:16:01.369438 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:16:01.369416    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-4x5nn"]
Sep 09 21:16:01.409457 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:16:01.409423    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="32d5b9ca-ad2b-4f74-acfc-791e617b65e1" path="/var/lib/kubelet/pods/32d5b9ca-ad2b-4f74-acfc-791e617b65e1/volumes"
Sep 09 21:16:03.341146 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:16:03.341107    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:16:13.190025 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:16:13.189986    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:16:13.281553 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:16:13.281520    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:16:50.059703 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:16:50.059667    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:16:50.143881 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:16:50.143839    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:16:53.819632 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:16:53.819598    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-59kkj"]
Sep 09 21:16:53.820119 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:16:53.820098    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32d5b9ca-ad2b-4f74-acfc-791e617b65e1" containerName="main"
Sep 09 21:16:53.820203 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:16:53.820122    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="32d5b9ca-ad2b-4f74-acfc-791e617b65e1" containerName="main"
Sep 09 21:16:53.820252 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:16:53.820219    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="32d5b9ca-ad2b-4f74-acfc-791e617b65e1" containerName="main"
Sep 09 21:16:53.824317 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:16:53.824293    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-59kkj"
Sep 09 21:16:53.831736 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:16:53.831717    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-kserve-self-signed-certs\""
Sep 09 21:16:53.831835 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:16:53.831731    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"openshift-service-ca.crt\""
Sep 09 21:16:53.831835 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:16:53.831788    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"default-dockercfg-4n9lv\""
Sep 09 21:16:53.831936 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:16:53.831829    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"kube-root-ca.crt\""
Sep 09 21:16:53.841648 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:16:53.841627    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-59kkj"]
Sep 09 21:16:53.870175 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:16:53.870149    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:16:53.885026 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:16:53.884999    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7600db64-50ff-4e82-bab9-9b70ea9a4224-model-cache\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-59kkj\" (UID: \"7600db64-50ff-4e82-bab9-9b70ea9a4224\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-59kkj"
Sep 09 21:16:53.885159 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:16:53.885042    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7600db64-50ff-4e82-bab9-9b70ea9a4224-dshm\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-59kkj\" (UID: \"7600db64-50ff-4e82-bab9-9b70ea9a4224\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-59kkj"
Sep 09 21:16:53.885159 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:16:53.885087    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7600db64-50ff-4e82-bab9-9b70ea9a4224-home\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-59kkj\" (UID: \"7600db64-50ff-4e82-bab9-9b70ea9a4224\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-59kkj"
Sep 09 21:16:53.885159 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:16:53.885149    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mntw7\" (UniqueName: \"kubernetes.io/projected/7600db64-50ff-4e82-bab9-9b70ea9a4224-kube-api-access-mntw7\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-59kkj\" (UID: \"7600db64-50ff-4e82-bab9-9b70ea9a4224\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-59kkj"
Sep 09 21:16:53.885339 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:16:53.885207    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7600db64-50ff-4e82-bab9-9b70ea9a4224-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-59kkj\" (UID: \"7600db64-50ff-4e82-bab9-9b70ea9a4224\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-59kkj"
Sep 09 21:16:53.885339 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:16:53.885292    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7600db64-50ff-4e82-bab9-9b70ea9a4224-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-59kkj\" (UID: \"7600db64-50ff-4e82-bab9-9b70ea9a4224\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-59kkj"
Sep 09 21:16:53.985942 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:16:53.985905    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7600db64-50ff-4e82-bab9-9b70ea9a4224-model-cache\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-59kkj\" (UID: \"7600db64-50ff-4e82-bab9-9b70ea9a4224\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-59kkj"
Sep 09 21:16:53.985942 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:16:53.985944    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7600db64-50ff-4e82-bab9-9b70ea9a4224-dshm\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-59kkj\" (UID: \"7600db64-50ff-4e82-bab9-9b70ea9a4224\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-59kkj"
Sep 09 21:16:53.986178 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:16:53.986092    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7600db64-50ff-4e82-bab9-9b70ea9a4224-home\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-59kkj\" (UID: \"7600db64-50ff-4e82-bab9-9b70ea9a4224\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-59kkj"
Sep 09 21:16:53.986241 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:16:53.986171    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mntw7\" (UniqueName: \"kubernetes.io/projected/7600db64-50ff-4e82-bab9-9b70ea9a4224-kube-api-access-mntw7\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-59kkj\" (UID: \"7600db64-50ff-4e82-bab9-9b70ea9a4224\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-59kkj"
Sep 09 21:16:53.986241 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:16:53.986231    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7600db64-50ff-4e82-bab9-9b70ea9a4224-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-59kkj\" (UID: \"7600db64-50ff-4e82-bab9-9b70ea9a4224\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-59kkj"
Sep 09 21:16:53.986346 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:16:53.986264    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7600db64-50ff-4e82-bab9-9b70ea9a4224-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-59kkj\" (UID: \"7600db64-50ff-4e82-bab9-9b70ea9a4224\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-59kkj"
Sep 09 21:16:53.986346 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:16:53.986332    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7600db64-50ff-4e82-bab9-9b70ea9a4224-model-cache\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-59kkj\" (UID: \"7600db64-50ff-4e82-bab9-9b70ea9a4224\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-59kkj"
Sep 09 21:16:53.986491 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:16:53.986444    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7600db64-50ff-4e82-bab9-9b70ea9a4224-home\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-59kkj\" (UID: \"7600db64-50ff-4e82-bab9-9b70ea9a4224\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-59kkj"
Sep 09 21:16:53.986640 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:16:53.986618    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7600db64-50ff-4e82-bab9-9b70ea9a4224-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-59kkj\" (UID: \"7600db64-50ff-4e82-bab9-9b70ea9a4224\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-59kkj"
Sep 09 21:16:53.988441 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:16:53.988414    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7600db64-50ff-4e82-bab9-9b70ea9a4224-dshm\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-59kkj\" (UID: \"7600db64-50ff-4e82-bab9-9b70ea9a4224\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-59kkj"
Sep 09 21:16:53.988714 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:16:53.988698    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7600db64-50ff-4e82-bab9-9b70ea9a4224-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-59kkj\" (UID: \"7600db64-50ff-4e82-bab9-9b70ea9a4224\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-59kkj"
Sep 09 21:16:54.002819 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:16:54.002796    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mntw7\" (UniqueName: \"kubernetes.io/projected/7600db64-50ff-4e82-bab9-9b70ea9a4224-kube-api-access-mntw7\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-59kkj\" (UID: \"7600db64-50ff-4e82-bab9-9b70ea9a4224\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-59kkj"
Sep 09 21:16:54.084093 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:16:54.084022    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5bqktz"]
Sep 09 21:16:54.087552 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:16:54.087530    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5bqktz"
Sep 09 21:16:54.091711 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:16:54.091693    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-epp-sa-dockercfg-9nq4p\""
Sep 09 21:16:54.103517 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:16:54.103492    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5bqktz"]
Sep 09 21:16:54.132671 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:16:54.132643    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:16:54.134505 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:16:54.134489    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-59kkj"
Sep 09 21:16:54.188028 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:16:54.188000    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba38eab6-c4fb-4dc0-8d50-6991ba5aa7e2-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5bqktz\" (UID: \"ba38eab6-c4fb-4dc0-8d50-6991ba5aa7e2\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5bqktz"
Sep 09 21:16:54.188169 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:16:54.188077    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmqtk\" (UniqueName: \"kubernetes.io/projected/ba38eab6-c4fb-4dc0-8d50-6991ba5aa7e2-kube-api-access-bmqtk\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5bqktz\" (UID: \"ba38eab6-c4fb-4dc0-8d50-6991ba5aa7e2\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5bqktz"
Sep 09 21:16:54.289268 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:16:54.289239    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmqtk\" (UniqueName: \"kubernetes.io/projected/ba38eab6-c4fb-4dc0-8d50-6991ba5aa7e2-kube-api-access-bmqtk\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5bqktz\" (UID: \"ba38eab6-c4fb-4dc0-8d50-6991ba5aa7e2\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5bqktz"
Sep 09 21:16:54.289469 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:16:54.289295    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba38eab6-c4fb-4dc0-8d50-6991ba5aa7e2-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5bqktz\" (UID: \"ba38eab6-c4fb-4dc0-8d50-6991ba5aa7e2\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5bqktz"
Sep 09 21:16:54.291867 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:16:54.291845    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba38eab6-c4fb-4dc0-8d50-6991ba5aa7e2-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5bqktz\" (UID: \"ba38eab6-c4fb-4dc0-8d50-6991ba5aa7e2\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5bqktz"
Sep 09 21:16:54.300212 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:16:54.300188    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmqtk\" (UniqueName: \"kubernetes.io/projected/ba38eab6-c4fb-4dc0-8d50-6991ba5aa7e2-kube-api-access-bmqtk\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5bqktz\" (UID: \"ba38eab6-c4fb-4dc0-8d50-6991ba5aa7e2\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5bqktz"
Sep 09 21:16:54.397715 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:16:54.397673    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5bqktz"
Sep 09 21:16:54.412520 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:16:54.412497    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-59kkj"]
Sep 09 21:16:54.414187 ip-10-0-134-72 kubenswrapper[2576]: W0909 21:16:54.414156    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7600db64_50ff_4e82_bab9_9b70ea9a4224.slice/crio-e1bcd696caad4b19bc0473e341c09c6bac027dc8df30b5b99a27714af10a4e88 WatchSource:0}: Error finding container e1bcd696caad4b19bc0473e341c09c6bac027dc8df30b5b99a27714af10a4e88: Status 404 returned error can't find the container with id e1bcd696caad4b19bc0473e341c09c6bac027dc8df30b5b99a27714af10a4e88
Sep 09 21:16:54.606384 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:16:54.606336    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-59kkj" event={"ID":"7600db64-50ff-4e82-bab9-9b70ea9a4224","Type":"ContainerStarted","Data":"2f914e52deba53229d3ad8fa71e20c472773ceeb5fd3d66607147fb53d6c58fd"}
Sep 09 21:16:54.606576 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:16:54.606412    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-59kkj" event={"ID":"7600db64-50ff-4e82-bab9-9b70ea9a4224","Type":"ContainerStarted","Data":"e1bcd696caad4b19bc0473e341c09c6bac027dc8df30b5b99a27714af10a4e88"}
Sep 09 21:16:54.693753 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:16:54.693706    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-59kkj" podStartSLOduration=1.693689102 podStartE2EDuration="1.693689102s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:16:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:16:54.655986713 +0000 UTC m=+1457.769672572" watchObservedRunningTime="2026-09-09 21:16:54.693689102 +0000 UTC m=+1457.807374961"
Sep 09 21:16:54.694970 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:16:54.694950    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5bqktz"]
Sep 09 21:16:54.696433 ip-10-0-134-72 kubenswrapper[2576]: W0909 21:16:54.696404    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba38eab6_c4fb_4dc0_8d50_6991ba5aa7e2.slice/crio-c17c131eeb35fbd3c1f3f8255749d4e0b5bf0c93e4e1f2124124223f50250c12 WatchSource:0}: Error finding container c17c131eeb35fbd3c1f3f8255749d4e0b5bf0c93e4e1f2124124223f50250c12: Status 404 returned error can't find the container with id c17c131eeb35fbd3c1f3f8255749d4e0b5bf0c93e4e1f2124124223f50250c12
Sep 09 21:16:54.698442 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:16:54.698425    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 21:16:55.134984 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:16:55.134941    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-59kkj"
Sep 09 21:16:55.615054 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:16:55.615005    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5bqktz" event={"ID":"ba38eab6-c4fb-4dc0-8d50-6991ba5aa7e2","Type":"ContainerStarted","Data":"c17c131eeb35fbd3c1f3f8255749d4e0b5bf0c93e4e1f2124124223f50250c12"}
Sep 09 21:16:57.625345 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:16:57.625307    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:16:57.627702 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:16:57.627669    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5bqktz" event={"ID":"ba38eab6-c4fb-4dc0-8d50-6991ba5aa7e2","Type":"ContainerStarted","Data":"2a4389d055c6f9dc89c6bcc032aad25a41a0de4ea2834432dc570832061df927"}
Sep 09 21:16:57.627831 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:16:57.627757    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5bqktz"
Sep 09 21:16:57.649952 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:16:57.649902    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5bqktz" podStartSLOduration=1.192086234 podStartE2EDuration="3.649889419s" totalImagesPullingTime="2.457803185s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:16:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:16:57.647579046 +0000 UTC m=+1460.761264934" watchObservedRunningTime="2026-09-09 21:16:57.649889419 +0000 UTC m=+1460.763575312"
Sep 09 21:17:00.576962 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:17:00.576927    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:17:03.283879 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:17:03.283830    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:17:03.380148 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:17:03.380112    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:17:04.134946 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:17:04.134907    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-59kkj"
Sep 09 21:17:04.147614 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:17:04.147585    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-59kkj"
Sep 09 21:17:04.664181 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:17:04.664150    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:17:04.673899 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:17:04.673880    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-59kkj"
Sep 09 21:17:04.766992 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:17:04.766961    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:17:10.374279 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:17:10.374244    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:17:11.118958 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:17:11.118923    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:17:11.386948 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:17:11.386857    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:17:16.600006 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:17:16.599966    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:17:16.664860 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:17:16.664829    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:17:19.849347 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:17:19.849305    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:17:28.240454 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:17:28.240419    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:17:28.271247 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:17:28.271205    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5bqktz"]
Sep 09 21:17:28.278015 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:17:28.277976    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5bqktz" podUID="ba38eab6-c4fb-4dc0-8d50-6991ba5aa7e2" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:17:28.281491 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:17:28.281468    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-59kkj"]
Sep 09 21:17:28.289689 ip-10-0-134-72 kubenswrapper[2576]: E0909 21:17:28.289656    2576 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 09 21:17:28.289689 ip-10-0-134-72 kubenswrapper[2576]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 21:17:28.289689 ip-10-0-134-72 kubenswrapper[2576]:         , stdout: , stderr: , exit code -1
Sep 09 21:17:28.289689 ip-10-0-134-72 kubenswrapper[2576]:  > containerID="2f914e52deba53229d3ad8fa71e20c472773ceeb5fd3d66607147fb53d6c58fd" cmd=["/bin/sleep","15"]
Sep 09 21:17:28.289947 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:17:28.289704    2576 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-59kkj" message="" err=<
Sep 09 21:17:28.289947 ip-10-0-134-72 kubenswrapper[2576]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 21:17:28.289947 ip-10-0-134-72 kubenswrapper[2576]:         , stdout: , stderr: , exit code -1
Sep 09 21:17:28.289947 ip-10-0-134-72 kubenswrapper[2576]:  >
Sep 09 21:17:28.289947 ip-10-0-134-72 kubenswrapper[2576]: E0909 21:17:28.289717    2576 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 09 21:17:28.289947 ip-10-0-134-72 kubenswrapper[2576]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 21:17:28.289947 ip-10-0-134-72 kubenswrapper[2576]:         , stdout: , stderr: , exit code -1
Sep 09 21:17:28.289947 ip-10-0-134-72 kubenswrapper[2576]:  > pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-59kkj" podUID="7600db64-50ff-4e82-bab9-9b70ea9a4224" containerName="main" containerID="cri-o://2f914e52deba53229d3ad8fa71e20c472773ceeb5fd3d66607147fb53d6c58fd"
Sep 09 21:17:28.289947 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:17:28.289748    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-59kkj" podUID="7600db64-50ff-4e82-bab9-9b70ea9a4224" containerName="main" containerID="cri-o://2f914e52deba53229d3ad8fa71e20c472773ceeb5fd3d66607147fb53d6c58fd" gracePeriod=60
Sep 09 21:17:28.336287 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:17:28.336255    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:17:28.650134 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:17:28.650109    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-59kkj"
Sep 09 21:17:28.783996 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:17:28.783903    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7600db64-50ff-4e82-bab9-9b70ea9a4224-model-cache\") pod \"7600db64-50ff-4e82-bab9-9b70ea9a4224\" (UID: \"7600db64-50ff-4e82-bab9-9b70ea9a4224\") "
Sep 09 21:17:28.783996 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:17:28.783953    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7600db64-50ff-4e82-bab9-9b70ea9a4224-tmp-dir\") pod \"7600db64-50ff-4e82-bab9-9b70ea9a4224\" (UID: \"7600db64-50ff-4e82-bab9-9b70ea9a4224\") "
Sep 09 21:17:28.783996 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:17:28.783979    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mntw7\" (UniqueName: \"kubernetes.io/projected/7600db64-50ff-4e82-bab9-9b70ea9a4224-kube-api-access-mntw7\") pod \"7600db64-50ff-4e82-bab9-9b70ea9a4224\" (UID: \"7600db64-50ff-4e82-bab9-9b70ea9a4224\") "
Sep 09 21:17:28.784287 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:17:28.784026    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7600db64-50ff-4e82-bab9-9b70ea9a4224-tls-certs\") pod \"7600db64-50ff-4e82-bab9-9b70ea9a4224\" (UID: \"7600db64-50ff-4e82-bab9-9b70ea9a4224\") "
Sep 09 21:17:28.784287 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:17:28.784087    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7600db64-50ff-4e82-bab9-9b70ea9a4224-home\") pod \"7600db64-50ff-4e82-bab9-9b70ea9a4224\" (UID: \"7600db64-50ff-4e82-bab9-9b70ea9a4224\") "
Sep 09 21:17:28.784287 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:17:28.784133    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7600db64-50ff-4e82-bab9-9b70ea9a4224-dshm\") pod \"7600db64-50ff-4e82-bab9-9b70ea9a4224\" (UID: \"7600db64-50ff-4e82-bab9-9b70ea9a4224\") "
Sep 09 21:17:28.784287 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:17:28.784190    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7600db64-50ff-4e82-bab9-9b70ea9a4224-model-cache" (OuterVolumeSpecName: "model-cache") pod "7600db64-50ff-4e82-bab9-9b70ea9a4224" (UID: "7600db64-50ff-4e82-bab9-9b70ea9a4224"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:17:28.784287 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:17:28.784203    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7600db64-50ff-4e82-bab9-9b70ea9a4224-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7600db64-50ff-4e82-bab9-9b70ea9a4224" (UID: "7600db64-50ff-4e82-bab9-9b70ea9a4224"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:17:28.784578 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:17:28.784414    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7600db64-50ff-4e82-bab9-9b70ea9a4224-home" (OuterVolumeSpecName: "home") pod "7600db64-50ff-4e82-bab9-9b70ea9a4224" (UID: "7600db64-50ff-4e82-bab9-9b70ea9a4224"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:17:28.784578 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:17:28.784459    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7600db64-50ff-4e82-bab9-9b70ea9a4224-model-cache\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:17:28.784578 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:17:28.784478    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7600db64-50ff-4e82-bab9-9b70ea9a4224-tmp-dir\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:17:28.786627 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:17:28.786600    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7600db64-50ff-4e82-bab9-9b70ea9a4224-kube-api-access-mntw7" (OuterVolumeSpecName: "kube-api-access-mntw7") pod "7600db64-50ff-4e82-bab9-9b70ea9a4224" (UID: "7600db64-50ff-4e82-bab9-9b70ea9a4224"). InnerVolumeSpecName "kube-api-access-mntw7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:17:28.787326 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:17:28.787299    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7600db64-50ff-4e82-bab9-9b70ea9a4224-dshm" (OuterVolumeSpecName: "dshm") pod "7600db64-50ff-4e82-bab9-9b70ea9a4224" (UID: "7600db64-50ff-4e82-bab9-9b70ea9a4224"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:17:28.787326 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:17:28.787302    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7600db64-50ff-4e82-bab9-9b70ea9a4224-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7600db64-50ff-4e82-bab9-9b70ea9a4224" (UID: "7600db64-50ff-4e82-bab9-9b70ea9a4224"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:17:28.790598 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:17:28.790571    2576 generic.go:356] "Generic (PLEG): container finished" podID="7600db64-50ff-4e82-bab9-9b70ea9a4224" containerID="2f914e52deba53229d3ad8fa71e20c472773ceeb5fd3d66607147fb53d6c58fd" exitCode=0
Sep 09 21:17:28.790700 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:17:28.790603    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-59kkj" event={"ID":"7600db64-50ff-4e82-bab9-9b70ea9a4224","Type":"ContainerDied","Data":"2f914e52deba53229d3ad8fa71e20c472773ceeb5fd3d66607147fb53d6c58fd"}
Sep 09 21:17:28.790700 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:17:28.790620    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-59kkj"
Sep 09 21:17:28.790700 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:17:28.790659    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-59kkj" event={"ID":"7600db64-50ff-4e82-bab9-9b70ea9a4224","Type":"ContainerDied","Data":"e1bcd696caad4b19bc0473e341c09c6bac027dc8df30b5b99a27714af10a4e88"}
Sep 09 21:17:28.790700 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:17:28.790686    2576 scope.go:117] "RemoveContainer" containerID="2f914e52deba53229d3ad8fa71e20c472773ceeb5fd3d66607147fb53d6c58fd"
Sep 09 21:17:28.811217 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:17:28.811196    2576 scope.go:117] "RemoveContainer" containerID="2f914e52deba53229d3ad8fa71e20c472773ceeb5fd3d66607147fb53d6c58fd"
Sep 09 21:17:28.811504 ip-10-0-134-72 kubenswrapper[2576]: E0909 21:17:28.811486    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f914e52deba53229d3ad8fa71e20c472773ceeb5fd3d66607147fb53d6c58fd\": container with ID starting with 2f914e52deba53229d3ad8fa71e20c472773ceeb5fd3d66607147fb53d6c58fd not found: ID does not exist" containerID="2f914e52deba53229d3ad8fa71e20c472773ceeb5fd3d66607147fb53d6c58fd"
Sep 09 21:17:28.811568 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:17:28.811514    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f914e52deba53229d3ad8fa71e20c472773ceeb5fd3d66607147fb53d6c58fd"} err="failed to get container status \"2f914e52deba53229d3ad8fa71e20c472773ceeb5fd3d66607147fb53d6c58fd\": rpc error: code = NotFound desc = could not find container \"2f914e52deba53229d3ad8fa71e20c472773ceeb5fd3d66607147fb53d6c58fd\": container with ID starting with 2f914e52deba53229d3ad8fa71e20c472773ceeb5fd3d66607147fb53d6c58fd not found: ID does not exist"
Sep 09 21:17:28.838215 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:17:28.838186    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-59kkj"]
Sep 09 21:17:28.851287 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:17:28.851260    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-59kkj"]
Sep 09 21:17:28.885104 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:17:28.885076    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mntw7\" (UniqueName: \"kubernetes.io/projected/7600db64-50ff-4e82-bab9-9b70ea9a4224-kube-api-access-mntw7\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:17:28.885104 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:17:28.885103    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7600db64-50ff-4e82-bab9-9b70ea9a4224-tls-certs\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:17:28.885266 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:17:28.885117    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7600db64-50ff-4e82-bab9-9b70ea9a4224-home\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:17:28.885266 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:17:28.885125    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7600db64-50ff-4e82-bab9-9b70ea9a4224-dshm\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:17:29.409460 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:17:29.409426    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7600db64-50ff-4e82-bab9-9b70ea9a4224" path="/var/lib/kubelet/pods/7600db64-50ff-4e82-bab9-9b70ea9a4224/volumes"
Sep 09 21:17:38.273330 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:17:38.273290    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5bqktz" podUID="ba38eab6-c4fb-4dc0-8d50-6991ba5aa7e2" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:17:43.298404 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:17:43.298346    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5bqktz" podUID="ba38eab6-c4fb-4dc0-8d50-6991ba5aa7e2" containerName="main" containerID="cri-o://2a4389d055c6f9dc89c6bcc032aad25a41a0de4ea2834432dc570832061df927" gracePeriod=45
Sep 09 21:17:48.273522 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:17:48.273485    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5bqktz" podUID="ba38eab6-c4fb-4dc0-8d50-6991ba5aa7e2" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:17:58.273745 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:17:58.273709    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5bqktz" podUID="ba38eab6-c4fb-4dc0-8d50-6991ba5aa7e2" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:18:08.273783 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:18:08.273743    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5bqktz" podUID="ba38eab6-c4fb-4dc0-8d50-6991ba5aa7e2" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:18:13.538947 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:18:13.538921    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5bqktz"
Sep 09 21:18:13.642305 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:18:13.642275    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bmqtk\" (UniqueName: \"kubernetes.io/projected/ba38eab6-c4fb-4dc0-8d50-6991ba5aa7e2-kube-api-access-bmqtk\") pod \"ba38eab6-c4fb-4dc0-8d50-6991ba5aa7e2\" (UID: \"ba38eab6-c4fb-4dc0-8d50-6991ba5aa7e2\") "
Sep 09 21:18:13.642469 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:18:13.642389    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba38eab6-c4fb-4dc0-8d50-6991ba5aa7e2-tls-certs\") pod \"ba38eab6-c4fb-4dc0-8d50-6991ba5aa7e2\" (UID: \"ba38eab6-c4fb-4dc0-8d50-6991ba5aa7e2\") "
Sep 09 21:18:13.644588 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:18:13.644559    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba38eab6-c4fb-4dc0-8d50-6991ba5aa7e2-kube-api-access-bmqtk" (OuterVolumeSpecName: "kube-api-access-bmqtk") pod "ba38eab6-c4fb-4dc0-8d50-6991ba5aa7e2" (UID: "ba38eab6-c4fb-4dc0-8d50-6991ba5aa7e2"). InnerVolumeSpecName "kube-api-access-bmqtk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:18:13.644698 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:18:13.644599    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba38eab6-c4fb-4dc0-8d50-6991ba5aa7e2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ba38eab6-c4fb-4dc0-8d50-6991ba5aa7e2" (UID: "ba38eab6-c4fb-4dc0-8d50-6991ba5aa7e2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:18:13.743187 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:18:13.743162    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba38eab6-c4fb-4dc0-8d50-6991ba5aa7e2-tls-certs\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:18:13.743187 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:18:13.743183    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmqtk\" (UniqueName: \"kubernetes.io/projected/ba38eab6-c4fb-4dc0-8d50-6991ba5aa7e2-kube-api-access-bmqtk\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:18:14.013582 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:18:14.013492    2576 generic.go:356] "Generic (PLEG): container finished" podID="ba38eab6-c4fb-4dc0-8d50-6991ba5aa7e2" containerID="2a4389d055c6f9dc89c6bcc032aad25a41a0de4ea2834432dc570832061df927" exitCode=0
Sep 09 21:18:14.013582 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:18:14.013546    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5bqktz"
Sep 09 21:18:14.013800 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:18:14.013581    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5bqktz" event={"ID":"ba38eab6-c4fb-4dc0-8d50-6991ba5aa7e2","Type":"ContainerDied","Data":"2a4389d055c6f9dc89c6bcc032aad25a41a0de4ea2834432dc570832061df927"}
Sep 09 21:18:14.013800 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:18:14.013637    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5bqktz" event={"ID":"ba38eab6-c4fb-4dc0-8d50-6991ba5aa7e2","Type":"ContainerDied","Data":"c17c131eeb35fbd3c1f3f8255749d4e0b5bf0c93e4e1f2124124223f50250c12"}
Sep 09 21:18:14.013800 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:18:14.013658    2576 scope.go:117] "RemoveContainer" containerID="2a4389d055c6f9dc89c6bcc032aad25a41a0de4ea2834432dc570832061df927"
Sep 09 21:18:14.023114 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:18:14.023098    2576 scope.go:117] "RemoveContainer" containerID="2a4389d055c6f9dc89c6bcc032aad25a41a0de4ea2834432dc570832061df927"
Sep 09 21:18:14.023400 ip-10-0-134-72 kubenswrapper[2576]: E0909 21:18:14.023353    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a4389d055c6f9dc89c6bcc032aad25a41a0de4ea2834432dc570832061df927\": container with ID starting with 2a4389d055c6f9dc89c6bcc032aad25a41a0de4ea2834432dc570832061df927 not found: ID does not exist" containerID="2a4389d055c6f9dc89c6bcc032aad25a41a0de4ea2834432dc570832061df927"
Sep 09 21:18:14.023446 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:18:14.023410    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a4389d055c6f9dc89c6bcc032aad25a41a0de4ea2834432dc570832061df927"} err="failed to get container status \"2a4389d055c6f9dc89c6bcc032aad25a41a0de4ea2834432dc570832061df927\": rpc error: code = NotFound desc = could not find container \"2a4389d055c6f9dc89c6bcc032aad25a41a0de4ea2834432dc570832061df927\": container with ID starting with 2a4389d055c6f9dc89c6bcc032aad25a41a0de4ea2834432dc570832061df927 not found: ID does not exist"
Sep 09 21:18:14.039282 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:18:14.039258    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5bqktz"]
Sep 09 21:18:14.043230 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:18:14.043211    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5bqktz"]
Sep 09 21:18:15.409804 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:18:15.409758    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ba38eab6-c4fb-4dc0-8d50-6991ba5aa7e2" path="/var/lib/kubelet/pods/ba38eab6-c4fb-4dc0-8d50-6991ba5aa7e2/volumes"
Sep 09 21:18:18.403765 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:18:18.403728    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:18:18.559356 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:18:18.559298    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:18:27.113230 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:18:27.113197    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:18:32.526071 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:18:32.526030    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:18:32.662140 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:18:32.662098    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:18:40.142258 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:18:40.142227    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-k4n42"]
Sep 09 21:18:40.142616 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:18:40.142555    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba38eab6-c4fb-4dc0-8d50-6991ba5aa7e2" containerName="main"
Sep 09 21:18:40.142616 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:18:40.142566    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba38eab6-c4fb-4dc0-8d50-6991ba5aa7e2" containerName="main"
Sep 09 21:18:40.142616 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:18:40.142582    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7600db64-50ff-4e82-bab9-9b70ea9a4224" containerName="main"
Sep 09 21:18:40.142616 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:18:40.142587    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="7600db64-50ff-4e82-bab9-9b70ea9a4224" containerName="main"
Sep 09 21:18:40.142748 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:18:40.142648    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba38eab6-c4fb-4dc0-8d50-6991ba5aa7e2" containerName="main"
Sep 09 21:18:40.142748 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:18:40.142660    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="7600db64-50ff-4e82-bab9-9b70ea9a4224" containerName="main"
Sep 09 21:18:40.146620 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:18:40.146601    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-k4n42"
Sep 09 21:18:40.155897 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:18:40.155877    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"scheduler-custom-template-test-kserve-self-signed-certs\""
Sep 09 21:18:40.155897 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:18:40.155888    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"openshift-service-ca.crt\""
Sep 09 21:18:40.156056 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:18:40.155879    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"default-dockercfg-r8ssr\""
Sep 09 21:18:40.156056 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:18:40.155924    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"kube-root-ca.crt\""
Sep 09 21:18:40.162667 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:18:40.162647    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82024824-18cb-4440-9874-4e1bda9127b7-tls-certs\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-k4n42\" (UID: \"82024824-18cb-4440-9874-4e1bda9127b7\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-k4n42"
Sep 09 21:18:40.162721 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:18:40.162679    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82024824-18cb-4440-9874-4e1bda9127b7-model-cache\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-k4n42\" (UID: \"82024824-18cb-4440-9874-4e1bda9127b7\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-k4n42"
Sep 09 21:18:40.162754 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:18:40.162737    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82024824-18cb-4440-9874-4e1bda9127b7-dshm\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-k4n42\" (UID: \"82024824-18cb-4440-9874-4e1bda9127b7\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-k4n42"
Sep 09 21:18:40.162791 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:18:40.162754    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82024824-18cb-4440-9874-4e1bda9127b7-home\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-k4n42\" (UID: \"82024824-18cb-4440-9874-4e1bda9127b7\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-k4n42"
Sep 09 21:18:40.162791 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:18:40.162768    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82024824-18cb-4440-9874-4e1bda9127b7-tmp-dir\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-k4n42\" (UID: \"82024824-18cb-4440-9874-4e1bda9127b7\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-k4n42"
Sep 09 21:18:40.162858 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:18:40.162830    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gftgk\" (UniqueName: \"kubernetes.io/projected/82024824-18cb-4440-9874-4e1bda9127b7-kube-api-access-gftgk\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-k4n42\" (UID: \"82024824-18cb-4440-9874-4e1bda9127b7\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-k4n42"
Sep 09 21:18:40.179164 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:18:40.179141    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-k4n42"]
Sep 09 21:18:40.198754 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:18:40.198721    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:18:40.264079 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:18:40.264039    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gftgk\" (UniqueName: \"kubernetes.io/projected/82024824-18cb-4440-9874-4e1bda9127b7-kube-api-access-gftgk\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-k4n42\" (UID: \"82024824-18cb-4440-9874-4e1bda9127b7\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-k4n42"
Sep 09 21:18:40.264273 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:18:40.264087    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82024824-18cb-4440-9874-4e1bda9127b7-tls-certs\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-k4n42\" (UID: \"82024824-18cb-4440-9874-4e1bda9127b7\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-k4n42"
Sep 09 21:18:40.264273 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:18:40.264124    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82024824-18cb-4440-9874-4e1bda9127b7-model-cache\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-k4n42\" (UID: \"82024824-18cb-4440-9874-4e1bda9127b7\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-k4n42"
Sep 09 21:18:40.264273 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:18:40.264173    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82024824-18cb-4440-9874-4e1bda9127b7-dshm\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-k4n42\" (UID: \"82024824-18cb-4440-9874-4e1bda9127b7\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-k4n42"
Sep 09 21:18:40.264273 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:18:40.264190    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82024824-18cb-4440-9874-4e1bda9127b7-home\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-k4n42\" (UID: \"82024824-18cb-4440-9874-4e1bda9127b7\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-k4n42"
Sep 09 21:18:40.264273 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:18:40.264207    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82024824-18cb-4440-9874-4e1bda9127b7-tmp-dir\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-k4n42\" (UID: \"82024824-18cb-4440-9874-4e1bda9127b7\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-k4n42"
Sep 09 21:18:40.264655 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:18:40.264629    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82024824-18cb-4440-9874-4e1bda9127b7-model-cache\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-k4n42\" (UID: \"82024824-18cb-4440-9874-4e1bda9127b7\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-k4n42"
Sep 09 21:18:40.264714 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:18:40.264664    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82024824-18cb-4440-9874-4e1bda9127b7-tmp-dir\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-k4n42\" (UID: \"82024824-18cb-4440-9874-4e1bda9127b7\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-k4n42"
Sep 09 21:18:40.264714 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:18:40.264687    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82024824-18cb-4440-9874-4e1bda9127b7-home\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-k4n42\" (UID: \"82024824-18cb-4440-9874-4e1bda9127b7\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-k4n42"
Sep 09 21:18:40.266816 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:18:40.266787    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82024824-18cb-4440-9874-4e1bda9127b7-dshm\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-k4n42\" (UID: \"82024824-18cb-4440-9874-4e1bda9127b7\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-k4n42"
Sep 09 21:18:40.267054 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:18:40.267037    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82024824-18cb-4440-9874-4e1bda9127b7-tls-certs\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-k4n42\" (UID: \"82024824-18cb-4440-9874-4e1bda9127b7\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-k4n42"
Sep 09 21:18:40.279832 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:18:40.279804    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gftgk\" (UniqueName: \"kubernetes.io/projected/82024824-18cb-4440-9874-4e1bda9127b7-kube-api-access-gftgk\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-k4n42\" (UID: \"82024824-18cb-4440-9874-4e1bda9127b7\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-k4n42"
Sep 09 21:18:40.456209 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:18:40.456127    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-k4n42"
Sep 09 21:18:40.511672 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:18:40.511641    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffckjbcn"]
Sep 09 21:18:40.515864 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:18:40.515839    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffckjbcn"
Sep 09 21:18:40.519605 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:18:40.519508    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"scheduler-custom-template-test-epp-sa-dockercfg-vhrrn\""
Sep 09 21:18:40.544630 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:18:40.544602    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffckjbcn"]
Sep 09 21:18:40.566107 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:18:40.566081    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/af9f957f-1981-4373-83ec-66663290993c-tokenizer-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffckjbcn\" (UID: \"af9f957f-1981-4373-83ec-66663290993c\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffckjbcn"
Sep 09 21:18:40.566239 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:18:40.566138    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bk4q\" (UniqueName: \"kubernetes.io/projected/af9f957f-1981-4373-83ec-66663290993c-kube-api-access-2bk4q\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffckjbcn\" (UID: \"af9f957f-1981-4373-83ec-66663290993c\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffckjbcn"
Sep 09 21:18:40.566239 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:18:40.566177    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af9f957f-1981-4373-83ec-66663290993c-tls-certs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffckjbcn\" (UID: \"af9f957f-1981-4373-83ec-66663290993c\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffckjbcn"
Sep 09 21:18:40.566387 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:18:40.566241    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/af9f957f-1981-4373-83ec-66663290993c-cachi2-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffckjbcn\" (UID: \"af9f957f-1981-4373-83ec-66663290993c\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffckjbcn"
Sep 09 21:18:40.566447 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:18:40.566400    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:18:40.667265 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:18:40.667232    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/af9f957f-1981-4373-83ec-66663290993c-tokenizer-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffckjbcn\" (UID: \"af9f957f-1981-4373-83ec-66663290993c\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffckjbcn"
Sep 09 21:18:40.667461 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:18:40.667274    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bk4q\" (UniqueName: \"kubernetes.io/projected/af9f957f-1981-4373-83ec-66663290993c-kube-api-access-2bk4q\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffckjbcn\" (UID: \"af9f957f-1981-4373-83ec-66663290993c\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffckjbcn"
Sep 09 21:18:40.667461 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:18:40.667299    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af9f957f-1981-4373-83ec-66663290993c-tls-certs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffckjbcn\" (UID: \"af9f957f-1981-4373-83ec-66663290993c\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffckjbcn"
Sep 09 21:18:40.667461 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:18:40.667344    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/af9f957f-1981-4373-83ec-66663290993c-cachi2-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffckjbcn\" (UID: \"af9f957f-1981-4373-83ec-66663290993c\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffckjbcn"
Sep 09 21:18:40.667665 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:18:40.667648    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/af9f957f-1981-4373-83ec-66663290993c-tokenizer-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffckjbcn\" (UID: \"af9f957f-1981-4373-83ec-66663290993c\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffckjbcn"
Sep 09 21:18:40.667741 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:18:40.667722    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/af9f957f-1981-4373-83ec-66663290993c-cachi2-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffckjbcn\" (UID: \"af9f957f-1981-4373-83ec-66663290993c\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffckjbcn"
Sep 09 21:18:40.669950 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:18:40.669931    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af9f957f-1981-4373-83ec-66663290993c-tls-certs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffckjbcn\" (UID: \"af9f957f-1981-4373-83ec-66663290993c\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffckjbcn"
Sep 09 21:18:40.677642 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:18:40.677618    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bk4q\" (UniqueName: \"kubernetes.io/projected/af9f957f-1981-4373-83ec-66663290993c-kube-api-access-2bk4q\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffckjbcn\" (UID: \"af9f957f-1981-4373-83ec-66663290993c\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffckjbcn"
Sep 09 21:18:40.742901 ip-10-0-134-72 kubenswrapper[2576]: W0909 21:18:40.742871    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82024824_18cb_4440_9874_4e1bda9127b7.slice/crio-6faed1b1b892a8f6624c72d71a37f0286bf5c923fc1ecb8a719c40608704eac5 WatchSource:0}: Error finding container 6faed1b1b892a8f6624c72d71a37f0286bf5c923fc1ecb8a719c40608704eac5: Status 404 returned error can't find the container with id 6faed1b1b892a8f6624c72d71a37f0286bf5c923fc1ecb8a719c40608704eac5
Sep 09 21:18:40.743015 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:18:40.742907    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-k4n42"]
Sep 09 21:18:40.828532 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:18:40.828501    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffckjbcn"
Sep 09 21:18:41.107151 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:18:41.107121    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffckjbcn"]
Sep 09 21:18:41.107774 ip-10-0-134-72 kubenswrapper[2576]: W0909 21:18:41.107749    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf9f957f_1981_4373_83ec_66663290993c.slice/crio-0ef01c5ae663a94f593e8b38636e9ac0d016a6148ac955e341676befe5f14e95 WatchSource:0}: Error finding container 0ef01c5ae663a94f593e8b38636e9ac0d016a6148ac955e341676befe5f14e95: Status 404 returned error can't find the container with id 0ef01c5ae663a94f593e8b38636e9ac0d016a6148ac955e341676befe5f14e95
Sep 09 21:18:41.154546 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:18:41.154515    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffckjbcn" event={"ID":"af9f957f-1981-4373-83ec-66663290993c","Type":"ContainerStarted","Data":"0ef01c5ae663a94f593e8b38636e9ac0d016a6148ac955e341676befe5f14e95"}
Sep 09 21:18:41.156034 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:18:41.156001    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-k4n42" event={"ID":"82024824-18cb-4440-9874-4e1bda9127b7","Type":"ContainerStarted","Data":"186a093b09569d8f31f2e22adc7f823f5d2f31bf135f6626b6ef91977bab7a4c"}
Sep 09 21:18:41.156127 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:18:41.156059    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-k4n42" event={"ID":"82024824-18cb-4440-9874-4e1bda9127b7","Type":"ContainerStarted","Data":"6faed1b1b892a8f6624c72d71a37f0286bf5c923fc1ecb8a719c40608704eac5"}
Sep 09 21:18:41.197105 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:18:41.197048    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-k4n42" podStartSLOduration=1.19702561 podStartE2EDuration="1.19702561s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:18:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:18:41.196487406 +0000 UTC m=+1564.310173265" watchObservedRunningTime="2026-09-09 21:18:41.19702561 +0000 UTC m=+1564.310711471"
Sep 09 21:18:41.456696 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:18:41.456610    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-k4n42"
Sep 09 21:18:42.162126 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:18:42.162082    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffckjbcn" event={"ID":"af9f957f-1981-4373-83ec-66663290993c","Type":"ContainerStarted","Data":"379d88d287b7bc4ded9a96e43f8e4fa6f9b74d71da41fc5d6a64340fe92cb4e0"}
Sep 09 21:18:42.162533 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:18:42.162300    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffckjbcn"
Sep 09 21:18:42.190648 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:18:42.190604    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffckjbcn" podStartSLOduration=2.190587841 podStartE2EDuration="2.190587841s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:18:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:18:42.189632226 +0000 UTC m=+1565.303318084" watchObservedRunningTime="2026-09-09 21:18:42.190587841 +0000 UTC m=+1565.304273699"
Sep 09 21:18:50.456712 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:18:50.456674    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-k4n42"
Sep 09 21:18:50.469763 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:18:50.469739    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-k4n42"
Sep 09 21:18:51.219939 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:18:51.219903    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-k4n42"
Sep 09 21:18:53.040169 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:18:53.040124    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:19:13.169537 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:19:13.169507    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffckjbcn"
Sep 09 21:19:14.703749 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:19:14.703714    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-k4n42"]
Sep 09 21:19:14.711875 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:19:14.711846    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffckjbcn"]
Sep 09 21:19:14.713841 ip-10-0-134-72 kubenswrapper[2576]: E0909 21:19:14.713731    2576 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 09 21:19:14.713841 ip-10-0-134-72 kubenswrapper[2576]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 21:19:14.713841 ip-10-0-134-72 kubenswrapper[2576]:         , stdout: , stderr: , exit code -1
Sep 09 21:19:14.713841 ip-10-0-134-72 kubenswrapper[2576]:  > containerID="186a093b09569d8f31f2e22adc7f823f5d2f31bf135f6626b6ef91977bab7a4c" cmd=["/bin/sleep","15"]
Sep 09 21:19:14.713841 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:19:14.713787    2576 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-k4n42" message="" err=<
Sep 09 21:19:14.713841 ip-10-0-134-72 kubenswrapper[2576]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 21:19:14.713841 ip-10-0-134-72 kubenswrapper[2576]:         , stdout: , stderr: , exit code -1
Sep 09 21:19:14.713841 ip-10-0-134-72 kubenswrapper[2576]:  >
Sep 09 21:19:14.713841 ip-10-0-134-72 kubenswrapper[2576]: E0909 21:19:14.713803    2576 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 09 21:19:14.713841 ip-10-0-134-72 kubenswrapper[2576]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 21:19:14.713841 ip-10-0-134-72 kubenswrapper[2576]:         , stdout: , stderr: , exit code -1
Sep 09 21:19:14.713841 ip-10-0-134-72 kubenswrapper[2576]:  > pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-k4n42" podUID="82024824-18cb-4440-9874-4e1bda9127b7" containerName="main" containerID="cri-o://186a093b09569d8f31f2e22adc7f823f5d2f31bf135f6626b6ef91977bab7a4c"
Sep 09 21:19:14.713841 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:19:14.713841    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-k4n42" podUID="82024824-18cb-4440-9874-4e1bda9127b7" containerName="main" containerID="cri-o://186a093b09569d8f31f2e22adc7f823f5d2f31bf135f6626b6ef91977bab7a4c" gracePeriod=60
Sep 09 21:19:14.720918 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:19:14.720669    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:19:14.789428 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:19:14.789385    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:19:15.070072 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:19:15.070048    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-k4n42"
Sep 09 21:19:15.179520 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:19:15.179474    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82024824-18cb-4440-9874-4e1bda9127b7-dshm\") pod \"82024824-18cb-4440-9874-4e1bda9127b7\" (UID: \"82024824-18cb-4440-9874-4e1bda9127b7\") "
Sep 09 21:19:15.179680 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:19:15.179537    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82024824-18cb-4440-9874-4e1bda9127b7-model-cache\") pod \"82024824-18cb-4440-9874-4e1bda9127b7\" (UID: \"82024824-18cb-4440-9874-4e1bda9127b7\") "
Sep 09 21:19:15.179680 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:19:15.179588    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82024824-18cb-4440-9874-4e1bda9127b7-home\") pod \"82024824-18cb-4440-9874-4e1bda9127b7\" (UID: \"82024824-18cb-4440-9874-4e1bda9127b7\") "
Sep 09 21:19:15.179680 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:19:15.179635    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82024824-18cb-4440-9874-4e1bda9127b7-tmp-dir\") pod \"82024824-18cb-4440-9874-4e1bda9127b7\" (UID: \"82024824-18cb-4440-9874-4e1bda9127b7\") "
Sep 09 21:19:15.179680 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:19:15.179663    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gftgk\" (UniqueName: \"kubernetes.io/projected/82024824-18cb-4440-9874-4e1bda9127b7-kube-api-access-gftgk\") pod \"82024824-18cb-4440-9874-4e1bda9127b7\" (UID: \"82024824-18cb-4440-9874-4e1bda9127b7\") "
Sep 09 21:19:15.179910 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:19:15.179730    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82024824-18cb-4440-9874-4e1bda9127b7-tls-certs\") pod \"82024824-18cb-4440-9874-4e1bda9127b7\" (UID: \"82024824-18cb-4440-9874-4e1bda9127b7\") "
Sep 09 21:19:15.179910 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:19:15.179818    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82024824-18cb-4440-9874-4e1bda9127b7-model-cache" (OuterVolumeSpecName: "model-cache") pod "82024824-18cb-4440-9874-4e1bda9127b7" (UID: "82024824-18cb-4440-9874-4e1bda9127b7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:19:15.180027 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:19:15.179934    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82024824-18cb-4440-9874-4e1bda9127b7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "82024824-18cb-4440-9874-4e1bda9127b7" (UID: "82024824-18cb-4440-9874-4e1bda9127b7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:19:15.180027 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:19:15.179988    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82024824-18cb-4440-9874-4e1bda9127b7-model-cache\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:19:15.180027 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:19:15.180014    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82024824-18cb-4440-9874-4e1bda9127b7-home" (OuterVolumeSpecName: "home") pod "82024824-18cb-4440-9874-4e1bda9127b7" (UID: "82024824-18cb-4440-9874-4e1bda9127b7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:19:15.181926 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:19:15.181904    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82024824-18cb-4440-9874-4e1bda9127b7-dshm" (OuterVolumeSpecName: "dshm") pod "82024824-18cb-4440-9874-4e1bda9127b7" (UID: "82024824-18cb-4440-9874-4e1bda9127b7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:19:15.182025 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:19:15.182002    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82024824-18cb-4440-9874-4e1bda9127b7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "82024824-18cb-4440-9874-4e1bda9127b7" (UID: "82024824-18cb-4440-9874-4e1bda9127b7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:19:15.182402 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:19:15.182357    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82024824-18cb-4440-9874-4e1bda9127b7-kube-api-access-gftgk" (OuterVolumeSpecName: "kube-api-access-gftgk") pod "82024824-18cb-4440-9874-4e1bda9127b7" (UID: "82024824-18cb-4440-9874-4e1bda9127b7"). InnerVolumeSpecName "kube-api-access-gftgk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:19:15.281159 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:19:15.281088    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82024824-18cb-4440-9874-4e1bda9127b7-tmp-dir\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:19:15.281159 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:19:15.281115    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gftgk\" (UniqueName: \"kubernetes.io/projected/82024824-18cb-4440-9874-4e1bda9127b7-kube-api-access-gftgk\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:19:15.281159 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:19:15.281125    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82024824-18cb-4440-9874-4e1bda9127b7-tls-certs\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:19:15.281159 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:19:15.281134    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82024824-18cb-4440-9874-4e1bda9127b7-dshm\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:19:15.281159 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:19:15.281141    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82024824-18cb-4440-9874-4e1bda9127b7-home\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:19:15.334600 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:19:15.334566    2576 generic.go:356] "Generic (PLEG): container finished" podID="82024824-18cb-4440-9874-4e1bda9127b7" containerID="186a093b09569d8f31f2e22adc7f823f5d2f31bf135f6626b6ef91977bab7a4c" exitCode=0
Sep 09 21:19:15.334740 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:19:15.334623    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-k4n42"
Sep 09 21:19:15.334740 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:19:15.334653    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-k4n42" event={"ID":"82024824-18cb-4440-9874-4e1bda9127b7","Type":"ContainerDied","Data":"186a093b09569d8f31f2e22adc7f823f5d2f31bf135f6626b6ef91977bab7a4c"}
Sep 09 21:19:15.334740 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:19:15.334725    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-k4n42" event={"ID":"82024824-18cb-4440-9874-4e1bda9127b7","Type":"ContainerDied","Data":"6faed1b1b892a8f6624c72d71a37f0286bf5c923fc1ecb8a719c40608704eac5"}
Sep 09 21:19:15.334844 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:19:15.334746    2576 scope.go:117] "RemoveContainer" containerID="186a093b09569d8f31f2e22adc7f823f5d2f31bf135f6626b6ef91977bab7a4c"
Sep 09 21:19:15.344347 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:19:15.344326    2576 scope.go:117] "RemoveContainer" containerID="186a093b09569d8f31f2e22adc7f823f5d2f31bf135f6626b6ef91977bab7a4c"
Sep 09 21:19:15.344695 ip-10-0-134-72 kubenswrapper[2576]: E0909 21:19:15.344678    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"186a093b09569d8f31f2e22adc7f823f5d2f31bf135f6626b6ef91977bab7a4c\": container with ID starting with 186a093b09569d8f31f2e22adc7f823f5d2f31bf135f6626b6ef91977bab7a4c not found: ID does not exist" containerID="186a093b09569d8f31f2e22adc7f823f5d2f31bf135f6626b6ef91977bab7a4c"
Sep 09 21:19:15.344764 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:19:15.344702    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"186a093b09569d8f31f2e22adc7f823f5d2f31bf135f6626b6ef91977bab7a4c"} err="failed to get container status \"186a093b09569d8f31f2e22adc7f823f5d2f31bf135f6626b6ef91977bab7a4c\": rpc error: code = NotFound desc = could not find container \"186a093b09569d8f31f2e22adc7f823f5d2f31bf135f6626b6ef91977bab7a4c\": container with ID starting with 186a093b09569d8f31f2e22adc7f823f5d2f31bf135f6626b6ef91977bab7a4c not found: ID does not exist"
Sep 09 21:19:15.360842 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:19:15.360819    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-k4n42"]
Sep 09 21:19:15.369248 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:19:15.369230    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-k4n42"]
Sep 09 21:19:15.409817 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:19:15.409786    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="82024824-18cb-4440-9874-4e1bda9127b7" path="/var/lib/kubelet/pods/82024824-18cb-4440-9874-4e1bda9127b7/volumes"
Sep 09 21:19:16.637350 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:19:16.637320    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:19:16.725405 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:19:16.725298    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:19:23.168917 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:19:23.168878    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffckjbcn" podUID="af9f957f-1981-4373-83ec-66663290993c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:19:29.738311 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:19:29.738201    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffckjbcn" podUID="af9f957f-1981-4373-83ec-66663290993c" containerName="main" containerID="cri-o://379d88d287b7bc4ded9a96e43f8e4fa6f9b74d71da41fc5d6a64340fe92cb4e0" gracePeriod=45
Sep 09 21:19:33.169592 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:19:33.169551    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffckjbcn" podUID="af9f957f-1981-4373-83ec-66663290993c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:19:43.168710 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:19:43.168673    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffckjbcn" podUID="af9f957f-1981-4373-83ec-66663290993c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:19:43.169097 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:19:43.168786    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffckjbcn"
Sep 09 21:19:53.168681 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:19:53.168643    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffckjbcn" podUID="af9f957f-1981-4373-83ec-66663290993c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:19:53.997898 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:19:53.997862    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:19:54.099697 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:19:54.099656    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:19:59.979212 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:19:59.979182    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffckjbcn"
Sep 09 21:20:00.037999 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:20:00.037902    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af9f957f-1981-4373-83ec-66663290993c-tls-certs\") pod \"af9f957f-1981-4373-83ec-66663290993c\" (UID: \"af9f957f-1981-4373-83ec-66663290993c\") "
Sep 09 21:20:00.037999 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:20:00.037967    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/af9f957f-1981-4373-83ec-66663290993c-cachi2-cache\") pod \"af9f957f-1981-4373-83ec-66663290993c\" (UID: \"af9f957f-1981-4373-83ec-66663290993c\") "
Sep 09 21:20:00.038235 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:20:00.038017    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/af9f957f-1981-4373-83ec-66663290993c-tokenizer-cache\") pod \"af9f957f-1981-4373-83ec-66663290993c\" (UID: \"af9f957f-1981-4373-83ec-66663290993c\") "
Sep 09 21:20:00.038235 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:20:00.038039    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2bk4q\" (UniqueName: \"kubernetes.io/projected/af9f957f-1981-4373-83ec-66663290993c-kube-api-access-2bk4q\") pod \"af9f957f-1981-4373-83ec-66663290993c\" (UID: \"af9f957f-1981-4373-83ec-66663290993c\") "
Sep 09 21:20:00.038337 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:20:00.038278    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af9f957f-1981-4373-83ec-66663290993c-cachi2-cache" (OuterVolumeSpecName: "cachi2-cache") pod "af9f957f-1981-4373-83ec-66663290993c" (UID: "af9f957f-1981-4373-83ec-66663290993c"). InnerVolumeSpecName "cachi2-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:20:00.038337 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:20:00.038305    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af9f957f-1981-4373-83ec-66663290993c-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "af9f957f-1981-4373-83ec-66663290993c" (UID: "af9f957f-1981-4373-83ec-66663290993c"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:20:00.040317 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:20:00.040290    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af9f957f-1981-4373-83ec-66663290993c-kube-api-access-2bk4q" (OuterVolumeSpecName: "kube-api-access-2bk4q") pod "af9f957f-1981-4373-83ec-66663290993c" (UID: "af9f957f-1981-4373-83ec-66663290993c"). InnerVolumeSpecName "kube-api-access-2bk4q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:20:00.040410 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:20:00.040386    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af9f957f-1981-4373-83ec-66663290993c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "af9f957f-1981-4373-83ec-66663290993c" (UID: "af9f957f-1981-4373-83ec-66663290993c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:20:00.139019 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:20:00.138972    2576 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/af9f957f-1981-4373-83ec-66663290993c-tokenizer-cache\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:20:00.139019 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:20:00.139010    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2bk4q\" (UniqueName: \"kubernetes.io/projected/af9f957f-1981-4373-83ec-66663290993c-kube-api-access-2bk4q\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:20:00.139019 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:20:00.139026    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af9f957f-1981-4373-83ec-66663290993c-tls-certs\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:20:00.139315 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:20:00.139042    2576 reconciler_common.go:299] "Volume detached for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/af9f957f-1981-4373-83ec-66663290993c-cachi2-cache\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:20:00.560454 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:20:00.560421    2576 generic.go:356] "Generic (PLEG): container finished" podID="af9f957f-1981-4373-83ec-66663290993c" containerID="379d88d287b7bc4ded9a96e43f8e4fa6f9b74d71da41fc5d6a64340fe92cb4e0" exitCode=0
Sep 09 21:20:00.560454 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:20:00.560462    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffckjbcn"
Sep 09 21:20:00.560688 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:20:00.560511    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffckjbcn" event={"ID":"af9f957f-1981-4373-83ec-66663290993c","Type":"ContainerDied","Data":"379d88d287b7bc4ded9a96e43f8e4fa6f9b74d71da41fc5d6a64340fe92cb4e0"}
Sep 09 21:20:00.560688 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:20:00.560565    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffckjbcn" event={"ID":"af9f957f-1981-4373-83ec-66663290993c","Type":"ContainerDied","Data":"0ef01c5ae663a94f593e8b38636e9ac0d016a6148ac955e341676befe5f14e95"}
Sep 09 21:20:00.560688 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:20:00.560594    2576 scope.go:117] "RemoveContainer" containerID="379d88d287b7bc4ded9a96e43f8e4fa6f9b74d71da41fc5d6a64340fe92cb4e0"
Sep 09 21:20:00.569823 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:20:00.569807    2576 scope.go:117] "RemoveContainer" containerID="379d88d287b7bc4ded9a96e43f8e4fa6f9b74d71da41fc5d6a64340fe92cb4e0"
Sep 09 21:20:00.570083 ip-10-0-134-72 kubenswrapper[2576]: E0909 21:20:00.570063    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"379d88d287b7bc4ded9a96e43f8e4fa6f9b74d71da41fc5d6a64340fe92cb4e0\": container with ID starting with 379d88d287b7bc4ded9a96e43f8e4fa6f9b74d71da41fc5d6a64340fe92cb4e0 not found: ID does not exist" containerID="379d88d287b7bc4ded9a96e43f8e4fa6f9b74d71da41fc5d6a64340fe92cb4e0"
Sep 09 21:20:00.570145 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:20:00.570094    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"379d88d287b7bc4ded9a96e43f8e4fa6f9b74d71da41fc5d6a64340fe92cb4e0"} err="failed to get container status \"379d88d287b7bc4ded9a96e43f8e4fa6f9b74d71da41fc5d6a64340fe92cb4e0\": rpc error: code = NotFound desc = could not find container \"379d88d287b7bc4ded9a96e43f8e4fa6f9b74d71da41fc5d6a64340fe92cb4e0\": container with ID starting with 379d88d287b7bc4ded9a96e43f8e4fa6f9b74d71da41fc5d6a64340fe92cb4e0 not found: ID does not exist"
Sep 09 21:20:00.586065 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:20:00.586039    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffckjbcn"]
Sep 09 21:20:00.594245 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:20:00.594224    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffckjbcn"]
Sep 09 21:20:01.413715 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:20:01.413678    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="af9f957f-1981-4373-83ec-66663290993c" path="/var/lib/kubelet/pods/af9f957f-1981-4373-83ec-66663290993c/volumes"
Sep 09 21:20:04.629673 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:20:04.629621    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:20:04.724709 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:20:04.724675    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:20:06.213886 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:20:06.213849    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:20:06.738158 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:20:06.738129    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:20:06.842324 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:20:06.842283    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:20:08.946769 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:20:08.946740    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5skfq"]
Sep 09 21:20:08.947145 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:20:08.947075    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af9f957f-1981-4373-83ec-66663290993c" containerName="main"
Sep 09 21:20:08.947145 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:20:08.947085    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="af9f957f-1981-4373-83ec-66663290993c" containerName="main"
Sep 09 21:20:08.947145 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:20:08.947094    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82024824-18cb-4440-9874-4e1bda9127b7" containerName="main"
Sep 09 21:20:08.947145 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:20:08.947100    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="82024824-18cb-4440-9874-4e1bda9127b7" containerName="main"
Sep 09 21:20:08.947295 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:20:08.947153    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="82024824-18cb-4440-9874-4e1bda9127b7" containerName="main"
Sep 09 21:20:08.947295 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:20:08.947161    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="af9f957f-1981-4373-83ec-66663290993c" containerName="main"
Sep 09 21:20:08.949997 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:20:08.949974    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5skfq"
Sep 09 21:20:08.956595 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:20:08.956561    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"kube-root-ca.crt\""
Sep 09 21:20:08.956595 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:20:08.956573    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"openshift-service-ca.crt\""
Sep 09 21:20:08.956786 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:20:08.956564    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-kserve-self-signed-certs\""
Sep 09 21:20:08.956786 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:20:08.956602    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"default-dockercfg-6bqsz\""
Sep 09 21:20:08.967564 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:20:08.967539    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5skfq"]
Sep 09 21:20:08.990005 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:20:08.989975    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:20:09.015157 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:20:09.015129    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14487fe5-912c-4a86-a556-eb5ecb6c727c-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5skfq\" (UID: \"14487fe5-912c-4a86-a556-eb5ecb6c727c\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5skfq"
Sep 09 21:20:09.015274 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:20:09.015171    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14487fe5-912c-4a86-a556-eb5ecb6c727c-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5skfq\" (UID: \"14487fe5-912c-4a86-a556-eb5ecb6c727c\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5skfq"
Sep 09 21:20:09.015274 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:20:09.015199    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14487fe5-912c-4a86-a556-eb5ecb6c727c-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5skfq\" (UID: \"14487fe5-912c-4a86-a556-eb5ecb6c727c\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5skfq"
Sep 09 21:20:09.015359 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:20:09.015280    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14487fe5-912c-4a86-a556-eb5ecb6c727c-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5skfq\" (UID: \"14487fe5-912c-4a86-a556-eb5ecb6c727c\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5skfq"
Sep 09 21:20:09.015359 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:20:09.015337    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7r5m2\" (UniqueName: \"kubernetes.io/projected/14487fe5-912c-4a86-a556-eb5ecb6c727c-kube-api-access-7r5m2\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5skfq\" (UID: \"14487fe5-912c-4a86-a556-eb5ecb6c727c\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5skfq"
Sep 09 21:20:09.015461 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:20:09.015409    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14487fe5-912c-4a86-a556-eb5ecb6c727c-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5skfq\" (UID: \"14487fe5-912c-4a86-a556-eb5ecb6c727c\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5skfq"
Sep 09 21:20:09.116065 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:20:09.116031    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14487fe5-912c-4a86-a556-eb5ecb6c727c-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5skfq\" (UID: \"14487fe5-912c-4a86-a556-eb5ecb6c727c\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5skfq"
Sep 09 21:20:09.116229 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:20:09.116081    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7r5m2\" (UniqueName: \"kubernetes.io/projected/14487fe5-912c-4a86-a556-eb5ecb6c727c-kube-api-access-7r5m2\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5skfq\" (UID: \"14487fe5-912c-4a86-a556-eb5ecb6c727c\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5skfq"
Sep 09 21:20:09.116229 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:20:09.116116    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14487fe5-912c-4a86-a556-eb5ecb6c727c-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5skfq\" (UID: \"14487fe5-912c-4a86-a556-eb5ecb6c727c\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5skfq"
Sep 09 21:20:09.116229 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:20:09.116201    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14487fe5-912c-4a86-a556-eb5ecb6c727c-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5skfq\" (UID: \"14487fe5-912c-4a86-a556-eb5ecb6c727c\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5skfq"
Sep 09 21:20:09.116445 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:20:09.116238    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14487fe5-912c-4a86-a556-eb5ecb6c727c-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5skfq\" (UID: \"14487fe5-912c-4a86-a556-eb5ecb6c727c\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5skfq"
Sep 09 21:20:09.116445 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:20:09.116263    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14487fe5-912c-4a86-a556-eb5ecb6c727c-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5skfq\" (UID: \"14487fe5-912c-4a86-a556-eb5ecb6c727c\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5skfq"
Sep 09 21:20:09.116555 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:20:09.116503    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14487fe5-912c-4a86-a556-eb5ecb6c727c-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5skfq\" (UID: \"14487fe5-912c-4a86-a556-eb5ecb6c727c\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5skfq"
Sep 09 21:20:09.116555 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:20:09.116545    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14487fe5-912c-4a86-a556-eb5ecb6c727c-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5skfq\" (UID: \"14487fe5-912c-4a86-a556-eb5ecb6c727c\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5skfq"
Sep 09 21:20:09.116555 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:20:09.116560    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14487fe5-912c-4a86-a556-eb5ecb6c727c-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5skfq\" (UID: \"14487fe5-912c-4a86-a556-eb5ecb6c727c\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5skfq"
Sep 09 21:20:09.118718 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:20:09.118697    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14487fe5-912c-4a86-a556-eb5ecb6c727c-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5skfq\" (UID: \"14487fe5-912c-4a86-a556-eb5ecb6c727c\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5skfq"
Sep 09 21:20:09.118889 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:20:09.118871    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14487fe5-912c-4a86-a556-eb5ecb6c727c-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5skfq\" (UID: \"14487fe5-912c-4a86-a556-eb5ecb6c727c\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5skfq"
Sep 09 21:20:09.126720 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:20:09.126700    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7r5m2\" (UniqueName: \"kubernetes.io/projected/14487fe5-912c-4a86-a556-eb5ecb6c727c-kube-api-access-7r5m2\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5skfq\" (UID: \"14487fe5-912c-4a86-a556-eb5ecb6c727c\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5skfq"
Sep 09 21:20:09.256822 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:20:09.256725    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:20:09.259549 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:20:09.259527    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5skfq"
Sep 09 21:20:09.542813 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:20:09.542717    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5skfq"]
Sep 09 21:20:09.550398 ip-10-0-134-72 kubenswrapper[2576]: W0909 21:20:09.550347    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14487fe5_912c_4a86_a556_eb5ecb6c727c.slice/crio-b4237ae6a71d32ac0cf5cc9256e043559392656f47bd34dfda79d27f1101caa3 WatchSource:0}: Error finding container b4237ae6a71d32ac0cf5cc9256e043559392656f47bd34dfda79d27f1101caa3: Status 404 returned error can't find the container with id b4237ae6a71d32ac0cf5cc9256e043559392656f47bd34dfda79d27f1101caa3
Sep 09 21:20:09.613616 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:20:09.613584    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5skfq" event={"ID":"14487fe5-912c-4a86-a556-eb5ecb6c727c","Type":"ContainerStarted","Data":"b4237ae6a71d32ac0cf5cc9256e043559392656f47bd34dfda79d27f1101caa3"}
Sep 09 21:20:10.619804 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:20:10.619759    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5skfq" event={"ID":"14487fe5-912c-4a86-a556-eb5ecb6c727c","Type":"ContainerStarted","Data":"35978d1ec080f0624b764a62bb38c35e10b4a9d212449bb02b4bfc8802e417bf"}
Sep 09 21:20:10.644541 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:20:10.644497    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5skfq" podStartSLOduration=2.644482049 podStartE2EDuration="2.644482049s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:20:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:20:10.641721574 +0000 UTC m=+1653.755407435" watchObservedRunningTime="2026-09-09 21:20:10.644482049 +0000 UTC m=+1653.758167907"
Sep 09 21:20:11.259632 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:20:11.259594    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5skfq"
Sep 09 21:20:13.079025 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:20:13.078990    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:20:13.586746 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:20:13.586709    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:20:19.260184 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:20:19.260144    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5skfq"
Sep 09 21:20:19.272956 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:20:19.272933    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5skfq"
Sep 09 21:20:19.677106 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:20:19.677078    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5skfq"
Sep 09 21:20:19.984815 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:20:19.984729    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:20:21.782465 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:20:21.782430    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:20:32.414816 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:20:32.414780    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-wkq9c"]
Sep 09 21:20:32.418085 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:20:32.418068    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-wkq9c"
Sep 09 21:20:32.420773 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:20:32.420753    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-canary-service-backend-7b63470c\"/\"openshift-service-ca.crt\""
Sep 09 21:20:32.420895 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:20:32.420753    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-canary-service-backend-7b63470c\"/\"canary-v1-kserve-self-signed-certs\""
Sep 09 21:20:32.421716 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:20:32.421700    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-canary-service-backend-7b63470c\"/\"default-dockercfg-h2pxr\""
Sep 09 21:20:32.421762 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:20:32.421729    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-canary-service-backend-7b63470c\"/\"kube-root-ca.crt\""
Sep 09 21:20:32.440005 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:20:32.439985    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-wkq9c"]
Sep 09 21:20:32.466420 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:20:32.466375    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:20:32.504075 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:20:32.504040    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fabd1a0e-367b-4782-aa73-b53622397518-model-cache\") pod \"canary-v1-kserve-65d58ccc9-wkq9c\" (UID: \"fabd1a0e-367b-4782-aa73-b53622397518\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-wkq9c"
Sep 09 21:20:32.504238 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:20:32.504080    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fabd1a0e-367b-4782-aa73-b53622397518-dshm\") pod \"canary-v1-kserve-65d58ccc9-wkq9c\" (UID: \"fabd1a0e-367b-4782-aa73-b53622397518\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-wkq9c"
Sep 09 21:20:32.504238 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:20:32.504096    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22s4m\" (UniqueName: \"kubernetes.io/projected/fabd1a0e-367b-4782-aa73-b53622397518-kube-api-access-22s4m\") pod \"canary-v1-kserve-65d58ccc9-wkq9c\" (UID: \"fabd1a0e-367b-4782-aa73-b53622397518\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-wkq9c"
Sep 09 21:20:32.504238 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:20:32.504207    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fabd1a0e-367b-4782-aa73-b53622397518-tmp-dir\") pod \"canary-v1-kserve-65d58ccc9-wkq9c\" (UID: \"fabd1a0e-367b-4782-aa73-b53622397518\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-wkq9c"
Sep 09 21:20:32.504401 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:20:32.504272    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fabd1a0e-367b-4782-aa73-b53622397518-home\") pod \"canary-v1-kserve-65d58ccc9-wkq9c\" (UID: \"fabd1a0e-367b-4782-aa73-b53622397518\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-wkq9c"
Sep 09 21:20:32.504401 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:20:32.504290    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fabd1a0e-367b-4782-aa73-b53622397518-tls-certs\") pod \"canary-v1-kserve-65d58ccc9-wkq9c\" (UID: \"fabd1a0e-367b-4782-aa73-b53622397518\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-wkq9c"
Sep 09 21:20:32.604745 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:20:32.604706    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fabd1a0e-367b-4782-aa73-b53622397518-model-cache\") pod \"canary-v1-kserve-65d58ccc9-wkq9c\" (UID: \"fabd1a0e-367b-4782-aa73-b53622397518\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-wkq9c"
Sep 09 21:20:32.604745 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:20:32.604751    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fabd1a0e-367b-4782-aa73-b53622397518-dshm\") pod \"canary-v1-kserve-65d58ccc9-wkq9c\" (UID: \"fabd1a0e-367b-4782-aa73-b53622397518\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-wkq9c"
Sep 09 21:20:32.604982 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:20:32.604768    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22s4m\" (UniqueName: \"kubernetes.io/projected/fabd1a0e-367b-4782-aa73-b53622397518-kube-api-access-22s4m\") pod \"canary-v1-kserve-65d58ccc9-wkq9c\" (UID: \"fabd1a0e-367b-4782-aa73-b53622397518\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-wkq9c"
Sep 09 21:20:32.604982 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:20:32.604799    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fabd1a0e-367b-4782-aa73-b53622397518-tmp-dir\") pod \"canary-v1-kserve-65d58ccc9-wkq9c\" (UID: \"fabd1a0e-367b-4782-aa73-b53622397518\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-wkq9c"
Sep 09 21:20:32.604982 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:20:32.604837    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fabd1a0e-367b-4782-aa73-b53622397518-home\") pod \"canary-v1-kserve-65d58ccc9-wkq9c\" (UID: \"fabd1a0e-367b-4782-aa73-b53622397518\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-wkq9c"
Sep 09 21:20:32.604982 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:20:32.604851    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fabd1a0e-367b-4782-aa73-b53622397518-tls-certs\") pod \"canary-v1-kserve-65d58ccc9-wkq9c\" (UID: \"fabd1a0e-367b-4782-aa73-b53622397518\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-wkq9c"
Sep 09 21:20:32.605203 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:20:32.605175    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fabd1a0e-367b-4782-aa73-b53622397518-model-cache\") pod \"canary-v1-kserve-65d58ccc9-wkq9c\" (UID: \"fabd1a0e-367b-4782-aa73-b53622397518\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-wkq9c"
Sep 09 21:20:32.605276 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:20:32.605207    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fabd1a0e-367b-4782-aa73-b53622397518-home\") pod \"canary-v1-kserve-65d58ccc9-wkq9c\" (UID: \"fabd1a0e-367b-4782-aa73-b53622397518\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-wkq9c"
Sep 09 21:20:32.605276 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:20:32.605255    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fabd1a0e-367b-4782-aa73-b53622397518-tmp-dir\") pod \"canary-v1-kserve-65d58ccc9-wkq9c\" (UID: \"fabd1a0e-367b-4782-aa73-b53622397518\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-wkq9c"
Sep 09 21:20:32.607159 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:20:32.607139    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fabd1a0e-367b-4782-aa73-b53622397518-dshm\") pod \"canary-v1-kserve-65d58ccc9-wkq9c\" (UID: \"fabd1a0e-367b-4782-aa73-b53622397518\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-wkq9c"
Sep 09 21:20:32.607559 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:20:32.607540    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fabd1a0e-367b-4782-aa73-b53622397518-tls-certs\") pod \"canary-v1-kserve-65d58ccc9-wkq9c\" (UID: \"fabd1a0e-367b-4782-aa73-b53622397518\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-wkq9c"
Sep 09 21:20:32.616720 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:20:32.616698    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-22s4m\" (UniqueName: \"kubernetes.io/projected/fabd1a0e-367b-4782-aa73-b53622397518-kube-api-access-22s4m\") pod \"canary-v1-kserve-65d58ccc9-wkq9c\" (UID: \"fabd1a0e-367b-4782-aa73-b53622397518\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-wkq9c"
Sep 09 21:20:32.729054 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:20:32.728969    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-wkq9c"
Sep 09 21:20:33.017797 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:20:33.017766    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-wkq9c"]
Sep 09 21:20:33.020268 ip-10-0-134-72 kubenswrapper[2576]: W0909 21:20:33.020233    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfabd1a0e_367b_4782_aa73_b53622397518.slice/crio-59e49cebfaeb6275fcae60446901c1b02d5a097b0be461c2abeb005c59817dce WatchSource:0}: Error finding container 59e49cebfaeb6275fcae60446901c1b02d5a097b0be461c2abeb005c59817dce: Status 404 returned error can't find the container with id 59e49cebfaeb6275fcae60446901c1b02d5a097b0be461c2abeb005c59817dce
Sep 09 21:20:33.735759 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:20:33.735674    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-wkq9c" event={"ID":"fabd1a0e-367b-4782-aa73-b53622397518","Type":"ContainerStarted","Data":"59e49cebfaeb6275fcae60446901c1b02d5a097b0be461c2abeb005c59817dce"}
Sep 09 21:20:34.741833 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:20:34.741797    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-wkq9c" event={"ID":"fabd1a0e-367b-4782-aa73-b53622397518","Type":"ContainerStarted","Data":"a72b3dc2deaf39c862ba153d4fd597dcdc306e33142df81a507dc7309e89b9fb"}
Sep 09 21:20:34.768171 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:20:34.768122    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-wkq9c" podStartSLOduration=1.796144275 podStartE2EDuration="2.768106114s" totalImagesPullingTime="971.961839ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:20:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:20:34.765760396 +0000 UTC m=+1677.879446254" watchObservedRunningTime="2026-09-09 21:20:34.768106114 +0000 UTC m=+1677.881791972"
Sep 09 21:20:35.730051 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:20:35.730012    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-wkq9c"
Sep 09 21:20:37.113916 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:20:37.113886    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-bbhrn"]
Sep 09 21:20:37.117277 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:20:37.117260    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-bbhrn"
Sep 09 21:20:37.120169 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:20:37.120147    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-canary-service-backend-7b63470c\"/\"canary-v2-kserve-self-signed-certs\""
Sep 09 21:20:37.131395 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:20:37.131340    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-bbhrn"]
Sep 09 21:20:37.135751 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:20:37.135721    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:20:37.141746 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:20:37.141723    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqxwt\" (UniqueName: \"kubernetes.io/projected/cf8a27c9-0a46-49f1-afa9-e35a5bac691e-kube-api-access-nqxwt\") pod \"canary-v2-kserve-7f65dbbccc-bbhrn\" (UID: \"cf8a27c9-0a46-49f1-afa9-e35a5bac691e\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-bbhrn"
Sep 09 21:20:37.141832 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:20:37.141753    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf8a27c9-0a46-49f1-afa9-e35a5bac691e-home\") pod \"canary-v2-kserve-7f65dbbccc-bbhrn\" (UID: \"cf8a27c9-0a46-49f1-afa9-e35a5bac691e\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-bbhrn"
Sep 09 21:20:37.141832 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:20:37.141778    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf8a27c9-0a46-49f1-afa9-e35a5bac691e-dshm\") pod \"canary-v2-kserve-7f65dbbccc-bbhrn\" (UID: \"cf8a27c9-0a46-49f1-afa9-e35a5bac691e\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-bbhrn"
Sep 09 21:20:37.141921 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:20:37.141876    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf8a27c9-0a46-49f1-afa9-e35a5bac691e-tls-certs\") pod \"canary-v2-kserve-7f65dbbccc-bbhrn\" (UID: \"cf8a27c9-0a46-49f1-afa9-e35a5bac691e\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-bbhrn"
Sep 09 21:20:37.141963 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:20:37.141926    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf8a27c9-0a46-49f1-afa9-e35a5bac691e-tmp-dir\") pod \"canary-v2-kserve-7f65dbbccc-bbhrn\" (UID: \"cf8a27c9-0a46-49f1-afa9-e35a5bac691e\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-bbhrn"
Sep 09 21:20:37.142003 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:20:37.141973    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf8a27c9-0a46-49f1-afa9-e35a5bac691e-model-cache\") pod \"canary-v2-kserve-7f65dbbccc-bbhrn\" (UID: \"cf8a27c9-0a46-49f1-afa9-e35a5bac691e\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-bbhrn"
Sep 09 21:20:37.243091 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:20:37.243051    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqxwt\" (UniqueName: \"kubernetes.io/projected/cf8a27c9-0a46-49f1-afa9-e35a5bac691e-kube-api-access-nqxwt\") pod \"canary-v2-kserve-7f65dbbccc-bbhrn\" (UID: \"cf8a27c9-0a46-49f1-afa9-e35a5bac691e\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-bbhrn"
Sep 09 21:20:37.243091 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:20:37.243095    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf8a27c9-0a46-49f1-afa9-e35a5bac691e-home\") pod \"canary-v2-kserve-7f65dbbccc-bbhrn\" (UID: \"cf8a27c9-0a46-49f1-afa9-e35a5bac691e\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-bbhrn"
Sep 09 21:20:37.243402 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:20:37.243142    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf8a27c9-0a46-49f1-afa9-e35a5bac691e-dshm\") pod \"canary-v2-kserve-7f65dbbccc-bbhrn\" (UID: \"cf8a27c9-0a46-49f1-afa9-e35a5bac691e\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-bbhrn"
Sep 09 21:20:37.243402 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:20:37.243183    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf8a27c9-0a46-49f1-afa9-e35a5bac691e-tls-certs\") pod \"canary-v2-kserve-7f65dbbccc-bbhrn\" (UID: \"cf8a27c9-0a46-49f1-afa9-e35a5bac691e\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-bbhrn"
Sep 09 21:20:37.243402 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:20:37.243215    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf8a27c9-0a46-49f1-afa9-e35a5bac691e-tmp-dir\") pod \"canary-v2-kserve-7f65dbbccc-bbhrn\" (UID: \"cf8a27c9-0a46-49f1-afa9-e35a5bac691e\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-bbhrn"
Sep 09 21:20:37.243402 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:20:37.243261    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf8a27c9-0a46-49f1-afa9-e35a5bac691e-model-cache\") pod \"canary-v2-kserve-7f65dbbccc-bbhrn\" (UID: \"cf8a27c9-0a46-49f1-afa9-e35a5bac691e\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-bbhrn"
Sep 09 21:20:37.243624 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:20:37.243596    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf8a27c9-0a46-49f1-afa9-e35a5bac691e-home\") pod \"canary-v2-kserve-7f65dbbccc-bbhrn\" (UID: \"cf8a27c9-0a46-49f1-afa9-e35a5bac691e\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-bbhrn"
Sep 09 21:20:37.243707 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:20:37.243685    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf8a27c9-0a46-49f1-afa9-e35a5bac691e-model-cache\") pod \"canary-v2-kserve-7f65dbbccc-bbhrn\" (UID: \"cf8a27c9-0a46-49f1-afa9-e35a5bac691e\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-bbhrn"
Sep 09 21:20:37.243760 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:20:37.243721    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf8a27c9-0a46-49f1-afa9-e35a5bac691e-tmp-dir\") pod \"canary-v2-kserve-7f65dbbccc-bbhrn\" (UID: \"cf8a27c9-0a46-49f1-afa9-e35a5bac691e\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-bbhrn"
Sep 09 21:20:37.245721 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:20:37.245700    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf8a27c9-0a46-49f1-afa9-e35a5bac691e-dshm\") pod \"canary-v2-kserve-7f65dbbccc-bbhrn\" (UID: \"cf8a27c9-0a46-49f1-afa9-e35a5bac691e\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-bbhrn"
Sep 09 21:20:37.245882 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:20:37.245868    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf8a27c9-0a46-49f1-afa9-e35a5bac691e-tls-certs\") pod \"canary-v2-kserve-7f65dbbccc-bbhrn\" (UID: \"cf8a27c9-0a46-49f1-afa9-e35a5bac691e\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-bbhrn"
Sep 09 21:20:37.253353 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:20:37.253332    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqxwt\" (UniqueName: \"kubernetes.io/projected/cf8a27c9-0a46-49f1-afa9-e35a5bac691e-kube-api-access-nqxwt\") pod \"canary-v2-kserve-7f65dbbccc-bbhrn\" (UID: \"cf8a27c9-0a46-49f1-afa9-e35a5bac691e\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-bbhrn"
Sep 09 21:20:37.427259 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:20:37.427174    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-bbhrn"
Sep 09 21:20:37.706260 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:20:37.706176    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-bbhrn"]
Sep 09 21:20:37.709102 ip-10-0-134-72 kubenswrapper[2576]: W0909 21:20:37.709066    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf8a27c9_0a46_49f1_afa9_e35a5bac691e.slice/crio-1a0070e261662e52fbb3314dc12e6e14cad253d8d66796501fd97382e69747da WatchSource:0}: Error finding container 1a0070e261662e52fbb3314dc12e6e14cad253d8d66796501fd97382e69747da: Status 404 returned error can't find the container with id 1a0070e261662e52fbb3314dc12e6e14cad253d8d66796501fd97382e69747da
Sep 09 21:20:37.758679 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:20:37.758651    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-bbhrn" event={"ID":"cf8a27c9-0a46-49f1-afa9-e35a5bac691e","Type":"ContainerStarted","Data":"1a0070e261662e52fbb3314dc12e6e14cad253d8d66796501fd97382e69747da"}
Sep 09 21:20:38.768159 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:20:38.768125    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-bbhrn" event={"ID":"cf8a27c9-0a46-49f1-afa9-e35a5bac691e","Type":"ContainerStarted","Data":"c64bd2f5db50a990fc7fd748299cc2d254bf6e5e8f44d0afbf3d5982bff489c2"}
Sep 09 21:20:38.790022 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:20:38.789974    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-bbhrn" podStartSLOduration=1.789958361 podStartE2EDuration="1.789958361s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:20:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:20:38.7892222 +0000 UTC m=+1681.902908071" watchObservedRunningTime="2026-09-09 21:20:38.789958361 +0000 UTC m=+1681.903644220"
Sep 09 21:20:39.427532 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:20:39.427500    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-bbhrn"
Sep 09 21:20:42.729806 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:20:42.729769    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-wkq9c"
Sep 09 21:20:42.742011 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:20:42.741988    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-wkq9c"
Sep 09 21:20:42.799213 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:20:42.799191    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-wkq9c"
Sep 09 21:20:47.427723 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:20:47.427689    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-bbhrn"
Sep 09 21:20:47.440187 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:20:47.440166    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-bbhrn"
Sep 09 21:20:47.825324 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:20:47.825225    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-bbhrn"
Sep 09 21:20:48.466884 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:20:48.466850    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:21:06.536634 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:21:06.536537    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:21:17.629114 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:21:17.629080    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:21:43.206186 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:21:43.206147    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:21:43.306762 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:21:43.306723    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:21:59.640396 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:21:59.640342    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5skfq"]
Sep 09 21:21:59.649564 ip-10-0-134-72 kubenswrapper[2576]: E0909 21:21:59.649531    2576 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 09 21:21:59.649564 ip-10-0-134-72 kubenswrapper[2576]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 21:21:59.649564 ip-10-0-134-72 kubenswrapper[2576]:         , stdout: , stderr: , exit code -1
Sep 09 21:21:59.649564 ip-10-0-134-72 kubenswrapper[2576]:  > containerID="35978d1ec080f0624b764a62bb38c35e10b4a9d212449bb02b4bfc8802e417bf" cmd=["/bin/sleep","15"]
Sep 09 21:21:59.649837 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:21:59.649582    2576 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5skfq" message="" err=<
Sep 09 21:21:59.649837 ip-10-0-134-72 kubenswrapper[2576]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 21:21:59.649837 ip-10-0-134-72 kubenswrapper[2576]:         , stdout: , stderr: , exit code -1
Sep 09 21:21:59.649837 ip-10-0-134-72 kubenswrapper[2576]:  >
Sep 09 21:21:59.649837 ip-10-0-134-72 kubenswrapper[2576]: E0909 21:21:59.649594    2576 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 09 21:21:59.649837 ip-10-0-134-72 kubenswrapper[2576]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 21:21:59.649837 ip-10-0-134-72 kubenswrapper[2576]:         , stdout: , stderr: , exit code -1
Sep 09 21:21:59.649837 ip-10-0-134-72 kubenswrapper[2576]:  > pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5skfq" podUID="14487fe5-912c-4a86-a556-eb5ecb6c727c" containerName="main" containerID="cri-o://35978d1ec080f0624b764a62bb38c35e10b4a9d212449bb02b4bfc8802e417bf"
Sep 09 21:21:59.649837 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:21:59.649629    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5skfq" podUID="14487fe5-912c-4a86-a556-eb5ecb6c727c" containerName="main" containerID="cri-o://35978d1ec080f0624b764a62bb38c35e10b4a9d212449bb02b4bfc8802e417bf" gracePeriod=60
Sep 09 21:21:59.658497 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:21:59.658470    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:21:59.666480 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:21:59.666455    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5skfq" podUID="14487fe5-912c-4a86-a556-eb5ecb6c727c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/health\": dial tcp 10.132.0.55:8000: connect: connection refused"
Sep 09 21:21:59.753902 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:21:59.753856    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:22:00.002077 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:00.002053    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5skfq"
Sep 09 21:22:00.057216 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:00.057184    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14487fe5-912c-4a86-a556-eb5ecb6c727c-home\") pod \"14487fe5-912c-4a86-a556-eb5ecb6c727c\" (UID: \"14487fe5-912c-4a86-a556-eb5ecb6c727c\") "
Sep 09 21:22:00.057411 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:00.057245    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14487fe5-912c-4a86-a556-eb5ecb6c727c-tls-certs\") pod \"14487fe5-912c-4a86-a556-eb5ecb6c727c\" (UID: \"14487fe5-912c-4a86-a556-eb5ecb6c727c\") "
Sep 09 21:22:00.057411 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:00.057269    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14487fe5-912c-4a86-a556-eb5ecb6c727c-tmp-dir\") pod \"14487fe5-912c-4a86-a556-eb5ecb6c727c\" (UID: \"14487fe5-912c-4a86-a556-eb5ecb6c727c\") "
Sep 09 21:22:00.057411 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:00.057292    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14487fe5-912c-4a86-a556-eb5ecb6c727c-model-cache\") pod \"14487fe5-912c-4a86-a556-eb5ecb6c727c\" (UID: \"14487fe5-912c-4a86-a556-eb5ecb6c727c\") "
Sep 09 21:22:00.057411 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:00.057321    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14487fe5-912c-4a86-a556-eb5ecb6c727c-dshm\") pod \"14487fe5-912c-4a86-a556-eb5ecb6c727c\" (UID: \"14487fe5-912c-4a86-a556-eb5ecb6c727c\") "
Sep 09 21:22:00.057411 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:00.057382    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7r5m2\" (UniqueName: \"kubernetes.io/projected/14487fe5-912c-4a86-a556-eb5ecb6c727c-kube-api-access-7r5m2\") pod \"14487fe5-912c-4a86-a556-eb5ecb6c727c\" (UID: \"14487fe5-912c-4a86-a556-eb5ecb6c727c\") "
Sep 09 21:22:00.057687 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:00.057530    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14487fe5-912c-4a86-a556-eb5ecb6c727c-home" (OuterVolumeSpecName: "home") pod "14487fe5-912c-4a86-a556-eb5ecb6c727c" (UID: "14487fe5-912c-4a86-a556-eb5ecb6c727c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:22:00.057687 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:00.057553    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14487fe5-912c-4a86-a556-eb5ecb6c727c-model-cache" (OuterVolumeSpecName: "model-cache") pod "14487fe5-912c-4a86-a556-eb5ecb6c727c" (UID: "14487fe5-912c-4a86-a556-eb5ecb6c727c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:22:00.057687 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:00.057608    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14487fe5-912c-4a86-a556-eb5ecb6c727c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "14487fe5-912c-4a86-a556-eb5ecb6c727c" (UID: "14487fe5-912c-4a86-a556-eb5ecb6c727c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:22:00.057811 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:00.057735    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14487fe5-912c-4a86-a556-eb5ecb6c727c-home\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:22:00.057811 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:00.057752    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14487fe5-912c-4a86-a556-eb5ecb6c727c-tmp-dir\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:22:00.057811 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:00.057765    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14487fe5-912c-4a86-a556-eb5ecb6c727c-model-cache\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:22:00.059723 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:00.059690    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14487fe5-912c-4a86-a556-eb5ecb6c727c-kube-api-access-7r5m2" (OuterVolumeSpecName: "kube-api-access-7r5m2") pod "14487fe5-912c-4a86-a556-eb5ecb6c727c" (UID: "14487fe5-912c-4a86-a556-eb5ecb6c727c"). InnerVolumeSpecName "kube-api-access-7r5m2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:22:00.059837 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:00.059725    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14487fe5-912c-4a86-a556-eb5ecb6c727c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "14487fe5-912c-4a86-a556-eb5ecb6c727c" (UID: "14487fe5-912c-4a86-a556-eb5ecb6c727c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:22:00.059837 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:00.059728    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14487fe5-912c-4a86-a556-eb5ecb6c727c-dshm" (OuterVolumeSpecName: "dshm") pod "14487fe5-912c-4a86-a556-eb5ecb6c727c" (UID: "14487fe5-912c-4a86-a556-eb5ecb6c727c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:22:00.158521 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:00.158489    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14487fe5-912c-4a86-a556-eb5ecb6c727c-dshm\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:22:00.158521 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:00.158515    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7r5m2\" (UniqueName: \"kubernetes.io/projected/14487fe5-912c-4a86-a556-eb5ecb6c727c-kube-api-access-7r5m2\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:22:00.158521 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:00.158525    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14487fe5-912c-4a86-a556-eb5ecb6c727c-tls-certs\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:22:00.182087 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:00.181999    2576 generic.go:356] "Generic (PLEG): container finished" podID="14487fe5-912c-4a86-a556-eb5ecb6c727c" containerID="35978d1ec080f0624b764a62bb38c35e10b4a9d212449bb02b4bfc8802e417bf" exitCode=0
Sep 09 21:22:00.182087 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:00.182052    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5skfq"
Sep 09 21:22:00.182297 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:00.182082    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5skfq" event={"ID":"14487fe5-912c-4a86-a556-eb5ecb6c727c","Type":"ContainerDied","Data":"35978d1ec080f0624b764a62bb38c35e10b4a9d212449bb02b4bfc8802e417bf"}
Sep 09 21:22:00.182297 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:00.182135    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5skfq" event={"ID":"14487fe5-912c-4a86-a556-eb5ecb6c727c","Type":"ContainerDied","Data":"b4237ae6a71d32ac0cf5cc9256e043559392656f47bd34dfda79d27f1101caa3"}
Sep 09 21:22:00.182297 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:00.182159    2576 scope.go:117] "RemoveContainer" containerID="35978d1ec080f0624b764a62bb38c35e10b4a9d212449bb02b4bfc8802e417bf"
Sep 09 21:22:00.192214 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:00.192196    2576 scope.go:117] "RemoveContainer" containerID="35978d1ec080f0624b764a62bb38c35e10b4a9d212449bb02b4bfc8802e417bf"
Sep 09 21:22:00.192502 ip-10-0-134-72 kubenswrapper[2576]: E0909 21:22:00.192483    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35978d1ec080f0624b764a62bb38c35e10b4a9d212449bb02b4bfc8802e417bf\": container with ID starting with 35978d1ec080f0624b764a62bb38c35e10b4a9d212449bb02b4bfc8802e417bf not found: ID does not exist" containerID="35978d1ec080f0624b764a62bb38c35e10b4a9d212449bb02b4bfc8802e417bf"
Sep 09 21:22:00.192572 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:00.192516    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35978d1ec080f0624b764a62bb38c35e10b4a9d212449bb02b4bfc8802e417bf"} err="failed to get container status \"35978d1ec080f0624b764a62bb38c35e10b4a9d212449bb02b4bfc8802e417bf\": rpc error: code = NotFound desc = could not find container \"35978d1ec080f0624b764a62bb38c35e10b4a9d212449bb02b4bfc8802e417bf\": container with ID starting with 35978d1ec080f0624b764a62bb38c35e10b4a9d212449bb02b4bfc8802e417bf not found: ID does not exist"
Sep 09 21:22:00.208446 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:00.208388    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5skfq"]
Sep 09 21:22:00.210631 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:00.210611    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5skfq"]
Sep 09 21:22:01.408775 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:01.408740    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="14487fe5-912c-4a86-a556-eb5ecb6c727c" path="/var/lib/kubelet/pods/14487fe5-912c-4a86-a556-eb5ecb6c727c/volumes"
Sep 09 21:22:24.494046 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:24.494001    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:22:24.598584 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:24.598534    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:22:24.701172 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:24.701141    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:22:24.758827 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:24.758751    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-bbhrn"]
Sep 09 21:22:24.793196 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:24.793159    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:22:25.215069 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:25.215040    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-wkq9c"]
Sep 09 21:22:25.236404 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:25.236349    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:22:29.566100 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:29.566012    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:22:38.611955 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:38.611913    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-6jzgz"]
Sep 09 21:22:38.612492 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:38.612472    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14487fe5-912c-4a86-a556-eb5ecb6c727c" containerName="main"
Sep 09 21:22:38.612590 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:38.612495    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="14487fe5-912c-4a86-a556-eb5ecb6c727c" containerName="main"
Sep 09 21:22:38.612646 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:38.612603    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="14487fe5-912c-4a86-a556-eb5ecb6c727c" containerName="main"
Sep 09 21:22:38.615530 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:38.615509    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-6jzgz"
Sep 09 21:22:38.618589 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:38.618571    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-model-name-divergence-75f1ef44\"/\"kube-root-ca.crt\""
Sep 09 21:22:38.618830 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:38.618811    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-model-name-divergence-75f1ef44\"/\"openshift-service-ca.crt\""
Sep 09 21:22:38.619594 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:38.619577    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-model-name-divergence-75f1ef44\"/\"default-dockercfg-ck9jb\""
Sep 09 21:22:38.619693 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:38.619593    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-model-name-divergence-75f1ef44\"/\"test-div-v1-kserve-self-signed-certs\""
Sep 09 21:22:38.626953 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:38.626927    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:22:38.637113 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:38.637092    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-6jzgz"]
Sep 09 21:22:38.680023 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:38.679990    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c557d9d-e865-408e-b7a7-42281ff091a4-model-cache\") pod \"test-div-v1-kserve-fb566d667-6jzgz\" (UID: \"2c557d9d-e865-408e-b7a7-42281ff091a4\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-6jzgz"
Sep 09 21:22:38.680169 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:38.680040    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jv98c\" (UniqueName: \"kubernetes.io/projected/2c557d9d-e865-408e-b7a7-42281ff091a4-kube-api-access-jv98c\") pod \"test-div-v1-kserve-fb566d667-6jzgz\" (UID: \"2c557d9d-e865-408e-b7a7-42281ff091a4\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-6jzgz"
Sep 09 21:22:38.680169 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:38.680108    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c557d9d-e865-408e-b7a7-42281ff091a4-dshm\") pod \"test-div-v1-kserve-fb566d667-6jzgz\" (UID: \"2c557d9d-e865-408e-b7a7-42281ff091a4\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-6jzgz"
Sep 09 21:22:38.680169 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:38.680139    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c557d9d-e865-408e-b7a7-42281ff091a4-home\") pod \"test-div-v1-kserve-fb566d667-6jzgz\" (UID: \"2c557d9d-e865-408e-b7a7-42281ff091a4\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-6jzgz"
Sep 09 21:22:38.680307 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:38.680204    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c557d9d-e865-408e-b7a7-42281ff091a4-tmp-dir\") pod \"test-div-v1-kserve-fb566d667-6jzgz\" (UID: \"2c557d9d-e865-408e-b7a7-42281ff091a4\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-6jzgz"
Sep 09 21:22:38.680307 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:38.680283    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c557d9d-e865-408e-b7a7-42281ff091a4-tls-certs\") pod \"test-div-v1-kserve-fb566d667-6jzgz\" (UID: \"2c557d9d-e865-408e-b7a7-42281ff091a4\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-6jzgz"
Sep 09 21:22:38.781549 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:38.781514    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c557d9d-e865-408e-b7a7-42281ff091a4-tls-certs\") pod \"test-div-v1-kserve-fb566d667-6jzgz\" (UID: \"2c557d9d-e865-408e-b7a7-42281ff091a4\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-6jzgz"
Sep 09 21:22:38.781709 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:38.781561    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c557d9d-e865-408e-b7a7-42281ff091a4-model-cache\") pod \"test-div-v1-kserve-fb566d667-6jzgz\" (UID: \"2c557d9d-e865-408e-b7a7-42281ff091a4\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-6jzgz"
Sep 09 21:22:38.781709 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:38.781599    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jv98c\" (UniqueName: \"kubernetes.io/projected/2c557d9d-e865-408e-b7a7-42281ff091a4-kube-api-access-jv98c\") pod \"test-div-v1-kserve-fb566d667-6jzgz\" (UID: \"2c557d9d-e865-408e-b7a7-42281ff091a4\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-6jzgz"
Sep 09 21:22:38.781709 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:38.781645    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c557d9d-e865-408e-b7a7-42281ff091a4-dshm\") pod \"test-div-v1-kserve-fb566d667-6jzgz\" (UID: \"2c557d9d-e865-408e-b7a7-42281ff091a4\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-6jzgz"
Sep 09 21:22:38.781709 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:38.781673    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c557d9d-e865-408e-b7a7-42281ff091a4-home\") pod \"test-div-v1-kserve-fb566d667-6jzgz\" (UID: \"2c557d9d-e865-408e-b7a7-42281ff091a4\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-6jzgz"
Sep 09 21:22:38.781908 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:38.781794    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c557d9d-e865-408e-b7a7-42281ff091a4-tmp-dir\") pod \"test-div-v1-kserve-fb566d667-6jzgz\" (UID: \"2c557d9d-e865-408e-b7a7-42281ff091a4\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-6jzgz"
Sep 09 21:22:38.782107 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:38.782083    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c557d9d-e865-408e-b7a7-42281ff091a4-model-cache\") pod \"test-div-v1-kserve-fb566d667-6jzgz\" (UID: \"2c557d9d-e865-408e-b7a7-42281ff091a4\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-6jzgz"
Sep 09 21:22:38.782180 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:38.782119    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c557d9d-e865-408e-b7a7-42281ff091a4-home\") pod \"test-div-v1-kserve-fb566d667-6jzgz\" (UID: \"2c557d9d-e865-408e-b7a7-42281ff091a4\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-6jzgz"
Sep 09 21:22:38.782180 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:38.782147    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c557d9d-e865-408e-b7a7-42281ff091a4-tmp-dir\") pod \"test-div-v1-kserve-fb566d667-6jzgz\" (UID: \"2c557d9d-e865-408e-b7a7-42281ff091a4\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-6jzgz"
Sep 09 21:22:38.784043 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:38.784022    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c557d9d-e865-408e-b7a7-42281ff091a4-dshm\") pod \"test-div-v1-kserve-fb566d667-6jzgz\" (UID: \"2c557d9d-e865-408e-b7a7-42281ff091a4\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-6jzgz"
Sep 09 21:22:38.784212 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:38.784193    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c557d9d-e865-408e-b7a7-42281ff091a4-tls-certs\") pod \"test-div-v1-kserve-fb566d667-6jzgz\" (UID: \"2c557d9d-e865-408e-b7a7-42281ff091a4\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-6jzgz"
Sep 09 21:22:38.791294 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:38.791269    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jv98c\" (UniqueName: \"kubernetes.io/projected/2c557d9d-e865-408e-b7a7-42281ff091a4-kube-api-access-jv98c\") pod \"test-div-v1-kserve-fb566d667-6jzgz\" (UID: \"2c557d9d-e865-408e-b7a7-42281ff091a4\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-6jzgz"
Sep 09 21:22:38.925585 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:38.925497    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-6jzgz"
Sep 09 21:22:39.214119 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:39.214089    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-6jzgz"]
Sep 09 21:22:39.215934 ip-10-0-134-72 kubenswrapper[2576]: W0909 21:22:39.215899    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c557d9d_e865_408e_b7a7_42281ff091a4.slice/crio-7967f9a72d9740d74a24b93c3a28ef1efaf294e0fb901f5f1f9d9e103074986e WatchSource:0}: Error finding container 7967f9a72d9740d74a24b93c3a28ef1efaf294e0fb901f5f1f9d9e103074986e: Status 404 returned error can't find the container with id 7967f9a72d9740d74a24b93c3a28ef1efaf294e0fb901f5f1f9d9e103074986e
Sep 09 21:22:39.387126 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:39.387094    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-6jzgz" event={"ID":"2c557d9d-e865-408e-b7a7-42281ff091a4","Type":"ContainerStarted","Data":"6c46d7b6e2d3ef6fab4e5a89f5f60d2bcfc35917389df5606c55d401d67e9a6e"}
Sep 09 21:22:39.387319 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:39.387145    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-6jzgz" event={"ID":"2c557d9d-e865-408e-b7a7-42281ff091a4","Type":"ContainerStarted","Data":"7967f9a72d9740d74a24b93c3a28ef1efaf294e0fb901f5f1f9d9e103074986e"}
Sep 09 21:22:39.412288 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:39.412240    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-6jzgz" podStartSLOduration=1.412225208 podStartE2EDuration="1.412225208s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:22:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:22:39.410784263 +0000 UTC m=+1802.524470125" watchObservedRunningTime="2026-09-09 21:22:39.412225208 +0000 UTC m=+1802.525911068"
Sep 09 21:22:39.778354 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:39.778307    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-bbhrn" podUID="cf8a27c9-0a46-49f1-afa9-e35a5bac691e" containerName="main" containerID="cri-o://c64bd2f5db50a990fc7fd748299cc2d254bf6e5e8f44d0afbf3d5982bff489c2" gracePeriod=45
Sep 09 21:22:39.814442 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:39.814408    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-bbhrn" podUID="cf8a27c9-0a46-49f1-afa9-e35a5bac691e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": dial tcp 10.132.0.57:8000: connect: connection refused"
Sep 09 21:22:39.926187 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:39.926145    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-6jzgz"
Sep 09 21:22:40.024467 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:40.024444    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-bbhrn"
Sep 09 21:22:40.094557 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:40.094530    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf8a27c9-0a46-49f1-afa9-e35a5bac691e-tls-certs\") pod \"cf8a27c9-0a46-49f1-afa9-e35a5bac691e\" (UID: \"cf8a27c9-0a46-49f1-afa9-e35a5bac691e\") "
Sep 09 21:22:40.094764 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:40.094577    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nqxwt\" (UniqueName: \"kubernetes.io/projected/cf8a27c9-0a46-49f1-afa9-e35a5bac691e-kube-api-access-nqxwt\") pod \"cf8a27c9-0a46-49f1-afa9-e35a5bac691e\" (UID: \"cf8a27c9-0a46-49f1-afa9-e35a5bac691e\") "
Sep 09 21:22:40.094764 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:40.094600    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf8a27c9-0a46-49f1-afa9-e35a5bac691e-tmp-dir\") pod \"cf8a27c9-0a46-49f1-afa9-e35a5bac691e\" (UID: \"cf8a27c9-0a46-49f1-afa9-e35a5bac691e\") "
Sep 09 21:22:40.094764 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:40.094647    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf8a27c9-0a46-49f1-afa9-e35a5bac691e-home\") pod \"cf8a27c9-0a46-49f1-afa9-e35a5bac691e\" (UID: \"cf8a27c9-0a46-49f1-afa9-e35a5bac691e\") "
Sep 09 21:22:40.094764 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:40.094674    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf8a27c9-0a46-49f1-afa9-e35a5bac691e-dshm\") pod \"cf8a27c9-0a46-49f1-afa9-e35a5bac691e\" (UID: \"cf8a27c9-0a46-49f1-afa9-e35a5bac691e\") "
Sep 09 21:22:40.094764 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:40.094723    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf8a27c9-0a46-49f1-afa9-e35a5bac691e-model-cache\") pod \"cf8a27c9-0a46-49f1-afa9-e35a5bac691e\" (UID: \"cf8a27c9-0a46-49f1-afa9-e35a5bac691e\") "
Sep 09 21:22:40.095043 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:40.094884    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf8a27c9-0a46-49f1-afa9-e35a5bac691e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cf8a27c9-0a46-49f1-afa9-e35a5bac691e" (UID: "cf8a27c9-0a46-49f1-afa9-e35a5bac691e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:22:40.095043 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:40.094932    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf8a27c9-0a46-49f1-afa9-e35a5bac691e-home" (OuterVolumeSpecName: "home") pod "cf8a27c9-0a46-49f1-afa9-e35a5bac691e" (UID: "cf8a27c9-0a46-49f1-afa9-e35a5bac691e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:22:40.095043 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:40.094992    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf8a27c9-0a46-49f1-afa9-e35a5bac691e-model-cache" (OuterVolumeSpecName: "model-cache") pod "cf8a27c9-0a46-49f1-afa9-e35a5bac691e" (UID: "cf8a27c9-0a46-49f1-afa9-e35a5bac691e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:22:40.095149 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:40.095047    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf8a27c9-0a46-49f1-afa9-e35a5bac691e-tmp-dir\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:22:40.095149 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:40.095067    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf8a27c9-0a46-49f1-afa9-e35a5bac691e-home\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:22:40.097028 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:40.097005    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf8a27c9-0a46-49f1-afa9-e35a5bac691e-dshm" (OuterVolumeSpecName: "dshm") pod "cf8a27c9-0a46-49f1-afa9-e35a5bac691e" (UID: "cf8a27c9-0a46-49f1-afa9-e35a5bac691e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:22:40.097327 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:40.097308    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf8a27c9-0a46-49f1-afa9-e35a5bac691e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cf8a27c9-0a46-49f1-afa9-e35a5bac691e" (UID: "cf8a27c9-0a46-49f1-afa9-e35a5bac691e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:22:40.097415 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:40.097310    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf8a27c9-0a46-49f1-afa9-e35a5bac691e-kube-api-access-nqxwt" (OuterVolumeSpecName: "kube-api-access-nqxwt") pod "cf8a27c9-0a46-49f1-afa9-e35a5bac691e" (UID: "cf8a27c9-0a46-49f1-afa9-e35a5bac691e"). InnerVolumeSpecName "kube-api-access-nqxwt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:22:40.196005 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:40.195915    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf8a27c9-0a46-49f1-afa9-e35a5bac691e-tls-certs\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:22:40.196005 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:40.195949    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqxwt\" (UniqueName: \"kubernetes.io/projected/cf8a27c9-0a46-49f1-afa9-e35a5bac691e-kube-api-access-nqxwt\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:22:40.196005 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:40.195960    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf8a27c9-0a46-49f1-afa9-e35a5bac691e-dshm\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:22:40.196005 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:40.195972    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf8a27c9-0a46-49f1-afa9-e35a5bac691e-model-cache\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:22:40.234930 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:40.234885    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-wkq9c" podUID="fabd1a0e-367b-4782-aa73-b53622397518" containerName="main" containerID="cri-o://a72b3dc2deaf39c862ba153d4fd597dcdc306e33142df81a507dc7309e89b9fb" gracePeriod=45
Sep 09 21:22:40.394535 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:40.394504    2576 generic.go:356] "Generic (PLEG): container finished" podID="fabd1a0e-367b-4782-aa73-b53622397518" containerID="a72b3dc2deaf39c862ba153d4fd597dcdc306e33142df81a507dc7309e89b9fb" exitCode=0
Sep 09 21:22:40.394703 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:40.394580    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-wkq9c" event={"ID":"fabd1a0e-367b-4782-aa73-b53622397518","Type":"ContainerDied","Data":"a72b3dc2deaf39c862ba153d4fd597dcdc306e33142df81a507dc7309e89b9fb"}
Sep 09 21:22:40.395918 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:40.395884    2576 generic.go:356] "Generic (PLEG): container finished" podID="cf8a27c9-0a46-49f1-afa9-e35a5bac691e" containerID="c64bd2f5db50a990fc7fd748299cc2d254bf6e5e8f44d0afbf3d5982bff489c2" exitCode=0
Sep 09 21:22:40.395918 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:40.395910    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-bbhrn" event={"ID":"cf8a27c9-0a46-49f1-afa9-e35a5bac691e","Type":"ContainerDied","Data":"c64bd2f5db50a990fc7fd748299cc2d254bf6e5e8f44d0afbf3d5982bff489c2"}
Sep 09 21:22:40.396107 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:40.395933    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-bbhrn"
Sep 09 21:22:40.396107 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:40.395970    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-bbhrn" event={"ID":"cf8a27c9-0a46-49f1-afa9-e35a5bac691e","Type":"ContainerDied","Data":"1a0070e261662e52fbb3314dc12e6e14cad253d8d66796501fd97382e69747da"}
Sep 09 21:22:40.396107 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:40.396011    2576 scope.go:117] "RemoveContainer" containerID="c64bd2f5db50a990fc7fd748299cc2d254bf6e5e8f44d0afbf3d5982bff489c2"
Sep 09 21:22:40.405983 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:40.405966    2576 scope.go:117] "RemoveContainer" containerID="c64bd2f5db50a990fc7fd748299cc2d254bf6e5e8f44d0afbf3d5982bff489c2"
Sep 09 21:22:40.406263 ip-10-0-134-72 kubenswrapper[2576]: E0909 21:22:40.406244    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c64bd2f5db50a990fc7fd748299cc2d254bf6e5e8f44d0afbf3d5982bff489c2\": container with ID starting with c64bd2f5db50a990fc7fd748299cc2d254bf6e5e8f44d0afbf3d5982bff489c2 not found: ID does not exist" containerID="c64bd2f5db50a990fc7fd748299cc2d254bf6e5e8f44d0afbf3d5982bff489c2"
Sep 09 21:22:40.406334 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:40.406270    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c64bd2f5db50a990fc7fd748299cc2d254bf6e5e8f44d0afbf3d5982bff489c2"} err="failed to get container status \"c64bd2f5db50a990fc7fd748299cc2d254bf6e5e8f44d0afbf3d5982bff489c2\": rpc error: code = NotFound desc = could not find container \"c64bd2f5db50a990fc7fd748299cc2d254bf6e5e8f44d0afbf3d5982bff489c2\": container with ID starting with c64bd2f5db50a990fc7fd748299cc2d254bf6e5e8f44d0afbf3d5982bff489c2 not found: ID does not exist"
Sep 09 21:22:40.428177 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:40.420530    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-bbhrn"]
Sep 09 21:22:40.431739 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:40.431716    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-bbhrn"]
Sep 09 21:22:40.488531 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:40.488508    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-wkq9c"
Sep 09 21:22:40.599512 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:40.599477    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fabd1a0e-367b-4782-aa73-b53622397518-model-cache\") pod \"fabd1a0e-367b-4782-aa73-b53622397518\" (UID: \"fabd1a0e-367b-4782-aa73-b53622397518\") "
Sep 09 21:22:40.599512 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:40.599514    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fabd1a0e-367b-4782-aa73-b53622397518-tls-certs\") pod \"fabd1a0e-367b-4782-aa73-b53622397518\" (UID: \"fabd1a0e-367b-4782-aa73-b53622397518\") "
Sep 09 21:22:40.599759 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:40.599544    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-22s4m\" (UniqueName: \"kubernetes.io/projected/fabd1a0e-367b-4782-aa73-b53622397518-kube-api-access-22s4m\") pod \"fabd1a0e-367b-4782-aa73-b53622397518\" (UID: \"fabd1a0e-367b-4782-aa73-b53622397518\") "
Sep 09 21:22:40.599759 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:40.599577    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fabd1a0e-367b-4782-aa73-b53622397518-dshm\") pod \"fabd1a0e-367b-4782-aa73-b53622397518\" (UID: \"fabd1a0e-367b-4782-aa73-b53622397518\") "
Sep 09 21:22:40.599759 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:40.599597    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fabd1a0e-367b-4782-aa73-b53622397518-tmp-dir\") pod \"fabd1a0e-367b-4782-aa73-b53622397518\" (UID: \"fabd1a0e-367b-4782-aa73-b53622397518\") "
Sep 09 21:22:40.599759 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:40.599610    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fabd1a0e-367b-4782-aa73-b53622397518-home\") pod \"fabd1a0e-367b-4782-aa73-b53622397518\" (UID: \"fabd1a0e-367b-4782-aa73-b53622397518\") "
Sep 09 21:22:40.599962 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:40.599750    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fabd1a0e-367b-4782-aa73-b53622397518-model-cache" (OuterVolumeSpecName: "model-cache") pod "fabd1a0e-367b-4782-aa73-b53622397518" (UID: "fabd1a0e-367b-4782-aa73-b53622397518"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:22:40.599962 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:40.599903    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fabd1a0e-367b-4782-aa73-b53622397518-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fabd1a0e-367b-4782-aa73-b53622397518" (UID: "fabd1a0e-367b-4782-aa73-b53622397518"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:22:40.599962 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:40.599932    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fabd1a0e-367b-4782-aa73-b53622397518-model-cache\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:22:40.599962 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:40.599942    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fabd1a0e-367b-4782-aa73-b53622397518-home" (OuterVolumeSpecName: "home") pod "fabd1a0e-367b-4782-aa73-b53622397518" (UID: "fabd1a0e-367b-4782-aa73-b53622397518"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:22:40.601920 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:40.601893    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fabd1a0e-367b-4782-aa73-b53622397518-dshm" (OuterVolumeSpecName: "dshm") pod "fabd1a0e-367b-4782-aa73-b53622397518" (UID: "fabd1a0e-367b-4782-aa73-b53622397518"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:22:40.601920 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:40.601907    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fabd1a0e-367b-4782-aa73-b53622397518-kube-api-access-22s4m" (OuterVolumeSpecName: "kube-api-access-22s4m") pod "fabd1a0e-367b-4782-aa73-b53622397518" (UID: "fabd1a0e-367b-4782-aa73-b53622397518"). InnerVolumeSpecName "kube-api-access-22s4m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:22:40.601920 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:40.601917    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fabd1a0e-367b-4782-aa73-b53622397518-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fabd1a0e-367b-4782-aa73-b53622397518" (UID: "fabd1a0e-367b-4782-aa73-b53622397518"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:22:40.701177 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:40.701136    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fabd1a0e-367b-4782-aa73-b53622397518-tls-certs\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:22:40.701177 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:40.701168    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22s4m\" (UniqueName: \"kubernetes.io/projected/fabd1a0e-367b-4782-aa73-b53622397518-kube-api-access-22s4m\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:22:40.701177 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:40.701177    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fabd1a0e-367b-4782-aa73-b53622397518-dshm\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:22:40.701177 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:40.701185    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fabd1a0e-367b-4782-aa73-b53622397518-tmp-dir\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:22:40.701479 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:40.701193    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fabd1a0e-367b-4782-aa73-b53622397518-home\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:22:41.402310 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:41.402283    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-wkq9c"
Sep 09 21:22:41.402310 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:41.402297    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-wkq9c" event={"ID":"fabd1a0e-367b-4782-aa73-b53622397518","Type":"ContainerDied","Data":"59e49cebfaeb6275fcae60446901c1b02d5a097b0be461c2abeb005c59817dce"}
Sep 09 21:22:41.402863 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:41.402420    2576 scope.go:117] "RemoveContainer" containerID="a72b3dc2deaf39c862ba153d4fd597dcdc306e33142df81a507dc7309e89b9fb"
Sep 09 21:22:41.409630 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:41.409603    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cf8a27c9-0a46-49f1-afa9-e35a5bac691e" path="/var/lib/kubelet/pods/cf8a27c9-0a46-49f1-afa9-e35a5bac691e/volumes"
Sep 09 21:22:41.429068 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:41.429039    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-wkq9c"]
Sep 09 21:22:41.434701 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:41.434677    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-wkq9c"]
Sep 09 21:22:42.879614 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:42.879583    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-9kz62"]
Sep 09 21:22:42.879964 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:42.879898    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fabd1a0e-367b-4782-aa73-b53622397518" containerName="main"
Sep 09 21:22:42.879964 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:42.879908    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="fabd1a0e-367b-4782-aa73-b53622397518" containerName="main"
Sep 09 21:22:42.879964 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:42.879928    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf8a27c9-0a46-49f1-afa9-e35a5bac691e" containerName="main"
Sep 09 21:22:42.879964 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:42.879934    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf8a27c9-0a46-49f1-afa9-e35a5bac691e" containerName="main"
Sep 09 21:22:42.880098 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:42.879994    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="fabd1a0e-367b-4782-aa73-b53622397518" containerName="main"
Sep 09 21:22:42.880098 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:42.880004    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf8a27c9-0a46-49f1-afa9-e35a5bac691e" containerName="main"
Sep 09 21:22:42.883999 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:42.883980    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-9kz62"
Sep 09 21:22:42.886963 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:42.886945    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-model-name-divergence-75f1ef44\"/\"test-div-v2-kserve-self-signed-certs\""
Sep 09 21:22:42.897800 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:42.897775    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-9kz62"]
Sep 09 21:22:42.921682 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:42.921642    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:22:43.022072 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:43.022036    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qgmr\" (UniqueName: \"kubernetes.io/projected/15733211-6c01-4839-b5d2-5c8add4ea164-kube-api-access-5qgmr\") pod \"test-div-v2-kserve-5f6fdb6b77-9kz62\" (UID: \"15733211-6c01-4839-b5d2-5c8add4ea164\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-9kz62"
Sep 09 21:22:43.022229 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:43.022097    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15733211-6c01-4839-b5d2-5c8add4ea164-tmp-dir\") pod \"test-div-v2-kserve-5f6fdb6b77-9kz62\" (UID: \"15733211-6c01-4839-b5d2-5c8add4ea164\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-9kz62"
Sep 09 21:22:43.022229 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:43.022149    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15733211-6c01-4839-b5d2-5c8add4ea164-tls-certs\") pod \"test-div-v2-kserve-5f6fdb6b77-9kz62\" (UID: \"15733211-6c01-4839-b5d2-5c8add4ea164\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-9kz62"
Sep 09 21:22:43.022229 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:43.022185    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15733211-6c01-4839-b5d2-5c8add4ea164-home\") pod \"test-div-v2-kserve-5f6fdb6b77-9kz62\" (UID: \"15733211-6c01-4839-b5d2-5c8add4ea164\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-9kz62"
Sep 09 21:22:43.022229 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:43.022217    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15733211-6c01-4839-b5d2-5c8add4ea164-dshm\") pod \"test-div-v2-kserve-5f6fdb6b77-9kz62\" (UID: \"15733211-6c01-4839-b5d2-5c8add4ea164\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-9kz62"
Sep 09 21:22:43.022413 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:43.022235    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15733211-6c01-4839-b5d2-5c8add4ea164-model-cache\") pod \"test-div-v2-kserve-5f6fdb6b77-9kz62\" (UID: \"15733211-6c01-4839-b5d2-5c8add4ea164\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-9kz62"
Sep 09 21:22:43.123102 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:43.123072    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15733211-6c01-4839-b5d2-5c8add4ea164-dshm\") pod \"test-div-v2-kserve-5f6fdb6b77-9kz62\" (UID: \"15733211-6c01-4839-b5d2-5c8add4ea164\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-9kz62"
Sep 09 21:22:43.123102 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:43.123106    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15733211-6c01-4839-b5d2-5c8add4ea164-model-cache\") pod \"test-div-v2-kserve-5f6fdb6b77-9kz62\" (UID: \"15733211-6c01-4839-b5d2-5c8add4ea164\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-9kz62"
Sep 09 21:22:43.123349 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:43.123157    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qgmr\" (UniqueName: \"kubernetes.io/projected/15733211-6c01-4839-b5d2-5c8add4ea164-kube-api-access-5qgmr\") pod \"test-div-v2-kserve-5f6fdb6b77-9kz62\" (UID: \"15733211-6c01-4839-b5d2-5c8add4ea164\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-9kz62"
Sep 09 21:22:43.123349 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:43.123197    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15733211-6c01-4839-b5d2-5c8add4ea164-tmp-dir\") pod \"test-div-v2-kserve-5f6fdb6b77-9kz62\" (UID: \"15733211-6c01-4839-b5d2-5c8add4ea164\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-9kz62"
Sep 09 21:22:43.123349 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:43.123215    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15733211-6c01-4839-b5d2-5c8add4ea164-tls-certs\") pod \"test-div-v2-kserve-5f6fdb6b77-9kz62\" (UID: \"15733211-6c01-4839-b5d2-5c8add4ea164\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-9kz62"
Sep 09 21:22:43.123349 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:43.123231    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15733211-6c01-4839-b5d2-5c8add4ea164-home\") pod \"test-div-v2-kserve-5f6fdb6b77-9kz62\" (UID: \"15733211-6c01-4839-b5d2-5c8add4ea164\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-9kz62"
Sep 09 21:22:43.123647 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:43.123621    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15733211-6c01-4839-b5d2-5c8add4ea164-model-cache\") pod \"test-div-v2-kserve-5f6fdb6b77-9kz62\" (UID: \"15733211-6c01-4839-b5d2-5c8add4ea164\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-9kz62"
Sep 09 21:22:43.123723 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:43.123671    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15733211-6c01-4839-b5d2-5c8add4ea164-tmp-dir\") pod \"test-div-v2-kserve-5f6fdb6b77-9kz62\" (UID: \"15733211-6c01-4839-b5d2-5c8add4ea164\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-9kz62"
Sep 09 21:22:43.123723 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:43.123683    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15733211-6c01-4839-b5d2-5c8add4ea164-home\") pod \"test-div-v2-kserve-5f6fdb6b77-9kz62\" (UID: \"15733211-6c01-4839-b5d2-5c8add4ea164\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-9kz62"
Sep 09 21:22:43.125833 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:43.125811    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15733211-6c01-4839-b5d2-5c8add4ea164-tls-certs\") pod \"test-div-v2-kserve-5f6fdb6b77-9kz62\" (UID: \"15733211-6c01-4839-b5d2-5c8add4ea164\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-9kz62"
Sep 09 21:22:43.125966 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:43.125949    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15733211-6c01-4839-b5d2-5c8add4ea164-dshm\") pod \"test-div-v2-kserve-5f6fdb6b77-9kz62\" (UID: \"15733211-6c01-4839-b5d2-5c8add4ea164\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-9kz62"
Sep 09 21:22:43.132653 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:43.132589    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qgmr\" (UniqueName: \"kubernetes.io/projected/15733211-6c01-4839-b5d2-5c8add4ea164-kube-api-access-5qgmr\") pod \"test-div-v2-kserve-5f6fdb6b77-9kz62\" (UID: \"15733211-6c01-4839-b5d2-5c8add4ea164\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-9kz62"
Sep 09 21:22:43.193815 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:43.193781    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-9kz62"
Sep 09 21:22:43.409618 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:43.409544    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fabd1a0e-367b-4782-aa73-b53622397518" path="/var/lib/kubelet/pods/fabd1a0e-367b-4782-aa73-b53622397518/volumes"
Sep 09 21:22:43.472118 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:43.472095    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-9kz62"]
Sep 09 21:22:43.474026 ip-10-0-134-72 kubenswrapper[2576]: W0909 21:22:43.473982    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15733211_6c01_4839_b5d2_5c8add4ea164.slice/crio-70979c6e19cf4f8ff2883b20294312a2ef29b10d03df68fe6b61dd3d0dfbe7fb WatchSource:0}: Error finding container 70979c6e19cf4f8ff2883b20294312a2ef29b10d03df68fe6b61dd3d0dfbe7fb: Status 404 returned error can't find the container with id 70979c6e19cf4f8ff2883b20294312a2ef29b10d03df68fe6b61dd3d0dfbe7fb
Sep 09 21:22:44.420896 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:44.420861    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-9kz62" event={"ID":"15733211-6c01-4839-b5d2-5c8add4ea164","Type":"ContainerStarted","Data":"107bad8a03a9016b1007067ab11b35f8a184cc02fcb02562a00a4f595b3c4b3d"}
Sep 09 21:22:44.421347 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:44.420906    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-9kz62" event={"ID":"15733211-6c01-4839-b5d2-5c8add4ea164","Type":"ContainerStarted","Data":"70979c6e19cf4f8ff2883b20294312a2ef29b10d03df68fe6b61dd3d0dfbe7fb"}
Sep 09 21:22:44.442542 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:44.442492    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-9kz62" podStartSLOduration=2.442474857 podStartE2EDuration="2.442474857s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:22:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:22:44.440503427 +0000 UTC m=+1807.554189285" watchObservedRunningTime="2026-09-09 21:22:44.442474857 +0000 UTC m=+1807.556160715"
Sep 09 21:22:45.194176 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:45.194140    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-9kz62"
Sep 09 21:22:48.136104 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:48.136068    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:22:48.237129 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:48.237084    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:22:48.489229 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:48.489128    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:22:48.926089 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:48.926061    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-6jzgz"
Sep 09 21:22:48.938271 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:48.938242    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-6jzgz"
Sep 09 21:22:49.466101 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:49.466074    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-6jzgz"
Sep 09 21:22:53.194671 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:53.194637    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-9kz62"
Sep 09 21:22:53.207067 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:53.207045    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-9kz62"
Sep 09 21:22:53.487398 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:53.487302    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-9kz62"
Sep 09 21:22:55.164704 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:22:55.164665    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:23:01.170540 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:23:01.170507    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:23:01.239275 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:23:01.239241    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:23:07.611287 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:23:07.611248    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:23:15.054788 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:23:15.054752    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-dsl6w"]
Sep 09 21:23:15.058805 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:23:15.058783    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-dsl6w"
Sep 09 21:23:15.062394 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:23:15.062345    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-kserve-self-signed-certs\""
Sep 09 21:23:15.062544 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:23:15.062398    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"openshift-service-ca.crt\""
Sep 09 21:23:15.062544 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:23:15.062396    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"default-dockercfg-lqht4\""
Sep 09 21:23:15.062544 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:23:15.062425    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"kube-root-ca.crt\""
Sep 09 21:23:15.062544 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:23:15.062396    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-kserve-dockercfg-k5hl9\""
Sep 09 21:23:15.074036 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:23:15.074017    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-dsl6w"]
Sep 09 21:23:15.086938 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:23:15.086915    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea219f5d-ce68-450a-8c70-e359e0207228-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-dsl6w\" (UID: \"ea219f5d-ce68-450a-8c70-e359e0207228\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-dsl6w"
Sep 09 21:23:15.087038 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:23:15.086968    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea219f5d-ce68-450a-8c70-e359e0207228-home\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-dsl6w\" (UID: \"ea219f5d-ce68-450a-8c70-e359e0207228\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-dsl6w"
Sep 09 21:23:15.087038 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:23:15.087016    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea219f5d-ce68-450a-8c70-e359e0207228-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-dsl6w\" (UID: \"ea219f5d-ce68-450a-8c70-e359e0207228\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-dsl6w"
Sep 09 21:23:15.087132 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:23:15.087043    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29d4v\" (UniqueName: \"kubernetes.io/projected/ea219f5d-ce68-450a-8c70-e359e0207228-kube-api-access-29d4v\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-dsl6w\" (UID: \"ea219f5d-ce68-450a-8c70-e359e0207228\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-dsl6w"
Sep 09 21:23:15.087132 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:23:15.087078    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea219f5d-ce68-450a-8c70-e359e0207228-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-dsl6w\" (UID: \"ea219f5d-ce68-450a-8c70-e359e0207228\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-dsl6w"
Sep 09 21:23:15.087132 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:23:15.087109    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea219f5d-ce68-450a-8c70-e359e0207228-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-dsl6w\" (UID: \"ea219f5d-ce68-450a-8c70-e359e0207228\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-dsl6w"
Sep 09 21:23:15.113132 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:23:15.113101    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:23:15.188018 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:23:15.187982    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea219f5d-ce68-450a-8c70-e359e0207228-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-dsl6w\" (UID: \"ea219f5d-ce68-450a-8c70-e359e0207228\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-dsl6w"
Sep 09 21:23:15.188166 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:23:15.188028    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea219f5d-ce68-450a-8c70-e359e0207228-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-dsl6w\" (UID: \"ea219f5d-ce68-450a-8c70-e359e0207228\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-dsl6w"
Sep 09 21:23:15.188166 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:23:15.188100    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea219f5d-ce68-450a-8c70-e359e0207228-home\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-dsl6w\" (UID: \"ea219f5d-ce68-450a-8c70-e359e0207228\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-dsl6w"
Sep 09 21:23:15.188166 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:23:15.188145    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea219f5d-ce68-450a-8c70-e359e0207228-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-dsl6w\" (UID: \"ea219f5d-ce68-450a-8c70-e359e0207228\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-dsl6w"
Sep 09 21:23:15.188335 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:23:15.188173    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29d4v\" (UniqueName: \"kubernetes.io/projected/ea219f5d-ce68-450a-8c70-e359e0207228-kube-api-access-29d4v\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-dsl6w\" (UID: \"ea219f5d-ce68-450a-8c70-e359e0207228\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-dsl6w"
Sep 09 21:23:15.188335 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:23:15.188202    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea219f5d-ce68-450a-8c70-e359e0207228-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-dsl6w\" (UID: \"ea219f5d-ce68-450a-8c70-e359e0207228\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-dsl6w"
Sep 09 21:23:15.188504 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:23:15.188426    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea219f5d-ce68-450a-8c70-e359e0207228-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-dsl6w\" (UID: \"ea219f5d-ce68-450a-8c70-e359e0207228\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-dsl6w"
Sep 09 21:23:15.188565 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:23:15.188527    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea219f5d-ce68-450a-8c70-e359e0207228-home\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-dsl6w\" (UID: \"ea219f5d-ce68-450a-8c70-e359e0207228\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-dsl6w"
Sep 09 21:23:15.188635 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:23:15.188609    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea219f5d-ce68-450a-8c70-e359e0207228-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-dsl6w\" (UID: \"ea219f5d-ce68-450a-8c70-e359e0207228\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-dsl6w"
Sep 09 21:23:15.190596 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:23:15.190574    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea219f5d-ce68-450a-8c70-e359e0207228-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-dsl6w\" (UID: \"ea219f5d-ce68-450a-8c70-e359e0207228\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-dsl6w"
Sep 09 21:23:15.190723 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:23:15.190705    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea219f5d-ce68-450a-8c70-e359e0207228-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-dsl6w\" (UID: \"ea219f5d-ce68-450a-8c70-e359e0207228\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-dsl6w"
Sep 09 21:23:15.197101 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:23:15.197077    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-29d4v\" (UniqueName: \"kubernetes.io/projected/ea219f5d-ce68-450a-8c70-e359e0207228-kube-api-access-29d4v\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-dsl6w\" (UID: \"ea219f5d-ce68-450a-8c70-e359e0207228\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-dsl6w"
Sep 09 21:23:15.368506 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:23:15.368475    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-dsl6w"
Sep 09 21:23:15.395783 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:23:15.395748    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:23:15.650836 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:23:15.650806    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-dsl6w"]
Sep 09 21:23:15.652204 ip-10-0-134-72 kubenswrapper[2576]: W0909 21:23:15.652177    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea219f5d_ce68_450a_8c70_e359e0207228.slice/crio-f9c0bb93db16d6ca173fcb245778b381933c14e32f35a7e07b8e7f210ab33e68 WatchSource:0}: Error finding container f9c0bb93db16d6ca173fcb245778b381933c14e32f35a7e07b8e7f210ab33e68: Status 404 returned error can't find the container with id f9c0bb93db16d6ca173fcb245778b381933c14e32f35a7e07b8e7f210ab33e68
Sep 09 21:23:15.654100 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:23:15.654085    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 21:23:16.594698 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:23:16.594666    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-dsl6w" event={"ID":"ea219f5d-ce68-450a-8c70-e359e0207228","Type":"ContainerStarted","Data":"f9c0bb93db16d6ca173fcb245778b381933c14e32f35a7e07b8e7f210ab33e68"}
Sep 09 21:23:17.601010 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:23:17.600977    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-dsl6w" event={"ID":"ea219f5d-ce68-450a-8c70-e359e0207228","Type":"ContainerStarted","Data":"2488b50490c49c80d17221af3d3ec466f2e0032d21e45e8d82266f88a78bdb7e"}
Sep 09 21:23:17.601438 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:23:17.601140    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-dsl6w"
Sep 09 21:23:18.607895 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:23:18.607864    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-dsl6w" event={"ID":"ea219f5d-ce68-450a-8c70-e359e0207228","Type":"ContainerStarted","Data":"513a5597778907c0e58b7efbfffeba4f892d61b66eab96d0b4358a3a890ade61"}
Sep 09 21:23:18.630949 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:23:18.630907    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-dsl6w" podStartSLOduration=1.9198878430000001 podStartE2EDuration="3.630892476s" totalImagesPullingTime="1.711004633s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:23:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:23:18.628909895 +0000 UTC m=+1841.742595745" watchObservedRunningTime="2026-09-09 21:23:18.630892476 +0000 UTC m=+1841.744578335"
Sep 09 21:23:19.369131 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:23:19.369095    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-dsl6w"
Sep 09 21:23:19.416213 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:23:19.416179    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:23:19.522592 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:23:19.522557    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:23:19.627958 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:23:19.627857    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:23:24.395075 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:23:24.395038    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-6jzgz"]
Sep 09 21:23:24.401193 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:23:24.401164    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-9kz62"]
Sep 09 21:23:24.488900 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:23:24.488869    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:23:25.368924 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:23:25.368894    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-dsl6w"
Sep 09 21:23:25.380931 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:23:25.380903    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-dsl6w"
Sep 09 21:23:25.653600 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:23:25.653517    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-dsl6w"
Sep 09 21:23:26.898189 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:23:26.898149    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:23:28.637486 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:23:28.637444    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:23:29.211786 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:23:29.211756    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:23:36.662980 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:23:36.662947    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-dsl6w"
Sep 09 21:23:38.328614 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:23:38.328572    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:23:39.415242 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:23:39.415205    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-6jzgz" podUID="2c557d9d-e865-408e-b7a7-42281ff091a4" containerName="main" containerID="cri-o://6c46d7b6e2d3ef6fab4e5a89f5f60d2bcfc35917389df5606c55d401d67e9a6e" gracePeriod=45
Sep 09 21:23:39.420218 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:23:39.420163    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-9kz62" podUID="15733211-6c01-4839-b5d2-5c8add4ea164" containerName="main" containerID="cri-o://107bad8a03a9016b1007067ab11b35f8a184cc02fcb02562a00a4f595b3c4b3d" gracePeriod=45
Sep 09 21:23:39.455966 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:23:39.455932    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-6jzgz" podUID="2c557d9d-e865-408e-b7a7-42281ff091a4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": dial tcp 10.132.0.58:8000: connect: connection refused"
Sep 09 21:23:39.476970 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:23:39.476938    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-9kz62" podUID="15733211-6c01-4839-b5d2-5c8add4ea164" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": dial tcp 10.132.0.59:8000: connect: connection refused"
Sep 09 21:23:39.693960 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:23:39.693929    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-6jzgz"
Sep 09 21:23:39.695887 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:23:39.695867    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-9kz62"
Sep 09 21:23:39.715253 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:23:39.715233    2576 generic.go:356] "Generic (PLEG): container finished" podID="15733211-6c01-4839-b5d2-5c8add4ea164" containerID="107bad8a03a9016b1007067ab11b35f8a184cc02fcb02562a00a4f595b3c4b3d" exitCode=0
Sep 09 21:23:39.715361 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:23:39.715269    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-9kz62"
Sep 09 21:23:39.715361 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:23:39.715315    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-9kz62" event={"ID":"15733211-6c01-4839-b5d2-5c8add4ea164","Type":"ContainerDied","Data":"107bad8a03a9016b1007067ab11b35f8a184cc02fcb02562a00a4f595b3c4b3d"}
Sep 09 21:23:39.715477 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:23:39.715399    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-9kz62" event={"ID":"15733211-6c01-4839-b5d2-5c8add4ea164","Type":"ContainerDied","Data":"70979c6e19cf4f8ff2883b20294312a2ef29b10d03df68fe6b61dd3d0dfbe7fb"}
Sep 09 21:23:39.715477 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:23:39.715445    2576 scope.go:117] "RemoveContainer" containerID="107bad8a03a9016b1007067ab11b35f8a184cc02fcb02562a00a4f595b3c4b3d"
Sep 09 21:23:39.716612 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:23:39.716589    2576 generic.go:356] "Generic (PLEG): container finished" podID="2c557d9d-e865-408e-b7a7-42281ff091a4" containerID="6c46d7b6e2d3ef6fab4e5a89f5f60d2bcfc35917389df5606c55d401d67e9a6e" exitCode=0
Sep 09 21:23:39.716709 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:23:39.716624    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-6jzgz"
Sep 09 21:23:39.716709 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:23:39.716680    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-6jzgz" event={"ID":"2c557d9d-e865-408e-b7a7-42281ff091a4","Type":"ContainerDied","Data":"6c46d7b6e2d3ef6fab4e5a89f5f60d2bcfc35917389df5606c55d401d67e9a6e"}
Sep 09 21:23:39.716829 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:23:39.716744    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-6jzgz" event={"ID":"2c557d9d-e865-408e-b7a7-42281ff091a4","Type":"ContainerDied","Data":"7967f9a72d9740d74a24b93c3a28ef1efaf294e0fb901f5f1f9d9e103074986e"}
Sep 09 21:23:39.725614 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:23:39.725596    2576 scope.go:117] "RemoveContainer" containerID="107bad8a03a9016b1007067ab11b35f8a184cc02fcb02562a00a4f595b3c4b3d"
Sep 09 21:23:39.725897 ip-10-0-134-72 kubenswrapper[2576]: E0909 21:23:39.725882    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"107bad8a03a9016b1007067ab11b35f8a184cc02fcb02562a00a4f595b3c4b3d\": container with ID starting with 107bad8a03a9016b1007067ab11b35f8a184cc02fcb02562a00a4f595b3c4b3d not found: ID does not exist" containerID="107bad8a03a9016b1007067ab11b35f8a184cc02fcb02562a00a4f595b3c4b3d"
Sep 09 21:23:39.725955 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:23:39.725902    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"107bad8a03a9016b1007067ab11b35f8a184cc02fcb02562a00a4f595b3c4b3d"} err="failed to get container status \"107bad8a03a9016b1007067ab11b35f8a184cc02fcb02562a00a4f595b3c4b3d\": rpc error: code = NotFound desc = could not find container \"107bad8a03a9016b1007067ab11b35f8a184cc02fcb02562a00a4f595b3c4b3d\": container with ID starting with 107bad8a03a9016b1007067ab11b35f8a184cc02fcb02562a00a4f595b3c4b3d not found: ID does not exist"
Sep 09 21:23:39.725955 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:23:39.725917    2576 scope.go:117] "RemoveContainer" containerID="6c46d7b6e2d3ef6fab4e5a89f5f60d2bcfc35917389df5606c55d401d67e9a6e"
Sep 09 21:23:39.734147 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:23:39.734123    2576 scope.go:117] "RemoveContainer" containerID="6c46d7b6e2d3ef6fab4e5a89f5f60d2bcfc35917389df5606c55d401d67e9a6e"
Sep 09 21:23:39.734446 ip-10-0-134-72 kubenswrapper[2576]: E0909 21:23:39.734426    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c46d7b6e2d3ef6fab4e5a89f5f60d2bcfc35917389df5606c55d401d67e9a6e\": container with ID starting with 6c46d7b6e2d3ef6fab4e5a89f5f60d2bcfc35917389df5606c55d401d67e9a6e not found: ID does not exist" containerID="6c46d7b6e2d3ef6fab4e5a89f5f60d2bcfc35917389df5606c55d401d67e9a6e"
Sep 09 21:23:39.734518 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:23:39.734463    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c46d7b6e2d3ef6fab4e5a89f5f60d2bcfc35917389df5606c55d401d67e9a6e"} err="failed to get container status \"6c46d7b6e2d3ef6fab4e5a89f5f60d2bcfc35917389df5606c55d401d67e9a6e\": rpc error: code = NotFound desc = could not find container \"6c46d7b6e2d3ef6fab4e5a89f5f60d2bcfc35917389df5606c55d401d67e9a6e\": container with ID starting with 6c46d7b6e2d3ef6fab4e5a89f5f60d2bcfc35917389df5606c55d401d67e9a6e not found: ID does not exist"
Sep 09 21:23:39.802350 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:23:39.802323    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15733211-6c01-4839-b5d2-5c8add4ea164-home\") pod \"15733211-6c01-4839-b5d2-5c8add4ea164\" (UID: \"15733211-6c01-4839-b5d2-5c8add4ea164\") "
Sep 09 21:23:39.802514 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:23:39.802402    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c557d9d-e865-408e-b7a7-42281ff091a4-tmp-dir\") pod \"2c557d9d-e865-408e-b7a7-42281ff091a4\" (UID: \"2c557d9d-e865-408e-b7a7-42281ff091a4\") "
Sep 09 21:23:39.802514 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:23:39.802435    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c557d9d-e865-408e-b7a7-42281ff091a4-tls-certs\") pod \"2c557d9d-e865-408e-b7a7-42281ff091a4\" (UID: \"2c557d9d-e865-408e-b7a7-42281ff091a4\") "
Sep 09 21:23:39.802514 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:23:39.802479    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c557d9d-e865-408e-b7a7-42281ff091a4-dshm\") pod \"2c557d9d-e865-408e-b7a7-42281ff091a4\" (UID: \"2c557d9d-e865-408e-b7a7-42281ff091a4\") "
Sep 09 21:23:39.802514 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:23:39.802502    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15733211-6c01-4839-b5d2-5c8add4ea164-model-cache\") pod \"15733211-6c01-4839-b5d2-5c8add4ea164\" (UID: \"15733211-6c01-4839-b5d2-5c8add4ea164\") "
Sep 09 21:23:39.802721 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:23:39.802529    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c557d9d-e865-408e-b7a7-42281ff091a4-home\") pod \"2c557d9d-e865-408e-b7a7-42281ff091a4\" (UID: \"2c557d9d-e865-408e-b7a7-42281ff091a4\") "
Sep 09 21:23:39.802721 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:23:39.802561    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15733211-6c01-4839-b5d2-5c8add4ea164-tmp-dir\") pod \"15733211-6c01-4839-b5d2-5c8add4ea164\" (UID: \"15733211-6c01-4839-b5d2-5c8add4ea164\") "
Sep 09 21:23:39.802721 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:23:39.802586    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5qgmr\" (UniqueName: \"kubernetes.io/projected/15733211-6c01-4839-b5d2-5c8add4ea164-kube-api-access-5qgmr\") pod \"15733211-6c01-4839-b5d2-5c8add4ea164\" (UID: \"15733211-6c01-4839-b5d2-5c8add4ea164\") "
Sep 09 21:23:39.802721 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:23:39.802602    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15733211-6c01-4839-b5d2-5c8add4ea164-home" (OuterVolumeSpecName: "home") pod "15733211-6c01-4839-b5d2-5c8add4ea164" (UID: "15733211-6c01-4839-b5d2-5c8add4ea164"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:23:39.802721 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:23:39.802607    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15733211-6c01-4839-b5d2-5c8add4ea164-tls-certs\") pod \"15733211-6c01-4839-b5d2-5c8add4ea164\" (UID: \"15733211-6c01-4839-b5d2-5c8add4ea164\") "
Sep 09 21:23:39.802721 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:23:39.802701    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jv98c\" (UniqueName: \"kubernetes.io/projected/2c557d9d-e865-408e-b7a7-42281ff091a4-kube-api-access-jv98c\") pod \"2c557d9d-e865-408e-b7a7-42281ff091a4\" (UID: \"2c557d9d-e865-408e-b7a7-42281ff091a4\") "
Sep 09 21:23:39.802721 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:23:39.802711    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c557d9d-e865-408e-b7a7-42281ff091a4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2c557d9d-e865-408e-b7a7-42281ff091a4" (UID: "2c557d9d-e865-408e-b7a7-42281ff091a4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:23:39.803084 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:23:39.802732    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15733211-6c01-4839-b5d2-5c8add4ea164-dshm\") pod \"15733211-6c01-4839-b5d2-5c8add4ea164\" (UID: \"15733211-6c01-4839-b5d2-5c8add4ea164\") "
Sep 09 21:23:39.803084 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:23:39.802764    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15733211-6c01-4839-b5d2-5c8add4ea164-model-cache" (OuterVolumeSpecName: "model-cache") pod "15733211-6c01-4839-b5d2-5c8add4ea164" (UID: "15733211-6c01-4839-b5d2-5c8add4ea164"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:23:39.803084 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:23:39.802785    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15733211-6c01-4839-b5d2-5c8add4ea164-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "15733211-6c01-4839-b5d2-5c8add4ea164" (UID: "15733211-6c01-4839-b5d2-5c8add4ea164"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:23:39.803084 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:23:39.802794    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c557d9d-e865-408e-b7a7-42281ff091a4-model-cache\") pod \"2c557d9d-e865-408e-b7a7-42281ff091a4\" (UID: \"2c557d9d-e865-408e-b7a7-42281ff091a4\") "
Sep 09 21:23:39.803084 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:23:39.802847    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c557d9d-e865-408e-b7a7-42281ff091a4-home" (OuterVolumeSpecName: "home") pod "2c557d9d-e865-408e-b7a7-42281ff091a4" (UID: "2c557d9d-e865-408e-b7a7-42281ff091a4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:23:39.803084 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:23:39.802983    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c557d9d-e865-408e-b7a7-42281ff091a4-model-cache" (OuterVolumeSpecName: "model-cache") pod "2c557d9d-e865-408e-b7a7-42281ff091a4" (UID: "2c557d9d-e865-408e-b7a7-42281ff091a4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:23:39.803455 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:23:39.803123    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15733211-6c01-4839-b5d2-5c8add4ea164-home\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:23:39.803455 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:23:39.803140    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c557d9d-e865-408e-b7a7-42281ff091a4-tmp-dir\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:23:39.803455 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:23:39.803153    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15733211-6c01-4839-b5d2-5c8add4ea164-model-cache\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:23:39.803455 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:23:39.803167    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c557d9d-e865-408e-b7a7-42281ff091a4-home\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:23:39.803455 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:23:39.803178    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15733211-6c01-4839-b5d2-5c8add4ea164-tmp-dir\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:23:39.803455 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:23:39.803190    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c557d9d-e865-408e-b7a7-42281ff091a4-model-cache\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:23:39.805226 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:23:39.805197    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15733211-6c01-4839-b5d2-5c8add4ea164-kube-api-access-5qgmr" (OuterVolumeSpecName: "kube-api-access-5qgmr") pod "15733211-6c01-4839-b5d2-5c8add4ea164" (UID: "15733211-6c01-4839-b5d2-5c8add4ea164"). InnerVolumeSpecName "kube-api-access-5qgmr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:23:39.805331 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:23:39.805273    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c557d9d-e865-408e-b7a7-42281ff091a4-kube-api-access-jv98c" (OuterVolumeSpecName: "kube-api-access-jv98c") pod "2c557d9d-e865-408e-b7a7-42281ff091a4" (UID: "2c557d9d-e865-408e-b7a7-42281ff091a4"). InnerVolumeSpecName "kube-api-access-jv98c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:23:39.805438 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:23:39.805392    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15733211-6c01-4839-b5d2-5c8add4ea164-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "15733211-6c01-4839-b5d2-5c8add4ea164" (UID: "15733211-6c01-4839-b5d2-5c8add4ea164"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:23:39.805438 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:23:39.805400    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15733211-6c01-4839-b5d2-5c8add4ea164-dshm" (OuterVolumeSpecName: "dshm") pod "15733211-6c01-4839-b5d2-5c8add4ea164" (UID: "15733211-6c01-4839-b5d2-5c8add4ea164"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:23:39.805572 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:23:39.805554    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c557d9d-e865-408e-b7a7-42281ff091a4-dshm" (OuterVolumeSpecName: "dshm") pod "2c557d9d-e865-408e-b7a7-42281ff091a4" (UID: "2c557d9d-e865-408e-b7a7-42281ff091a4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:23:39.805609 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:23:39.805584    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c557d9d-e865-408e-b7a7-42281ff091a4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2c557d9d-e865-408e-b7a7-42281ff091a4" (UID: "2c557d9d-e865-408e-b7a7-42281ff091a4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:23:39.903802 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:23:39.903773    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c557d9d-e865-408e-b7a7-42281ff091a4-tls-certs\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:23:39.903802 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:23:39.903796    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c557d9d-e865-408e-b7a7-42281ff091a4-dshm\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:23:39.903802 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:23:39.903805    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qgmr\" (UniqueName: \"kubernetes.io/projected/15733211-6c01-4839-b5d2-5c8add4ea164-kube-api-access-5qgmr\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:23:39.903985 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:23:39.903814    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15733211-6c01-4839-b5d2-5c8add4ea164-tls-certs\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:23:39.903985 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:23:39.903823    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jv98c\" (UniqueName: \"kubernetes.io/projected/2c557d9d-e865-408e-b7a7-42281ff091a4-kube-api-access-jv98c\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:23:39.903985 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:23:39.903830    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15733211-6c01-4839-b5d2-5c8add4ea164-dshm\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:23:40.053971 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:23:40.053932    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-9kz62"]
Sep 09 21:23:40.055967 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:23:40.055943    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-9kz62"]
Sep 09 21:23:40.076537 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:23:40.076507    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-6jzgz"]
Sep 09 21:23:40.083065 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:23:40.083045    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-6jzgz"]
Sep 09 21:23:41.409334 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:23:41.409301    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="15733211-6c01-4839-b5d2-5c8add4ea164" path="/var/lib/kubelet/pods/15733211-6c01-4839-b5d2-5c8add4ea164/volumes"
Sep 09 21:23:41.409705 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:23:41.409686    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c557d9d-e865-408e-b7a7-42281ff091a4" path="/var/lib/kubelet/pods/2c557d9d-e865-408e-b7a7-42281ff091a4/volumes"
Sep 09 21:23:48.110001 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:23:48.109972    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:24:02.169778 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:24:02.169678    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:24:08.122117 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:24:08.122084    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:24:08.200201 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:24:08.200175    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:24:10.916427 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:24:10.916393    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:24:10.968502 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:24:10.968470    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-dsl6w"]
Sep 09 21:24:10.977541 ip-10-0-134-72 kubenswrapper[2576]: E0909 21:24:10.977509    2576 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 09 21:24:10.977541 ip-10-0-134-72 kubenswrapper[2576]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 21:24:10.977541 ip-10-0-134-72 kubenswrapper[2576]:         , stdout: , stderr: , exit code -1
Sep 09 21:24:10.977541 ip-10-0-134-72 kubenswrapper[2576]:  > containerID="513a5597778907c0e58b7efbfffeba4f892d61b66eab96d0b4358a3a890ade61" cmd=["/bin/sleep","15"]
Sep 09 21:24:10.977773 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:24:10.977559    2576 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-dsl6w" message="" err=<
Sep 09 21:24:10.977773 ip-10-0-134-72 kubenswrapper[2576]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 21:24:10.977773 ip-10-0-134-72 kubenswrapper[2576]:         , stdout: , stderr: , exit code -1
Sep 09 21:24:10.977773 ip-10-0-134-72 kubenswrapper[2576]:  >
Sep 09 21:24:10.977773 ip-10-0-134-72 kubenswrapper[2576]: E0909 21:24:10.977571    2576 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 09 21:24:10.977773 ip-10-0-134-72 kubenswrapper[2576]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 21:24:10.977773 ip-10-0-134-72 kubenswrapper[2576]:         , stdout: , stderr: , exit code -1
Sep 09 21:24:10.977773 ip-10-0-134-72 kubenswrapper[2576]:  > pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-dsl6w" podUID="ea219f5d-ce68-450a-8c70-e359e0207228" containerName="main" containerID="cri-o://513a5597778907c0e58b7efbfffeba4f892d61b66eab96d0b4358a3a890ade61"
Sep 09 21:24:10.977773 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:24:10.977605    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-dsl6w" podUID="ea219f5d-ce68-450a-8c70-e359e0207228" containerName="main" containerID="cri-o://513a5597778907c0e58b7efbfffeba4f892d61b66eab96d0b4358a3a890ade61" gracePeriod=60
Sep 09 21:24:10.995137 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:24:10.995107    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:24:11.194203 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:24:11.194172    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-dsl6w" podUID="ea219f5d-ce68-450a-8c70-e359e0207228" containerName="llm-d-routing-sidecar" containerID="cri-o://2488b50490c49c80d17221af3d3ec466f2e0032d21e45e8d82266f88a78bdb7e" gracePeriod=60
Sep 09 21:24:11.440747 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:24:11.440724    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-dsl6w"
Sep 09 21:24:11.462845 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:24:11.462793    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea219f5d-ce68-450a-8c70-e359e0207228-tls-certs\") pod \"ea219f5d-ce68-450a-8c70-e359e0207228\" (UID: \"ea219f5d-ce68-450a-8c70-e359e0207228\") "
Sep 09 21:24:11.462845 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:24:11.462834    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea219f5d-ce68-450a-8c70-e359e0207228-home\") pod \"ea219f5d-ce68-450a-8c70-e359e0207228\" (UID: \"ea219f5d-ce68-450a-8c70-e359e0207228\") "
Sep 09 21:24:11.462995 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:24:11.462878    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea219f5d-ce68-450a-8c70-e359e0207228-tmp-dir\") pod \"ea219f5d-ce68-450a-8c70-e359e0207228\" (UID: \"ea219f5d-ce68-450a-8c70-e359e0207228\") "
Sep 09 21:24:11.462995 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:24:11.462893    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea219f5d-ce68-450a-8c70-e359e0207228-model-cache\") pod \"ea219f5d-ce68-450a-8c70-e359e0207228\" (UID: \"ea219f5d-ce68-450a-8c70-e359e0207228\") "
Sep 09 21:24:11.462995 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:24:11.462920    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-29d4v\" (UniqueName: \"kubernetes.io/projected/ea219f5d-ce68-450a-8c70-e359e0207228-kube-api-access-29d4v\") pod \"ea219f5d-ce68-450a-8c70-e359e0207228\" (UID: \"ea219f5d-ce68-450a-8c70-e359e0207228\") "
Sep 09 21:24:11.462995 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:24:11.462948    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea219f5d-ce68-450a-8c70-e359e0207228-dshm\") pod \"ea219f5d-ce68-450a-8c70-e359e0207228\" (UID: \"ea219f5d-ce68-450a-8c70-e359e0207228\") "
Sep 09 21:24:11.463208 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:24:11.463081    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea219f5d-ce68-450a-8c70-e359e0207228-home" (OuterVolumeSpecName: "home") pod "ea219f5d-ce68-450a-8c70-e359e0207228" (UID: "ea219f5d-ce68-450a-8c70-e359e0207228"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:24:11.463208 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:24:11.463160    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea219f5d-ce68-450a-8c70-e359e0207228-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ea219f5d-ce68-450a-8c70-e359e0207228" (UID: "ea219f5d-ce68-450a-8c70-e359e0207228"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:24:11.463208 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:24:11.463175    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea219f5d-ce68-450a-8c70-e359e0207228-model-cache" (OuterVolumeSpecName: "model-cache") pod "ea219f5d-ce68-450a-8c70-e359e0207228" (UID: "ea219f5d-ce68-450a-8c70-e359e0207228"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:24:11.464535 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:24:11.463733    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea219f5d-ce68-450a-8c70-e359e0207228-home\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:24:11.465929 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:24:11.465878    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea219f5d-ce68-450a-8c70-e359e0207228-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ea219f5d-ce68-450a-8c70-e359e0207228" (UID: "ea219f5d-ce68-450a-8c70-e359e0207228"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:24:11.466093 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:24:11.466071    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea219f5d-ce68-450a-8c70-e359e0207228-kube-api-access-29d4v" (OuterVolumeSpecName: "kube-api-access-29d4v") pod "ea219f5d-ce68-450a-8c70-e359e0207228" (UID: "ea219f5d-ce68-450a-8c70-e359e0207228"). InnerVolumeSpecName "kube-api-access-29d4v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:24:11.475062 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:24:11.475036    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea219f5d-ce68-450a-8c70-e359e0207228-dshm" (OuterVolumeSpecName: "dshm") pod "ea219f5d-ce68-450a-8c70-e359e0207228" (UID: "ea219f5d-ce68-450a-8c70-e359e0207228"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:24:11.564238 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:24:11.564212    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea219f5d-ce68-450a-8c70-e359e0207228-tls-certs\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:24:11.564336 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:24:11.564253    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea219f5d-ce68-450a-8c70-e359e0207228-tmp-dir\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:24:11.564336 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:24:11.564262    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea219f5d-ce68-450a-8c70-e359e0207228-model-cache\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:24:11.564336 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:24:11.564270    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29d4v\" (UniqueName: \"kubernetes.io/projected/ea219f5d-ce68-450a-8c70-e359e0207228-kube-api-access-29d4v\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:24:11.564336 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:24:11.564279    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea219f5d-ce68-450a-8c70-e359e0207228-dshm\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:24:11.882728 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:24:11.882692    2576 generic.go:356] "Generic (PLEG): container finished" podID="ea219f5d-ce68-450a-8c70-e359e0207228" containerID="513a5597778907c0e58b7efbfffeba4f892d61b66eab96d0b4358a3a890ade61" exitCode=0
Sep 09 21:24:11.882728 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:24:11.882718    2576 generic.go:356] "Generic (PLEG): container finished" podID="ea219f5d-ce68-450a-8c70-e359e0207228" containerID="2488b50490c49c80d17221af3d3ec466f2e0032d21e45e8d82266f88a78bdb7e" exitCode=0
Sep 09 21:24:11.882966 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:24:11.882760    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-dsl6w"
Sep 09 21:24:11.882966 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:24:11.882771    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-dsl6w" event={"ID":"ea219f5d-ce68-450a-8c70-e359e0207228","Type":"ContainerDied","Data":"513a5597778907c0e58b7efbfffeba4f892d61b66eab96d0b4358a3a890ade61"}
Sep 09 21:24:11.882966 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:24:11.882849    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-dsl6w" event={"ID":"ea219f5d-ce68-450a-8c70-e359e0207228","Type":"ContainerDied","Data":"2488b50490c49c80d17221af3d3ec466f2e0032d21e45e8d82266f88a78bdb7e"}
Sep 09 21:24:11.882966 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:24:11.882867    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-dsl6w" event={"ID":"ea219f5d-ce68-450a-8c70-e359e0207228","Type":"ContainerDied","Data":"f9c0bb93db16d6ca173fcb245778b381933c14e32f35a7e07b8e7f210ab33e68"}
Sep 09 21:24:11.882966 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:24:11.882887    2576 scope.go:117] "RemoveContainer" containerID="513a5597778907c0e58b7efbfffeba4f892d61b66eab96d0b4358a3a890ade61"
Sep 09 21:24:11.891934 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:24:11.891915    2576 scope.go:117] "RemoveContainer" containerID="2488b50490c49c80d17221af3d3ec466f2e0032d21e45e8d82266f88a78bdb7e"
Sep 09 21:24:11.899774 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:24:11.899755    2576 scope.go:117] "RemoveContainer" containerID="513a5597778907c0e58b7efbfffeba4f892d61b66eab96d0b4358a3a890ade61"
Sep 09 21:24:11.900019 ip-10-0-134-72 kubenswrapper[2576]: E0909 21:24:11.900000    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"513a5597778907c0e58b7efbfffeba4f892d61b66eab96d0b4358a3a890ade61\": container with ID starting with 513a5597778907c0e58b7efbfffeba4f892d61b66eab96d0b4358a3a890ade61 not found: ID does not exist" containerID="513a5597778907c0e58b7efbfffeba4f892d61b66eab96d0b4358a3a890ade61"
Sep 09 21:24:11.900092 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:24:11.900031    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"513a5597778907c0e58b7efbfffeba4f892d61b66eab96d0b4358a3a890ade61"} err="failed to get container status \"513a5597778907c0e58b7efbfffeba4f892d61b66eab96d0b4358a3a890ade61\": rpc error: code = NotFound desc = could not find container \"513a5597778907c0e58b7efbfffeba4f892d61b66eab96d0b4358a3a890ade61\": container with ID starting with 513a5597778907c0e58b7efbfffeba4f892d61b66eab96d0b4358a3a890ade61 not found: ID does not exist"
Sep 09 21:24:11.900092 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:24:11.900053    2576 scope.go:117] "RemoveContainer" containerID="2488b50490c49c80d17221af3d3ec466f2e0032d21e45e8d82266f88a78bdb7e"
Sep 09 21:24:11.900325 ip-10-0-134-72 kubenswrapper[2576]: E0909 21:24:11.900308    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2488b50490c49c80d17221af3d3ec466f2e0032d21e45e8d82266f88a78bdb7e\": container with ID starting with 2488b50490c49c80d17221af3d3ec466f2e0032d21e45e8d82266f88a78bdb7e not found: ID does not exist" containerID="2488b50490c49c80d17221af3d3ec466f2e0032d21e45e8d82266f88a78bdb7e"
Sep 09 21:24:11.900383 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:24:11.900331    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2488b50490c49c80d17221af3d3ec466f2e0032d21e45e8d82266f88a78bdb7e"} err="failed to get container status \"2488b50490c49c80d17221af3d3ec466f2e0032d21e45e8d82266f88a78bdb7e\": rpc error: code = NotFound desc = could not find container \"2488b50490c49c80d17221af3d3ec466f2e0032d21e45e8d82266f88a78bdb7e\": container with ID starting with 2488b50490c49c80d17221af3d3ec466f2e0032d21e45e8d82266f88a78bdb7e not found: ID does not exist"
Sep 09 21:24:11.900383 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:24:11.900346    2576 scope.go:117] "RemoveContainer" containerID="513a5597778907c0e58b7efbfffeba4f892d61b66eab96d0b4358a3a890ade61"
Sep 09 21:24:11.900590 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:24:11.900570    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"513a5597778907c0e58b7efbfffeba4f892d61b66eab96d0b4358a3a890ade61"} err="failed to get container status \"513a5597778907c0e58b7efbfffeba4f892d61b66eab96d0b4358a3a890ade61\": rpc error: code = NotFound desc = could not find container \"513a5597778907c0e58b7efbfffeba4f892d61b66eab96d0b4358a3a890ade61\": container with ID starting with 513a5597778907c0e58b7efbfffeba4f892d61b66eab96d0b4358a3a890ade61 not found: ID does not exist"
Sep 09 21:24:11.900704 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:24:11.900591    2576 scope.go:117] "RemoveContainer" containerID="2488b50490c49c80d17221af3d3ec466f2e0032d21e45e8d82266f88a78bdb7e"
Sep 09 21:24:11.900826 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:24:11.900806    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2488b50490c49c80d17221af3d3ec466f2e0032d21e45e8d82266f88a78bdb7e"} err="failed to get container status \"2488b50490c49c80d17221af3d3ec466f2e0032d21e45e8d82266f88a78bdb7e\": rpc error: code = NotFound desc = could not find container \"2488b50490c49c80d17221af3d3ec466f2e0032d21e45e8d82266f88a78bdb7e\": container with ID starting with 2488b50490c49c80d17221af3d3ec466f2e0032d21e45e8d82266f88a78bdb7e not found: ID does not exist"
Sep 09 21:24:11.909914 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:24:11.909892    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-dsl6w"]
Sep 09 21:24:11.914739 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:24:11.914715    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-dsl6w"]
Sep 09 21:24:13.409025 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:24:13.408993    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ea219f5d-ce68-450a-8c70-e359e0207228" path="/var/lib/kubelet/pods/ea219f5d-ce68-450a-8c70-e359e0207228/volumes"
Sep 09 21:24:19.311635 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:24:19.311589    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:24:19.421265 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:24:19.421230    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:24:19.507382 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:24:19.507342    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:24:19.623352 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:24:19.623321    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:24:19.726997 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:24:19.726963    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:24:19.839237 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:24:19.839205    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:24:20.150339 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:24:20.150301    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:24:20.783594 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:24:20.783560    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:24:24.330937 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:24:24.330902    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:24:25.855744 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:24:25.855710    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:24:32.479100 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:24:32.479064    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:24:32.574585 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:24:32.574553    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:24:43.709989 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:24:43.709956    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-bv9hp"]
Sep 09 21:24:43.710427 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:24:43.710290    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15733211-6c01-4839-b5d2-5c8add4ea164" containerName="main"
Sep 09 21:24:43.710427 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:24:43.710300    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="15733211-6c01-4839-b5d2-5c8add4ea164" containerName="main"
Sep 09 21:24:43.710427 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:24:43.710313    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea219f5d-ce68-450a-8c70-e359e0207228" containerName="main"
Sep 09 21:24:43.710427 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:24:43.710318    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea219f5d-ce68-450a-8c70-e359e0207228" containerName="main"
Sep 09 21:24:43.710427 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:24:43.710330    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea219f5d-ce68-450a-8c70-e359e0207228" containerName="llm-d-routing-sidecar"
Sep 09 21:24:43.710427 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:24:43.710338    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea219f5d-ce68-450a-8c70-e359e0207228" containerName="llm-d-routing-sidecar"
Sep 09 21:24:43.710427 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:24:43.710349    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c557d9d-e865-408e-b7a7-42281ff091a4" containerName="main"
Sep 09 21:24:43.710427 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:24:43.710357    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c557d9d-e865-408e-b7a7-42281ff091a4" containerName="main"
Sep 09 21:24:43.710427 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:24:43.710428    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea219f5d-ce68-450a-8c70-e359e0207228" containerName="main"
Sep 09 21:24:43.710747 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:24:43.710436    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea219f5d-ce68-450a-8c70-e359e0207228" containerName="llm-d-routing-sidecar"
Sep 09 21:24:43.710747 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:24:43.710444    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="15733211-6c01-4839-b5d2-5c8add4ea164" containerName="main"
Sep 09 21:24:43.710747 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:24:43.710450    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c557d9d-e865-408e-b7a7-42281ff091a4" containerName="main"
Sep 09 21:24:43.714675 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:24:43.714658    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-bv9hp"
Sep 09 21:24:43.718840 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:24:43.718818    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-force-stop-3e394e04\"/\"stop-v1-kserve-self-signed-certs\""
Sep 09 21:24:43.719705 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:24:43.719682    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-force-stop-3e394e04\"/\"default-dockercfg-h5q9x\""
Sep 09 21:24:43.719793 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:24:43.719682    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-force-stop-3e394e04\"/\"openshift-service-ca.crt\""
Sep 09 21:24:43.721093 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:24:43.721072    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-force-stop-3e394e04\"/\"kube-root-ca.crt\""
Sep 09 21:24:43.733849 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:24:43.733821    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-bv9hp"]
Sep 09 21:24:43.740505 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:24:43.740478    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:24:43.835020 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:24:43.834983    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a1a6b9c6-46bb-4e2a-a432-4711fed0c6b1-model-cache\") pod \"stop-v1-kserve-5657b88c95-bv9hp\" (UID: \"a1a6b9c6-46bb-4e2a-a432-4711fed0c6b1\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-bv9hp"
Sep 09 21:24:43.835169 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:24:43.835039    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a1a6b9c6-46bb-4e2a-a432-4711fed0c6b1-home\") pod \"stop-v1-kserve-5657b88c95-bv9hp\" (UID: \"a1a6b9c6-46bb-4e2a-a432-4711fed0c6b1\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-bv9hp"
Sep 09 21:24:43.835169 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:24:43.835087    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a1a6b9c6-46bb-4e2a-a432-4711fed0c6b1-dshm\") pod \"stop-v1-kserve-5657b88c95-bv9hp\" (UID: \"a1a6b9c6-46bb-4e2a-a432-4711fed0c6b1\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-bv9hp"
Sep 09 21:24:43.835169 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:24:43.835122    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1a6b9c6-46bb-4e2a-a432-4711fed0c6b1-tmp-dir\") pod \"stop-v1-kserve-5657b88c95-bv9hp\" (UID: \"a1a6b9c6-46bb-4e2a-a432-4711fed0c6b1\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-bv9hp"
Sep 09 21:24:43.835328 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:24:43.835170    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1a6b9c6-46bb-4e2a-a432-4711fed0c6b1-tls-certs\") pod \"stop-v1-kserve-5657b88c95-bv9hp\" (UID: \"a1a6b9c6-46bb-4e2a-a432-4711fed0c6b1\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-bv9hp"
Sep 09 21:24:43.835328 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:24:43.835202    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55g4b\" (UniqueName: \"kubernetes.io/projected/a1a6b9c6-46bb-4e2a-a432-4711fed0c6b1-kube-api-access-55g4b\") pod \"stop-v1-kserve-5657b88c95-bv9hp\" (UID: \"a1a6b9c6-46bb-4e2a-a432-4711fed0c6b1\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-bv9hp"
Sep 09 21:24:43.936233 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:24:43.936194    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1a6b9c6-46bb-4e2a-a432-4711fed0c6b1-tls-certs\") pod \"stop-v1-kserve-5657b88c95-bv9hp\" (UID: \"a1a6b9c6-46bb-4e2a-a432-4711fed0c6b1\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-bv9hp"
Sep 09 21:24:43.936233 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:24:43.936236    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55g4b\" (UniqueName: \"kubernetes.io/projected/a1a6b9c6-46bb-4e2a-a432-4711fed0c6b1-kube-api-access-55g4b\") pod \"stop-v1-kserve-5657b88c95-bv9hp\" (UID: \"a1a6b9c6-46bb-4e2a-a432-4711fed0c6b1\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-bv9hp"
Sep 09 21:24:43.936486 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:24:43.936282    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a1a6b9c6-46bb-4e2a-a432-4711fed0c6b1-model-cache\") pod \"stop-v1-kserve-5657b88c95-bv9hp\" (UID: \"a1a6b9c6-46bb-4e2a-a432-4711fed0c6b1\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-bv9hp"
Sep 09 21:24:43.936486 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:24:43.936319    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a1a6b9c6-46bb-4e2a-a432-4711fed0c6b1-home\") pod \"stop-v1-kserve-5657b88c95-bv9hp\" (UID: \"a1a6b9c6-46bb-4e2a-a432-4711fed0c6b1\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-bv9hp"
Sep 09 21:24:43.936486 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:24:43.936348    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a1a6b9c6-46bb-4e2a-a432-4711fed0c6b1-dshm\") pod \"stop-v1-kserve-5657b88c95-bv9hp\" (UID: \"a1a6b9c6-46bb-4e2a-a432-4711fed0c6b1\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-bv9hp"
Sep 09 21:24:43.936486 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:24:43.936408    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1a6b9c6-46bb-4e2a-a432-4711fed0c6b1-tmp-dir\") pod \"stop-v1-kserve-5657b88c95-bv9hp\" (UID: \"a1a6b9c6-46bb-4e2a-a432-4711fed0c6b1\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-bv9hp"
Sep 09 21:24:43.936795 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:24:43.936775    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1a6b9c6-46bb-4e2a-a432-4711fed0c6b1-tmp-dir\") pod \"stop-v1-kserve-5657b88c95-bv9hp\" (UID: \"a1a6b9c6-46bb-4e2a-a432-4711fed0c6b1\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-bv9hp"
Sep 09 21:24:43.936795 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:24:43.936787    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a1a6b9c6-46bb-4e2a-a432-4711fed0c6b1-model-cache\") pod \"stop-v1-kserve-5657b88c95-bv9hp\" (UID: \"a1a6b9c6-46bb-4e2a-a432-4711fed0c6b1\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-bv9hp"
Sep 09 21:24:43.936887 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:24:43.936850    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a1a6b9c6-46bb-4e2a-a432-4711fed0c6b1-home\") pod \"stop-v1-kserve-5657b88c95-bv9hp\" (UID: \"a1a6b9c6-46bb-4e2a-a432-4711fed0c6b1\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-bv9hp"
Sep 09 21:24:43.938761 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:24:43.938744    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a1a6b9c6-46bb-4e2a-a432-4711fed0c6b1-dshm\") pod \"stop-v1-kserve-5657b88c95-bv9hp\" (UID: \"a1a6b9c6-46bb-4e2a-a432-4711fed0c6b1\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-bv9hp"
Sep 09 21:24:43.939140 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:24:43.939120    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1a6b9c6-46bb-4e2a-a432-4711fed0c6b1-tls-certs\") pod \"stop-v1-kserve-5657b88c95-bv9hp\" (UID: \"a1a6b9c6-46bb-4e2a-a432-4711fed0c6b1\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-bv9hp"
Sep 09 21:24:43.945941 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:24:43.945913    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-55g4b\" (UniqueName: \"kubernetes.io/projected/a1a6b9c6-46bb-4e2a-a432-4711fed0c6b1-kube-api-access-55g4b\") pod \"stop-v1-kserve-5657b88c95-bv9hp\" (UID: \"a1a6b9c6-46bb-4e2a-a432-4711fed0c6b1\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-bv9hp"
Sep 09 21:24:44.024743 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:24:44.024664    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-bv9hp"
Sep 09 21:24:44.325890 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:24:44.325861    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-bv9hp"]
Sep 09 21:24:44.326846 ip-10-0-134-72 kubenswrapper[2576]: W0909 21:24:44.326823    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1a6b9c6_46bb_4e2a_a432_4711fed0c6b1.slice/crio-4544f81c55dd69b464b68c27e93618f0429c574e8c2a3f815ba7338589be6a23 WatchSource:0}: Error finding container 4544f81c55dd69b464b68c27e93618f0429c574e8c2a3f815ba7338589be6a23: Status 404 returned error can't find the container with id 4544f81c55dd69b464b68c27e93618f0429c574e8c2a3f815ba7338589be6a23
Sep 09 21:24:45.051829 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:24:45.051788    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-bv9hp" event={"ID":"a1a6b9c6-46bb-4e2a-a432-4711fed0c6b1","Type":"ContainerStarted","Data":"905f4b1e3898ffd1b6197ce5ebd605c1d4929d8eb712aef39eeb8e98797dbaf3"}
Sep 09 21:24:45.052429 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:24:45.051847    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-bv9hp" event={"ID":"a1a6b9c6-46bb-4e2a-a432-4711fed0c6b1","Type":"ContainerStarted","Data":"4544f81c55dd69b464b68c27e93618f0429c574e8c2a3f815ba7338589be6a23"}
Sep 09 21:24:45.081924 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:24:45.081836    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-bv9hp" podStartSLOduration=2.081822136 podStartE2EDuration="2.081822136s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:24:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:24:45.081508946 +0000 UTC m=+1928.195194807" watchObservedRunningTime="2026-09-09 21:24:45.081822136 +0000 UTC m=+1928.195507995"
Sep 09 21:24:46.025101 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:24:46.025069    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-bv9hp"
Sep 09 21:24:54.025606 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:24:54.025562    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-bv9hp"
Sep 09 21:24:54.038386 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:24:54.038346    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-bv9hp"
Sep 09 21:24:54.112051 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:24:54.112020    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-bv9hp"
Sep 09 21:24:55.724950 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:24:55.724891    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:24:59.345763 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:24:59.345729    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:24:59.444007 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:24:59.443951    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:25:02.576639 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:25:02.576602    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:25:02.668926 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:25:02.668890    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:25:04.796137 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:25:04.796097    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:25:07.410430 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:25:07.410394    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:25:13.719107 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:25:13.719073    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:25:13.791750 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:25:13.791719    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:25:26.030805 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:25:26.030772    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7585bbd4df-6szhg"]
Sep 09 21:25:26.033899 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:25:26.033882    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7585bbd4df-6szhg"
Sep 09 21:25:26.036397 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:25:26.036357    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"seaweedfs-s3-creds\""
Sep 09 21:25:26.036534 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:25:26.036467    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"default-dockercfg-dndtl\""
Sep 09 21:25:26.037779 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:25:26.037761    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"odh-kserve-custom-ca-bundle\""
Sep 09 21:25:26.037986 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:25:26.037973    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"kube-root-ca.crt\""
Sep 09 21:25:26.038675 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:25:26.038655    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"openshift-service-ca.crt\""
Sep 09 21:25:26.038769 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:25:26.038713    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"tokenizer-clean-path-test-kserve-self-signed-certs\""
Sep 09 21:25:26.049095 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:25:26.049072    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7585bbd4df-6szhg"]
Sep 09 21:25:26.060081 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:25:26.060055    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:25:26.088613 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:25:26.088580    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9b9638b-597f-42b1-9f5c-b329f244960d-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-7585bbd4df-6szhg\" (UID: \"e9b9638b-597f-42b1-9f5c-b329f244960d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7585bbd4df-6szhg"
Sep 09 21:25:26.088613 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:25:26.088614    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e9b9638b-597f-42b1-9f5c-b329f244960d-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-7585bbd4df-6szhg\" (UID: \"e9b9638b-597f-42b1-9f5c-b329f244960d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7585bbd4df-6szhg"
Sep 09 21:25:26.088851 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:25:26.088632    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9b9638b-597f-42b1-9f5c-b329f244960d-model-cache\") pod \"tokenizer-clean-path-test-kserve-7585bbd4df-6szhg\" (UID: \"e9b9638b-597f-42b1-9f5c-b329f244960d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7585bbd4df-6szhg"
Sep 09 21:25:26.088851 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:25:26.088785    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9b9638b-597f-42b1-9f5c-b329f244960d-home\") pod \"tokenizer-clean-path-test-kserve-7585bbd4df-6szhg\" (UID: \"e9b9638b-597f-42b1-9f5c-b329f244960d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7585bbd4df-6szhg"
Sep 09 21:25:26.088851 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:25:26.088822    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9b9638b-597f-42b1-9f5c-b329f244960d-dshm\") pod \"tokenizer-clean-path-test-kserve-7585bbd4df-6szhg\" (UID: \"e9b9638b-597f-42b1-9f5c-b329f244960d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7585bbd4df-6szhg"
Sep 09 21:25:26.089012 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:25:26.088900    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n57z2\" (UniqueName: \"kubernetes.io/projected/e9b9638b-597f-42b1-9f5c-b329f244960d-kube-api-access-n57z2\") pod \"tokenizer-clean-path-test-kserve-7585bbd4df-6szhg\" (UID: \"e9b9638b-597f-42b1-9f5c-b329f244960d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7585bbd4df-6szhg"
Sep 09 21:25:26.089012 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:25:26.088937    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e9b9638b-597f-42b1-9f5c-b329f244960d-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-7585bbd4df-6szhg\" (UID: \"e9b9638b-597f-42b1-9f5c-b329f244960d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7585bbd4df-6szhg"
Sep 09 21:25:26.089012 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:25:26.088969    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9b9638b-597f-42b1-9f5c-b329f244960d-tls-certs\") pod \"tokenizer-clean-path-test-kserve-7585bbd4df-6szhg\" (UID: \"e9b9638b-597f-42b1-9f5c-b329f244960d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7585bbd4df-6szhg"
Sep 09 21:25:26.189832 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:25:26.189794    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9b9638b-597f-42b1-9f5c-b329f244960d-home\") pod \"tokenizer-clean-path-test-kserve-7585bbd4df-6szhg\" (UID: \"e9b9638b-597f-42b1-9f5c-b329f244960d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7585bbd4df-6szhg"
Sep 09 21:25:26.189832 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:25:26.189827    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9b9638b-597f-42b1-9f5c-b329f244960d-dshm\") pod \"tokenizer-clean-path-test-kserve-7585bbd4df-6szhg\" (UID: \"e9b9638b-597f-42b1-9f5c-b329f244960d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7585bbd4df-6szhg"
Sep 09 21:25:26.190040 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:25:26.189966    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n57z2\" (UniqueName: \"kubernetes.io/projected/e9b9638b-597f-42b1-9f5c-b329f244960d-kube-api-access-n57z2\") pod \"tokenizer-clean-path-test-kserve-7585bbd4df-6szhg\" (UID: \"e9b9638b-597f-42b1-9f5c-b329f244960d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7585bbd4df-6szhg"
Sep 09 21:25:26.190040 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:25:26.190001    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e9b9638b-597f-42b1-9f5c-b329f244960d-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-7585bbd4df-6szhg\" (UID: \"e9b9638b-597f-42b1-9f5c-b329f244960d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7585bbd4df-6szhg"
Sep 09 21:25:26.190040 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:25:26.190032    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9b9638b-597f-42b1-9f5c-b329f244960d-tls-certs\") pod \"tokenizer-clean-path-test-kserve-7585bbd4df-6szhg\" (UID: \"e9b9638b-597f-42b1-9f5c-b329f244960d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7585bbd4df-6szhg"
Sep 09 21:25:26.190177 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:25:26.190074    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9b9638b-597f-42b1-9f5c-b329f244960d-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-7585bbd4df-6szhg\" (UID: \"e9b9638b-597f-42b1-9f5c-b329f244960d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7585bbd4df-6szhg"
Sep 09 21:25:26.190177 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:25:26.190097    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e9b9638b-597f-42b1-9f5c-b329f244960d-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-7585bbd4df-6szhg\" (UID: \"e9b9638b-597f-42b1-9f5c-b329f244960d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7585bbd4df-6szhg"
Sep 09 21:25:26.190177 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:25:26.190123    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9b9638b-597f-42b1-9f5c-b329f244960d-model-cache\") pod \"tokenizer-clean-path-test-kserve-7585bbd4df-6szhg\" (UID: \"e9b9638b-597f-42b1-9f5c-b329f244960d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7585bbd4df-6szhg"
Sep 09 21:25:26.190340 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:25:26.190206    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9b9638b-597f-42b1-9f5c-b329f244960d-home\") pod \"tokenizer-clean-path-test-kserve-7585bbd4df-6szhg\" (UID: \"e9b9638b-597f-42b1-9f5c-b329f244960d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7585bbd4df-6szhg"
Sep 09 21:25:26.190340 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:25:26.190295    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e9b9638b-597f-42b1-9f5c-b329f244960d-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-7585bbd4df-6szhg\" (UID: \"e9b9638b-597f-42b1-9f5c-b329f244960d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7585bbd4df-6szhg"
Sep 09 21:25:26.190495 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:25:26.190472    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9b9638b-597f-42b1-9f5c-b329f244960d-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-7585bbd4df-6szhg\" (UID: \"e9b9638b-597f-42b1-9f5c-b329f244960d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7585bbd4df-6szhg"
Sep 09 21:25:26.190543 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:25:26.190513    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9b9638b-597f-42b1-9f5c-b329f244960d-model-cache\") pod \"tokenizer-clean-path-test-kserve-7585bbd4df-6szhg\" (UID: \"e9b9638b-597f-42b1-9f5c-b329f244960d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7585bbd4df-6szhg"
Sep 09 21:25:26.190794 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:25:26.190771    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e9b9638b-597f-42b1-9f5c-b329f244960d-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-7585bbd4df-6szhg\" (UID: \"e9b9638b-597f-42b1-9f5c-b329f244960d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7585bbd4df-6szhg"
Sep 09 21:25:26.192303 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:25:26.192274    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9b9638b-597f-42b1-9f5c-b329f244960d-dshm\") pod \"tokenizer-clean-path-test-kserve-7585bbd4df-6szhg\" (UID: \"e9b9638b-597f-42b1-9f5c-b329f244960d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7585bbd4df-6szhg"
Sep 09 21:25:26.192666 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:25:26.192647    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9b9638b-597f-42b1-9f5c-b329f244960d-tls-certs\") pod \"tokenizer-clean-path-test-kserve-7585bbd4df-6szhg\" (UID: \"e9b9638b-597f-42b1-9f5c-b329f244960d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7585bbd4df-6szhg"
Sep 09 21:25:26.198904 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:25:26.198882    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n57z2\" (UniqueName: \"kubernetes.io/projected/e9b9638b-597f-42b1-9f5c-b329f244960d-kube-api-access-n57z2\") pod \"tokenizer-clean-path-test-kserve-7585bbd4df-6szhg\" (UID: \"e9b9638b-597f-42b1-9f5c-b329f244960d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7585bbd4df-6szhg"
Sep 09 21:25:26.314665 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:25:26.314584    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-67fb486b4d-c2vj2"]
Sep 09 21:25:26.318142 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:25:26.318123    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-67fb486b4d-c2vj2"
Sep 09 21:25:26.330356 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:25:26.330334    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-67fb486b4d-c2vj2"]
Sep 09 21:25:26.344391 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:25:26.344341    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7585bbd4df-6szhg"
Sep 09 21:25:26.392007 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:25:26.391970    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/92bf4c52-eeb9-4bc3-a6ad-0410d0654f68-cabundle-cert\") pod \"tokenizer-clean-path-test-tokenizer-67fb486b4d-c2vj2\" (UID: \"92bf4c52-eeb9-4bc3-a6ad-0410d0654f68\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-67fb486b4d-c2vj2"
Sep 09 21:25:26.392186 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:25:26.392036    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92bf4c52-eeb9-4bc3-a6ad-0410d0654f68-tls-certs\") pod \"tokenizer-clean-path-test-tokenizer-67fb486b4d-c2vj2\" (UID: \"92bf4c52-eeb9-4bc3-a6ad-0410d0654f68\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-67fb486b4d-c2vj2"
Sep 09 21:25:26.392186 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:25:26.392155    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92bf4c52-eeb9-4bc3-a6ad-0410d0654f68-tmp-dir\") pod \"tokenizer-clean-path-test-tokenizer-67fb486b4d-c2vj2\" (UID: \"92bf4c52-eeb9-4bc3-a6ad-0410d0654f68\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-67fb486b4d-c2vj2"
Sep 09 21:25:26.392305 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:25:26.392204    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/92bf4c52-eeb9-4bc3-a6ad-0410d0654f68-kserve-provision-location\") pod \"tokenizer-clean-path-test-tokenizer-67fb486b4d-c2vj2\" (UID: \"92bf4c52-eeb9-4bc3-a6ad-0410d0654f68\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-67fb486b4d-c2vj2"
Sep 09 21:25:26.493475 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:25:26.493440    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92bf4c52-eeb9-4bc3-a6ad-0410d0654f68-tmp-dir\") pod \"tokenizer-clean-path-test-tokenizer-67fb486b4d-c2vj2\" (UID: \"92bf4c52-eeb9-4bc3-a6ad-0410d0654f68\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-67fb486b4d-c2vj2"
Sep 09 21:25:26.493645 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:25:26.493498    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/92bf4c52-eeb9-4bc3-a6ad-0410d0654f68-kserve-provision-location\") pod \"tokenizer-clean-path-test-tokenizer-67fb486b4d-c2vj2\" (UID: \"92bf4c52-eeb9-4bc3-a6ad-0410d0654f68\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-67fb486b4d-c2vj2"
Sep 09 21:25:26.493645 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:25:26.493522    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/92bf4c52-eeb9-4bc3-a6ad-0410d0654f68-cabundle-cert\") pod \"tokenizer-clean-path-test-tokenizer-67fb486b4d-c2vj2\" (UID: \"92bf4c52-eeb9-4bc3-a6ad-0410d0654f68\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-67fb486b4d-c2vj2"
Sep 09 21:25:26.493645 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:25:26.493554    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92bf4c52-eeb9-4bc3-a6ad-0410d0654f68-tls-certs\") pod \"tokenizer-clean-path-test-tokenizer-67fb486b4d-c2vj2\" (UID: \"92bf4c52-eeb9-4bc3-a6ad-0410d0654f68\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-67fb486b4d-c2vj2"
Sep 09 21:25:26.493842 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:25:26.493823    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92bf4c52-eeb9-4bc3-a6ad-0410d0654f68-tmp-dir\") pod \"tokenizer-clean-path-test-tokenizer-67fb486b4d-c2vj2\" (UID: \"92bf4c52-eeb9-4bc3-a6ad-0410d0654f68\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-67fb486b4d-c2vj2"
Sep 09 21:25:26.493895 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:25:26.493872    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/92bf4c52-eeb9-4bc3-a6ad-0410d0654f68-kserve-provision-location\") pod \"tokenizer-clean-path-test-tokenizer-67fb486b4d-c2vj2\" (UID: \"92bf4c52-eeb9-4bc3-a6ad-0410d0654f68\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-67fb486b4d-c2vj2"
Sep 09 21:25:26.494158 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:25:26.494142    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/92bf4c52-eeb9-4bc3-a6ad-0410d0654f68-cabundle-cert\") pod \"tokenizer-clean-path-test-tokenizer-67fb486b4d-c2vj2\" (UID: \"92bf4c52-eeb9-4bc3-a6ad-0410d0654f68\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-67fb486b4d-c2vj2"
Sep 09 21:25:26.496146 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:25:26.496129    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92bf4c52-eeb9-4bc3-a6ad-0410d0654f68-tls-certs\") pod \"tokenizer-clean-path-test-tokenizer-67fb486b4d-c2vj2\" (UID: \"92bf4c52-eeb9-4bc3-a6ad-0410d0654f68\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-67fb486b4d-c2vj2"
Sep 09 21:25:26.626259 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:25:26.626236    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7585bbd4df-6szhg"]
Sep 09 21:25:26.627499 ip-10-0-134-72 kubenswrapper[2576]: W0909 21:25:26.627472    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9b9638b_597f_42b1_9f5c_b329f244960d.slice/crio-b9b35363b6ffd773eafab375c18e09b3db1af3b1b2f0936f0a607f44b8c19f9c WatchSource:0}: Error finding container b9b35363b6ffd773eafab375c18e09b3db1af3b1b2f0936f0a607f44b8c19f9c: Status 404 returned error can't find the container with id b9b35363b6ffd773eafab375c18e09b3db1af3b1b2f0936f0a607f44b8c19f9c
Sep 09 21:25:26.628143 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:25:26.628121    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-67fb486b4d-c2vj2"
Sep 09 21:25:26.912067 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:25:26.912037    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-67fb486b4d-c2vj2"]
Sep 09 21:25:26.914151 ip-10-0-134-72 kubenswrapper[2576]: W0909 21:25:26.914128    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92bf4c52_eeb9_4bc3_a6ad_0410d0654f68.slice/crio-c566364acc3ce334ecee5fdc089a45130ff96868c4994e443711f6936b5e4d0e WatchSource:0}: Error finding container c566364acc3ce334ecee5fdc089a45130ff96868c4994e443711f6936b5e4d0e: Status 404 returned error can't find the container with id c566364acc3ce334ecee5fdc089a45130ff96868c4994e443711f6936b5e4d0e
Sep 09 21:25:27.192393 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:25:27.192284    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:25:27.271749 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:25:27.271469    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7585bbd4df-6szhg" event={"ID":"e9b9638b-597f-42b1-9f5c-b329f244960d","Type":"ContainerStarted","Data":"c1da8b9f0875aa126e982f9ab1c948212cf92637d095ebdafe0a90e8aaa9281d"}
Sep 09 21:25:27.271749 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:25:27.271536    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7585bbd4df-6szhg" event={"ID":"e9b9638b-597f-42b1-9f5c-b329f244960d","Type":"ContainerStarted","Data":"b9b35363b6ffd773eafab375c18e09b3db1af3b1b2f0936f0a607f44b8c19f9c"}
Sep 09 21:25:27.273286 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:25:27.273256    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-67fb486b4d-c2vj2" event={"ID":"92bf4c52-eeb9-4bc3-a6ad-0410d0654f68","Type":"ContainerStarted","Data":"d9975dca5a9545b699f1cd6713b489a719da8e0db844f4d0afb32d03f93c4e8f"}
Sep 09 21:25:27.273464 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:25:27.273301    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-67fb486b4d-c2vj2" event={"ID":"92bf4c52-eeb9-4bc3-a6ad-0410d0654f68","Type":"ContainerStarted","Data":"c566364acc3ce334ecee5fdc089a45130ff96868c4994e443711f6936b5e4d0e"}
Sep 09 21:25:29.288842 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:25:29.288811    2576 generic.go:356] "Generic (PLEG): container finished" podID="e9b9638b-597f-42b1-9f5c-b329f244960d" containerID="c1da8b9f0875aa126e982f9ab1c948212cf92637d095ebdafe0a90e8aaa9281d" exitCode=0
Sep 09 21:25:29.289238 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:25:29.288864    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7585bbd4df-6szhg" event={"ID":"e9b9638b-597f-42b1-9f5c-b329f244960d","Type":"ContainerDied","Data":"c1da8b9f0875aa126e982f9ab1c948212cf92637d095ebdafe0a90e8aaa9281d"}
Sep 09 21:25:30.295668 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:25:30.295622    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7585bbd4df-6szhg" event={"ID":"e9b9638b-597f-42b1-9f5c-b329f244960d","Type":"ContainerStarted","Data":"2f46580eff450e1c095c2f656f9ef4b44e6562995b7245ac72ac531887174d68"}
Sep 09 21:25:30.297258 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:25:30.297231    2576 generic.go:356] "Generic (PLEG): container finished" podID="92bf4c52-eeb9-4bc3-a6ad-0410d0654f68" containerID="d9975dca5a9545b699f1cd6713b489a719da8e0db844f4d0afb32d03f93c4e8f" exitCode=0
Sep 09 21:25:30.297392 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:25:30.297284    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-67fb486b4d-c2vj2" event={"ID":"92bf4c52-eeb9-4bc3-a6ad-0410d0654f68","Type":"ContainerDied","Data":"d9975dca5a9545b699f1cd6713b489a719da8e0db844f4d0afb32d03f93c4e8f"}
Sep 09 21:25:30.321591 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:25:30.321544    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7585bbd4df-6szhg" podStartSLOduration=1.859718554 podStartE2EDuration="4.321530636s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.461812082s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:25:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:25:30.319608968 +0000 UTC m=+1973.433294826" watchObservedRunningTime="2026-09-09 21:25:30.321530636 +0000 UTC m=+1973.435216495"
Sep 09 21:25:30.344748 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:25:30.344721    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7585bbd4df-6szhg"
Sep 09 21:25:36.344772 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:25:36.344733    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7585bbd4df-6szhg"
Sep 09 21:25:36.364541 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:25:36.364501    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7585bbd4df-6szhg"
Sep 09 21:25:37.361772 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:25:37.361739    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7585bbd4df-6szhg"
Sep 09 21:25:38.733032 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:25:38.732984    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:25:50.780130 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:25:50.780084    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:25:55.941350 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:25:55.941303    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:26:17.609004 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:26:17.608967    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-67fb486b4d-c2vj2" event={"ID":"92bf4c52-eeb9-4bc3-a6ad-0410d0654f68","Type":"ContainerStarted","Data":"be35faa462d80716823c40a854519888cadd6397692d5d36652d87d3b74b6b24"}
Sep 09 21:26:17.609433 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:26:17.609178    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-67fb486b4d-c2vj2"
Sep 09 21:26:17.634213 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:26:17.634158    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-67fb486b4d-c2vj2" podStartSLOduration=2.175040068 podStartE2EDuration="51.634141622s" totalImagesPullingTime="47.00119691s" totalInitContainerRuntime="2.457904644s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:25:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:26:17.631682656 +0000 UTC m=+2020.745368515" watchObservedRunningTime="2026-09-09 21:26:17.634141622 +0000 UTC m=+2020.747827480"
Sep 09 21:26:17.689176 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:26:17.689137    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:26:17.801188 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:26:17.801135    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:26:17.895462 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:26:17.895341    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:26:22.674805 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:26:22.674767    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-bv9hp"]
Sep 09 21:26:22.767446 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:26:22.767398    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:26:37.694816 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:26:37.694775    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-bv9hp" podUID="a1a6b9c6-46bb-4e2a-a432-4711fed0c6b1" containerName="main" containerID="cri-o://905f4b1e3898ffd1b6197ce5ebd605c1d4929d8eb712aef39eeb8e98797dbaf3" gracePeriod=45
Sep 09 21:26:38.101506 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:26:38.101471    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-bv9hp" podUID="a1a6b9c6-46bb-4e2a-a432-4711fed0c6b1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": dial tcp 10.132.0.61:8000: connect: connection refused"
Sep 09 21:26:38.500777 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:26:38.500749    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-bv9hp"
Sep 09 21:26:38.570323 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:26:38.570295    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a1a6b9c6-46bb-4e2a-a432-4711fed0c6b1-home\") pod \"a1a6b9c6-46bb-4e2a-a432-4711fed0c6b1\" (UID: \"a1a6b9c6-46bb-4e2a-a432-4711fed0c6b1\") "
Sep 09 21:26:38.570514 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:26:38.570382    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1a6b9c6-46bb-4e2a-a432-4711fed0c6b1-tmp-dir\") pod \"a1a6b9c6-46bb-4e2a-a432-4711fed0c6b1\" (UID: \"a1a6b9c6-46bb-4e2a-a432-4711fed0c6b1\") "
Sep 09 21:26:38.570514 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:26:38.570410    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a1a6b9c6-46bb-4e2a-a432-4711fed0c6b1-model-cache\") pod \"a1a6b9c6-46bb-4e2a-a432-4711fed0c6b1\" (UID: \"a1a6b9c6-46bb-4e2a-a432-4711fed0c6b1\") "
Sep 09 21:26:38.570514 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:26:38.570436    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1a6b9c6-46bb-4e2a-a432-4711fed0c6b1-tls-certs\") pod \"a1a6b9c6-46bb-4e2a-a432-4711fed0c6b1\" (UID: \"a1a6b9c6-46bb-4e2a-a432-4711fed0c6b1\") "
Sep 09 21:26:38.570514 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:26:38.570464    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a1a6b9c6-46bb-4e2a-a432-4711fed0c6b1-dshm\") pod \"a1a6b9c6-46bb-4e2a-a432-4711fed0c6b1\" (UID: \"a1a6b9c6-46bb-4e2a-a432-4711fed0c6b1\") "
Sep 09 21:26:38.570832 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:26:38.570517    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-55g4b\" (UniqueName: \"kubernetes.io/projected/a1a6b9c6-46bb-4e2a-a432-4711fed0c6b1-kube-api-access-55g4b\") pod \"a1a6b9c6-46bb-4e2a-a432-4711fed0c6b1\" (UID: \"a1a6b9c6-46bb-4e2a-a432-4711fed0c6b1\") "
Sep 09 21:26:38.570832 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:26:38.570591    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1a6b9c6-46bb-4e2a-a432-4711fed0c6b1-home" (OuterVolumeSpecName: "home") pod "a1a6b9c6-46bb-4e2a-a432-4711fed0c6b1" (UID: "a1a6b9c6-46bb-4e2a-a432-4711fed0c6b1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:26:38.570832 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:26:38.570628    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1a6b9c6-46bb-4e2a-a432-4711fed0c6b1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a1a6b9c6-46bb-4e2a-a432-4711fed0c6b1" (UID: "a1a6b9c6-46bb-4e2a-a432-4711fed0c6b1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:26:38.570832 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:26:38.570689    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1a6b9c6-46bb-4e2a-a432-4711fed0c6b1-model-cache" (OuterVolumeSpecName: "model-cache") pod "a1a6b9c6-46bb-4e2a-a432-4711fed0c6b1" (UID: "a1a6b9c6-46bb-4e2a-a432-4711fed0c6b1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:26:38.570832 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:26:38.570825    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a1a6b9c6-46bb-4e2a-a432-4711fed0c6b1-home\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:26:38.571092 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:26:38.570841    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1a6b9c6-46bb-4e2a-a432-4711fed0c6b1-tmp-dir\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:26:38.571092 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:26:38.570854    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a1a6b9c6-46bb-4e2a-a432-4711fed0c6b1-model-cache\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:26:38.573067 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:26:38.573043    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1a6b9c6-46bb-4e2a-a432-4711fed0c6b1-kube-api-access-55g4b" (OuterVolumeSpecName: "kube-api-access-55g4b") pod "a1a6b9c6-46bb-4e2a-a432-4711fed0c6b1" (UID: "a1a6b9c6-46bb-4e2a-a432-4711fed0c6b1"). InnerVolumeSpecName "kube-api-access-55g4b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:26:38.573221 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:26:38.573199    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1a6b9c6-46bb-4e2a-a432-4711fed0c6b1-dshm" (OuterVolumeSpecName: "dshm") pod "a1a6b9c6-46bb-4e2a-a432-4711fed0c6b1" (UID: "a1a6b9c6-46bb-4e2a-a432-4711fed0c6b1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:26:38.573221 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:26:38.573196    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1a6b9c6-46bb-4e2a-a432-4711fed0c6b1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a1a6b9c6-46bb-4e2a-a432-4711fed0c6b1" (UID: "a1a6b9c6-46bb-4e2a-a432-4711fed0c6b1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:26:38.671645 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:26:38.671574    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1a6b9c6-46bb-4e2a-a432-4711fed0c6b1-tls-certs\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:26:38.671645 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:26:38.671600    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a1a6b9c6-46bb-4e2a-a432-4711fed0c6b1-dshm\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:26:38.671645 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:26:38.671610    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55g4b\" (UniqueName: \"kubernetes.io/projected/a1a6b9c6-46bb-4e2a-a432-4711fed0c6b1-kube-api-access-55g4b\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:26:38.729333 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:26:38.729296    2576 generic.go:356] "Generic (PLEG): container finished" podID="a1a6b9c6-46bb-4e2a-a432-4711fed0c6b1" containerID="905f4b1e3898ffd1b6197ce5ebd605c1d4929d8eb712aef39eeb8e98797dbaf3" exitCode=0
Sep 09 21:26:38.729809 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:26:38.729346    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-bv9hp"
Sep 09 21:26:38.729809 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:26:38.729391    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-bv9hp" event={"ID":"a1a6b9c6-46bb-4e2a-a432-4711fed0c6b1","Type":"ContainerDied","Data":"905f4b1e3898ffd1b6197ce5ebd605c1d4929d8eb712aef39eeb8e98797dbaf3"}
Sep 09 21:26:38.729809 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:26:38.729467    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-bv9hp" event={"ID":"a1a6b9c6-46bb-4e2a-a432-4711fed0c6b1","Type":"ContainerDied","Data":"4544f81c55dd69b464b68c27e93618f0429c574e8c2a3f815ba7338589be6a23"}
Sep 09 21:26:38.729809 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:26:38.729500    2576 scope.go:117] "RemoveContainer" containerID="905f4b1e3898ffd1b6197ce5ebd605c1d4929d8eb712aef39eeb8e98797dbaf3"
Sep 09 21:26:38.739020 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:26:38.738999    2576 scope.go:117] "RemoveContainer" containerID="905f4b1e3898ffd1b6197ce5ebd605c1d4929d8eb712aef39eeb8e98797dbaf3"
Sep 09 21:26:38.739299 ip-10-0-134-72 kubenswrapper[2576]: E0909 21:26:38.739279    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"905f4b1e3898ffd1b6197ce5ebd605c1d4929d8eb712aef39eeb8e98797dbaf3\": container with ID starting with 905f4b1e3898ffd1b6197ce5ebd605c1d4929d8eb712aef39eeb8e98797dbaf3 not found: ID does not exist" containerID="905f4b1e3898ffd1b6197ce5ebd605c1d4929d8eb712aef39eeb8e98797dbaf3"
Sep 09 21:26:38.739397 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:26:38.739306    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"905f4b1e3898ffd1b6197ce5ebd605c1d4929d8eb712aef39eeb8e98797dbaf3"} err="failed to get container status \"905f4b1e3898ffd1b6197ce5ebd605c1d4929d8eb712aef39eeb8e98797dbaf3\": rpc error: code = NotFound desc = could not find container \"905f4b1e3898ffd1b6197ce5ebd605c1d4929d8eb712aef39eeb8e98797dbaf3\": container with ID starting with 905f4b1e3898ffd1b6197ce5ebd605c1d4929d8eb712aef39eeb8e98797dbaf3 not found: ID does not exist"
Sep 09 21:26:38.752774 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:26:38.752746    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-bv9hp"]
Sep 09 21:26:38.758817 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:26:38.758792    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-bv9hp"]
Sep 09 21:26:39.409626 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:26:39.409587    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a1a6b9c6-46bb-4e2a-a432-4711fed0c6b1" path="/var/lib/kubelet/pods/a1a6b9c6-46bb-4e2a-a432-4711fed0c6b1/volumes"
Sep 09 21:26:44.195942 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:26:44.195908    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:26:48.626048 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:26:48.626019    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-67fb486b4d-c2vj2"
Sep 09 21:27:00.575829 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:27:00.575718    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:27:00.655024 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:27:00.654983    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:27:05.555237 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:27:05.555201    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:27:08.796176 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:27:08.796147    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-cf47448b_tokenizer-clean-path-test-tokenizer-67fb486b4d-c2vj2_92bf4c52-eeb9-4bc3-a6ad-0410d0654f68/main/0.log"
Sep 09 21:27:16.309958 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:27:16.309924    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:27:16.405887 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:27:16.405852    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:27:40.243955 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:27:40.243917    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:27:40.280885 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:27:40.280852    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7585bbd4df-6szhg"]
Sep 09 21:27:40.290135 ip-10-0-134-72 kubenswrapper[2576]: E0909 21:27:40.290088    2576 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 09 21:27:40.290135 ip-10-0-134-72 kubenswrapper[2576]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 21:27:40.290135 ip-10-0-134-72 kubenswrapper[2576]:         , stdout: , stderr: , exit code -1
Sep 09 21:27:40.290135 ip-10-0-134-72 kubenswrapper[2576]:  > containerID="2f46580eff450e1c095c2f656f9ef4b44e6562995b7245ac72ac531887174d68" cmd=["/bin/sleep","15"]
Sep 09 21:27:40.290442 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:27:40.290152    2576 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7585bbd4df-6szhg" message="" err=<
Sep 09 21:27:40.290442 ip-10-0-134-72 kubenswrapper[2576]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 21:27:40.290442 ip-10-0-134-72 kubenswrapper[2576]:         , stdout: , stderr: , exit code -1
Sep 09 21:27:40.290442 ip-10-0-134-72 kubenswrapper[2576]:  >
Sep 09 21:27:40.290442 ip-10-0-134-72 kubenswrapper[2576]: E0909 21:27:40.290165    2576 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 09 21:27:40.290442 ip-10-0-134-72 kubenswrapper[2576]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 21:27:40.290442 ip-10-0-134-72 kubenswrapper[2576]:         , stdout: , stderr: , exit code -1
Sep 09 21:27:40.290442 ip-10-0-134-72 kubenswrapper[2576]:  > pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7585bbd4df-6szhg" podUID="e9b9638b-597f-42b1-9f5c-b329f244960d" containerName="main" containerID="cri-o://2f46580eff450e1c095c2f656f9ef4b44e6562995b7245ac72ac531887174d68"
Sep 09 21:27:40.290442 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:27:40.290204    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7585bbd4df-6szhg" podUID="e9b9638b-597f-42b1-9f5c-b329f244960d" containerName="main" containerID="cri-o://2f46580eff450e1c095c2f656f9ef4b44e6562995b7245ac72ac531887174d68" gracePeriod=60
Sep 09 21:27:40.291495 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:27:40.291473    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-67fb486b4d-c2vj2"]
Sep 09 21:27:40.291731 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:27:40.291687    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-67fb486b4d-c2vj2" podUID="92bf4c52-eeb9-4bc3-a6ad-0410d0654f68" containerName="main" containerID="cri-o://be35faa462d80716823c40a854519888cadd6397692d5d36652d87d3b74b6b24" gracePeriod=30
Sep 09 21:27:40.339510 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:27:40.339480    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:27:40.362071 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:27:40.362031    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7585bbd4df-6szhg" podUID="e9b9638b-597f-42b1-9f5c-b329f244960d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": dial tcp 10.132.0.62:8000: connect: connection refused"
Sep 09 21:27:40.413795 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:27:40.413770    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:27:40.634871 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:27:40.634845    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7585bbd4df-6szhg"
Sep 09 21:27:40.685666 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:27:40.685642    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9b9638b-597f-42b1-9f5c-b329f244960d-home\") pod \"e9b9638b-597f-42b1-9f5c-b329f244960d\" (UID: \"e9b9638b-597f-42b1-9f5c-b329f244960d\") "
Sep 09 21:27:40.685766 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:27:40.685680    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9b9638b-597f-42b1-9f5c-b329f244960d-tls-certs\") pod \"e9b9638b-597f-42b1-9f5c-b329f244960d\" (UID: \"e9b9638b-597f-42b1-9f5c-b329f244960d\") "
Sep 09 21:27:40.685766 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:27:40.685731    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e9b9638b-597f-42b1-9f5c-b329f244960d-cabundle-cert\") pod \"e9b9638b-597f-42b1-9f5c-b329f244960d\" (UID: \"e9b9638b-597f-42b1-9f5c-b329f244960d\") "
Sep 09 21:27:40.685766 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:27:40.685757    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9b9638b-597f-42b1-9f5c-b329f244960d-tmp-dir\") pod \"e9b9638b-597f-42b1-9f5c-b329f244960d\" (UID: \"e9b9638b-597f-42b1-9f5c-b329f244960d\") "
Sep 09 21:27:40.685930 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:27:40.685812    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e9b9638b-597f-42b1-9f5c-b329f244960d-kserve-provision-location\") pod \"e9b9638b-597f-42b1-9f5c-b329f244960d\" (UID: \"e9b9638b-597f-42b1-9f5c-b329f244960d\") "
Sep 09 21:27:40.685930 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:27:40.685843    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9b9638b-597f-42b1-9f5c-b329f244960d-model-cache\") pod \"e9b9638b-597f-42b1-9f5c-b329f244960d\" (UID: \"e9b9638b-597f-42b1-9f5c-b329f244960d\") "
Sep 09 21:27:40.685930 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:27:40.685867    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9b9638b-597f-42b1-9f5c-b329f244960d-dshm\") pod \"e9b9638b-597f-42b1-9f5c-b329f244960d\" (UID: \"e9b9638b-597f-42b1-9f5c-b329f244960d\") "
Sep 09 21:27:40.685930 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:27:40.685901    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n57z2\" (UniqueName: \"kubernetes.io/projected/e9b9638b-597f-42b1-9f5c-b329f244960d-kube-api-access-n57z2\") pod \"e9b9638b-597f-42b1-9f5c-b329f244960d\" (UID: \"e9b9638b-597f-42b1-9f5c-b329f244960d\") "
Sep 09 21:27:40.686248 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:27:40.685994    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9b9638b-597f-42b1-9f5c-b329f244960d-home" (OuterVolumeSpecName: "home") pod "e9b9638b-597f-42b1-9f5c-b329f244960d" (UID: "e9b9638b-597f-42b1-9f5c-b329f244960d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:27:40.686248 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:27:40.686010    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9b9638b-597f-42b1-9f5c-b329f244960d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e9b9638b-597f-42b1-9f5c-b329f244960d" (UID: "e9b9638b-597f-42b1-9f5c-b329f244960d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:27:40.686248 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:27:40.686213    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9b9638b-597f-42b1-9f5c-b329f244960d-tmp-dir\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:27:40.686248 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:27:40.686235    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9b9638b-597f-42b1-9f5c-b329f244960d-home\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:27:40.686771 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:27:40.686166    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9b9638b-597f-42b1-9f5c-b329f244960d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e9b9638b-597f-42b1-9f5c-b329f244960d" (UID: "e9b9638b-597f-42b1-9f5c-b329f244960d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:27:40.686993 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:27:40.686967    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9b9638b-597f-42b1-9f5c-b329f244960d-model-cache" (OuterVolumeSpecName: "model-cache") pod "e9b9638b-597f-42b1-9f5c-b329f244960d" (UID: "e9b9638b-597f-42b1-9f5c-b329f244960d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:27:40.688193 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:27:40.688170    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9b9638b-597f-42b1-9f5c-b329f244960d-dshm" (OuterVolumeSpecName: "dshm") pod "e9b9638b-597f-42b1-9f5c-b329f244960d" (UID: "e9b9638b-597f-42b1-9f5c-b329f244960d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:27:40.688276 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:27:40.688217    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9b9638b-597f-42b1-9f5c-b329f244960d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e9b9638b-597f-42b1-9f5c-b329f244960d" (UID: "e9b9638b-597f-42b1-9f5c-b329f244960d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:27:40.688514 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:27:40.688495    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9b9638b-597f-42b1-9f5c-b329f244960d-kube-api-access-n57z2" (OuterVolumeSpecName: "kube-api-access-n57z2") pod "e9b9638b-597f-42b1-9f5c-b329f244960d" (UID: "e9b9638b-597f-42b1-9f5c-b329f244960d"). InnerVolumeSpecName "kube-api-access-n57z2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:27:40.707356 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:27:40.707324    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9b9638b-597f-42b1-9f5c-b329f244960d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e9b9638b-597f-42b1-9f5c-b329f244960d" (UID: "e9b9638b-597f-42b1-9f5c-b329f244960d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:27:40.787039 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:27:40.787007    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9b9638b-597f-42b1-9f5c-b329f244960d-tls-certs\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:27:40.787039 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:27:40.787037    2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e9b9638b-597f-42b1-9f5c-b329f244960d-cabundle-cert\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:27:40.787234 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:27:40.787046    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e9b9638b-597f-42b1-9f5c-b329f244960d-kserve-provision-location\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:27:40.787234 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:27:40.787055    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9b9638b-597f-42b1-9f5c-b329f244960d-model-cache\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:27:40.787234 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:27:40.787070    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9b9638b-597f-42b1-9f5c-b329f244960d-dshm\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:27:40.787234 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:27:40.787079    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n57z2\" (UniqueName: \"kubernetes.io/projected/e9b9638b-597f-42b1-9f5c-b329f244960d-kube-api-access-n57z2\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:27:40.847573 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:27:40.847545    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-67fb486b4d-c2vj2"
Sep 09 21:27:40.887954 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:27:40.887923    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/92bf4c52-eeb9-4bc3-a6ad-0410d0654f68-kserve-provision-location\") pod \"92bf4c52-eeb9-4bc3-a6ad-0410d0654f68\" (UID: \"92bf4c52-eeb9-4bc3-a6ad-0410d0654f68\") "
Sep 09 21:27:40.888113 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:27:40.887969    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92bf4c52-eeb9-4bc3-a6ad-0410d0654f68-tls-certs\") pod \"92bf4c52-eeb9-4bc3-a6ad-0410d0654f68\" (UID: \"92bf4c52-eeb9-4bc3-a6ad-0410d0654f68\") "
Sep 09 21:27:40.888113 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:27:40.888036    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/92bf4c52-eeb9-4bc3-a6ad-0410d0654f68-cabundle-cert\") pod \"92bf4c52-eeb9-4bc3-a6ad-0410d0654f68\" (UID: \"92bf4c52-eeb9-4bc3-a6ad-0410d0654f68\") "
Sep 09 21:27:40.888113 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:27:40.888052    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92bf4c52-eeb9-4bc3-a6ad-0410d0654f68-tmp-dir\") pod \"92bf4c52-eeb9-4bc3-a6ad-0410d0654f68\" (UID: \"92bf4c52-eeb9-4bc3-a6ad-0410d0654f68\") "
Sep 09 21:27:40.888427 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:27:40.888404    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92bf4c52-eeb9-4bc3-a6ad-0410d0654f68-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "92bf4c52-eeb9-4bc3-a6ad-0410d0654f68" (UID: "92bf4c52-eeb9-4bc3-a6ad-0410d0654f68"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:27:40.888592 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:27:40.888560    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92bf4c52-eeb9-4bc3-a6ad-0410d0654f68-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "92bf4c52-eeb9-4bc3-a6ad-0410d0654f68" (UID: "92bf4c52-eeb9-4bc3-a6ad-0410d0654f68"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:27:40.890150 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:27:40.890130    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92bf4c52-eeb9-4bc3-a6ad-0410d0654f68-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "92bf4c52-eeb9-4bc3-a6ad-0410d0654f68" (UID: "92bf4c52-eeb9-4bc3-a6ad-0410d0654f68"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:27:40.909309 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:27:40.909287    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92bf4c52-eeb9-4bc3-a6ad-0410d0654f68-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "92bf4c52-eeb9-4bc3-a6ad-0410d0654f68" (UID: "92bf4c52-eeb9-4bc3-a6ad-0410d0654f68"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:27:40.988754 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:27:40.988722    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92bf4c52-eeb9-4bc3-a6ad-0410d0654f68-tls-certs\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:27:40.988754 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:27:40.988748    2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/92bf4c52-eeb9-4bc3-a6ad-0410d0654f68-cabundle-cert\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:27:40.988754 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:27:40.988757    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92bf4c52-eeb9-4bc3-a6ad-0410d0654f68-tmp-dir\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:27:40.988954 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:27:40.988765    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/92bf4c52-eeb9-4bc3-a6ad-0410d0654f68-kserve-provision-location\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:27:41.050588 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:27:41.050509    2576 generic.go:356] "Generic (PLEG): container finished" podID="92bf4c52-eeb9-4bc3-a6ad-0410d0654f68" containerID="be35faa462d80716823c40a854519888cadd6397692d5d36652d87d3b74b6b24" exitCode=0
Sep 09 21:27:41.050588 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:27:41.050557    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-67fb486b4d-c2vj2"
Sep 09 21:27:41.050788 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:27:41.050586    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-67fb486b4d-c2vj2" event={"ID":"92bf4c52-eeb9-4bc3-a6ad-0410d0654f68","Type":"ContainerDied","Data":"be35faa462d80716823c40a854519888cadd6397692d5d36652d87d3b74b6b24"}
Sep 09 21:27:41.050788 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:27:41.050633    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-67fb486b4d-c2vj2" event={"ID":"92bf4c52-eeb9-4bc3-a6ad-0410d0654f68","Type":"ContainerDied","Data":"c566364acc3ce334ecee5fdc089a45130ff96868c4994e443711f6936b5e4d0e"}
Sep 09 21:27:41.050788 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:27:41.050656    2576 scope.go:117] "RemoveContainer" containerID="be35faa462d80716823c40a854519888cadd6397692d5d36652d87d3b74b6b24"
Sep 09 21:27:41.052336 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:27:41.052312    2576 generic.go:356] "Generic (PLEG): container finished" podID="e9b9638b-597f-42b1-9f5c-b329f244960d" containerID="2f46580eff450e1c095c2f656f9ef4b44e6562995b7245ac72ac531887174d68" exitCode=0
Sep 09 21:27:41.052472 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:27:41.052387    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7585bbd4df-6szhg"
Sep 09 21:27:41.052472 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:27:41.052413    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7585bbd4df-6szhg" event={"ID":"e9b9638b-597f-42b1-9f5c-b329f244960d","Type":"ContainerDied","Data":"2f46580eff450e1c095c2f656f9ef4b44e6562995b7245ac72ac531887174d68"}
Sep 09 21:27:41.052472 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:27:41.052469    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7585bbd4df-6szhg" event={"ID":"e9b9638b-597f-42b1-9f5c-b329f244960d","Type":"ContainerDied","Data":"b9b35363b6ffd773eafab375c18e09b3db1af3b1b2f0936f0a607f44b8c19f9c"}
Sep 09 21:27:41.060794 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:27:41.060779    2576 scope.go:117] "RemoveContainer" containerID="d9975dca5a9545b699f1cd6713b489a719da8e0db844f4d0afb32d03f93c4e8f"
Sep 09 21:27:41.069213 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:27:41.069193    2576 scope.go:117] "RemoveContainer" containerID="be35faa462d80716823c40a854519888cadd6397692d5d36652d87d3b74b6b24"
Sep 09 21:27:41.069491 ip-10-0-134-72 kubenswrapper[2576]: E0909 21:27:41.069473    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be35faa462d80716823c40a854519888cadd6397692d5d36652d87d3b74b6b24\": container with ID starting with be35faa462d80716823c40a854519888cadd6397692d5d36652d87d3b74b6b24 not found: ID does not exist" containerID="be35faa462d80716823c40a854519888cadd6397692d5d36652d87d3b74b6b24"
Sep 09 21:27:41.069574 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:27:41.069497    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be35faa462d80716823c40a854519888cadd6397692d5d36652d87d3b74b6b24"} err="failed to get container status \"be35faa462d80716823c40a854519888cadd6397692d5d36652d87d3b74b6b24\": rpc error: code = NotFound desc = could not find container \"be35faa462d80716823c40a854519888cadd6397692d5d36652d87d3b74b6b24\": container with ID starting with be35faa462d80716823c40a854519888cadd6397692d5d36652d87d3b74b6b24 not found: ID does not exist"
Sep 09 21:27:41.069574 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:27:41.069512    2576 scope.go:117] "RemoveContainer" containerID="d9975dca5a9545b699f1cd6713b489a719da8e0db844f4d0afb32d03f93c4e8f"
Sep 09 21:27:41.069730 ip-10-0-134-72 kubenswrapper[2576]: E0909 21:27:41.069712    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9975dca5a9545b699f1cd6713b489a719da8e0db844f4d0afb32d03f93c4e8f\": container with ID starting with d9975dca5a9545b699f1cd6713b489a719da8e0db844f4d0afb32d03f93c4e8f not found: ID does not exist" containerID="d9975dca5a9545b699f1cd6713b489a719da8e0db844f4d0afb32d03f93c4e8f"
Sep 09 21:27:41.069782 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:27:41.069735    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9975dca5a9545b699f1cd6713b489a719da8e0db844f4d0afb32d03f93c4e8f"} err="failed to get container status \"d9975dca5a9545b699f1cd6713b489a719da8e0db844f4d0afb32d03f93c4e8f\": rpc error: code = NotFound desc = could not find container \"d9975dca5a9545b699f1cd6713b489a719da8e0db844f4d0afb32d03f93c4e8f\": container with ID starting with d9975dca5a9545b699f1cd6713b489a719da8e0db844f4d0afb32d03f93c4e8f not found: ID does not exist"
Sep 09 21:27:41.069782 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:27:41.069748    2576 scope.go:117] "RemoveContainer" containerID="2f46580eff450e1c095c2f656f9ef4b44e6562995b7245ac72ac531887174d68"
Sep 09 21:27:41.078702 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:27:41.078683    2576 scope.go:117] "RemoveContainer" containerID="c1da8b9f0875aa126e982f9ab1c948212cf92637d095ebdafe0a90e8aaa9281d"
Sep 09 21:27:41.079165 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:27:41.079139    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-67fb486b4d-c2vj2"]
Sep 09 21:27:41.086841 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:27:41.086822    2576 scope.go:117] "RemoveContainer" containerID="2f46580eff450e1c095c2f656f9ef4b44e6562995b7245ac72ac531887174d68"
Sep 09 21:27:41.087121 ip-10-0-134-72 kubenswrapper[2576]: E0909 21:27:41.087101    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f46580eff450e1c095c2f656f9ef4b44e6562995b7245ac72ac531887174d68\": container with ID starting with 2f46580eff450e1c095c2f656f9ef4b44e6562995b7245ac72ac531887174d68 not found: ID does not exist" containerID="2f46580eff450e1c095c2f656f9ef4b44e6562995b7245ac72ac531887174d68"
Sep 09 21:27:41.087204 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:27:41.087128    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f46580eff450e1c095c2f656f9ef4b44e6562995b7245ac72ac531887174d68"} err="failed to get container status \"2f46580eff450e1c095c2f656f9ef4b44e6562995b7245ac72ac531887174d68\": rpc error: code = NotFound desc = could not find container \"2f46580eff450e1c095c2f656f9ef4b44e6562995b7245ac72ac531887174d68\": container with ID starting with 2f46580eff450e1c095c2f656f9ef4b44e6562995b7245ac72ac531887174d68 not found: ID does not exist"
Sep 09 21:27:41.087204 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:27:41.087149    2576 scope.go:117] "RemoveContainer" containerID="c1da8b9f0875aa126e982f9ab1c948212cf92637d095ebdafe0a90e8aaa9281d"
Sep 09 21:27:41.087476 ip-10-0-134-72 kubenswrapper[2576]: E0909 21:27:41.087451    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1da8b9f0875aa126e982f9ab1c948212cf92637d095ebdafe0a90e8aaa9281d\": container with ID starting with c1da8b9f0875aa126e982f9ab1c948212cf92637d095ebdafe0a90e8aaa9281d not found: ID does not exist" containerID="c1da8b9f0875aa126e982f9ab1c948212cf92637d095ebdafe0a90e8aaa9281d"
Sep 09 21:27:41.087550 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:27:41.087485    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1da8b9f0875aa126e982f9ab1c948212cf92637d095ebdafe0a90e8aaa9281d"} err="failed to get container status \"c1da8b9f0875aa126e982f9ab1c948212cf92637d095ebdafe0a90e8aaa9281d\": rpc error: code = NotFound desc = could not find container \"c1da8b9f0875aa126e982f9ab1c948212cf92637d095ebdafe0a90e8aaa9281d\": container with ID starting with c1da8b9f0875aa126e982f9ab1c948212cf92637d095ebdafe0a90e8aaa9281d not found: ID does not exist"
Sep 09 21:27:41.088428 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:27:41.088409    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-67fb486b4d-c2vj2"]
Sep 09 21:27:41.108223 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:27:41.108196    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7585bbd4df-6szhg"]
Sep 09 21:27:41.113996 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:27:41.113973    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7585bbd4df-6szhg"]
Sep 09 21:27:41.409738 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:27:41.409708    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="92bf4c52-eeb9-4bc3-a6ad-0410d0654f68" path="/var/lib/kubelet/pods/92bf4c52-eeb9-4bc3-a6ad-0410d0654f68/volumes"
Sep 09 21:27:41.410157 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:27:41.410143    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e9b9638b-597f-42b1-9f5c-b329f244960d" path="/var/lib/kubelet/pods/e9b9638b-597f-42b1-9f5c-b329f244960d/volumes"
Sep 09 21:27:43.130678 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:27:43.130639    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:27:43.223437 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:27:43.223384    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:27:54.741847 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:27:54.741809    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:28:11.188054 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:28:11.188011    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:28:11.328893 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:28:11.328862    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:28:11.453503 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:28:11.453404    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:28:16.219271 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:28:16.219225    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:28:26.281094 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:28:26.281060    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:28:29.252893 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:28:29.252812    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:28:29.773009 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:28:29.772978    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:28:29.878277 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:28:29.878238    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:28:37.032309 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:28:37.032278    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:28:43.506914 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:28:43.506882    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:28:43.608867 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:28:43.608837    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:28:53.501054 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:28:53.501016    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b95cf68d8-m87gf"]
Sep 09 21:28:53.501537 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:28:53.501515    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9b9638b-597f-42b1-9f5c-b329f244960d" containerName="storage-initializer"
Sep 09 21:28:53.501615 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:28:53.501541    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9b9638b-597f-42b1-9f5c-b329f244960d" containerName="storage-initializer"
Sep 09 21:28:53.501615 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:28:53.501558    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1a6b9c6-46bb-4e2a-a432-4711fed0c6b1" containerName="main"
Sep 09 21:28:53.501615 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:28:53.501566    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1a6b9c6-46bb-4e2a-a432-4711fed0c6b1" containerName="main"
Sep 09 21:28:53.501615 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:28:53.501579    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92bf4c52-eeb9-4bc3-a6ad-0410d0654f68" containerName="main"
Sep 09 21:28:53.501615 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:28:53.501587    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="92bf4c52-eeb9-4bc3-a6ad-0410d0654f68" containerName="main"
Sep 09 21:28:53.501615 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:28:53.501605    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92bf4c52-eeb9-4bc3-a6ad-0410d0654f68" containerName="storage-initializer"
Sep 09 21:28:53.501615 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:28:53.501614    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="92bf4c52-eeb9-4bc3-a6ad-0410d0654f68" containerName="storage-initializer"
Sep 09 21:28:53.501969 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:28:53.501626    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9b9638b-597f-42b1-9f5c-b329f244960d" containerName="main"
Sep 09 21:28:53.501969 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:28:53.501635    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9b9638b-597f-42b1-9f5c-b329f244960d" containerName="main"
Sep 09 21:28:53.501969 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:28:53.501729    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1a6b9c6-46bb-4e2a-a432-4711fed0c6b1" containerName="main"
Sep 09 21:28:53.501969 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:28:53.501742    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9b9638b-597f-42b1-9f5c-b329f244960d" containerName="main"
Sep 09 21:28:53.501969 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:28:53.501753    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="92bf4c52-eeb9-4bc3-a6ad-0410d0654f68" containerName="main"
Sep 09 21:28:53.504846 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:28:53.504825    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b95cf68d8-m87gf"
Sep 09 21:28:53.508062 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:28:53.508043    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"default-dockercfg-zkt28\""
Sep 09 21:28:53.508484 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:28:53.508458    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"tokenizer-migration-path-test-kserve-self-signed-certs\""
Sep 09 21:28:53.508599 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:28:53.508583    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"kube-root-ca.crt\""
Sep 09 21:28:53.508676 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:28:53.508661    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"seaweedfs-s3-creds\""
Sep 09 21:28:53.508795 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:28:53.508781    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"openshift-service-ca.crt\""
Sep 09 21:28:53.508856 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:28:53.508830    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"odh-kserve-custom-ca-bundle\""
Sep 09 21:28:53.524572 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:28:53.524551    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b95cf68d8-m87gf"]
Sep 09 21:28:53.547671 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:28:53.547637    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:28:53.585642 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:28:53.585617    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a12320a4-fe4c-457e-9733-f9a59b4cfc32-home\") pod \"tokenizer-migration-path-test-kserve-7b95cf68d8-m87gf\" (UID: \"a12320a4-fe4c-457e-9733-f9a59b4cfc32\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b95cf68d8-m87gf"
Sep 09 21:28:53.585758 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:28:53.585653    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a12320a4-fe4c-457e-9733-f9a59b4cfc32-model-cache\") pod \"tokenizer-migration-path-test-kserve-7b95cf68d8-m87gf\" (UID: \"a12320a4-fe4c-457e-9733-f9a59b4cfc32\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b95cf68d8-m87gf"
Sep 09 21:28:53.585758 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:28:53.585671    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnh9s\" (UniqueName: \"kubernetes.io/projected/a12320a4-fe4c-457e-9733-f9a59b4cfc32-kube-api-access-tnh9s\") pod \"tokenizer-migration-path-test-kserve-7b95cf68d8-m87gf\" (UID: \"a12320a4-fe4c-457e-9733-f9a59b4cfc32\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b95cf68d8-m87gf"
Sep 09 21:28:53.585758 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:28:53.585745    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a12320a4-fe4c-457e-9733-f9a59b4cfc32-tls-certs\") pod \"tokenizer-migration-path-test-kserve-7b95cf68d8-m87gf\" (UID: \"a12320a4-fe4c-457e-9733-f9a59b4cfc32\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b95cf68d8-m87gf"
Sep 09 21:28:53.585870 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:28:53.585780    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a12320a4-fe4c-457e-9733-f9a59b4cfc32-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-7b95cf68d8-m87gf\" (UID: \"a12320a4-fe4c-457e-9733-f9a59b4cfc32\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b95cf68d8-m87gf"
Sep 09 21:28:53.585870 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:28:53.585796    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a12320a4-fe4c-457e-9733-f9a59b4cfc32-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-7b95cf68d8-m87gf\" (UID: \"a12320a4-fe4c-457e-9733-f9a59b4cfc32\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b95cf68d8-m87gf"
Sep 09 21:28:53.585870 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:28:53.585817    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a12320a4-fe4c-457e-9733-f9a59b4cfc32-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-7b95cf68d8-m87gf\" (UID: \"a12320a4-fe4c-457e-9733-f9a59b4cfc32\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b95cf68d8-m87gf"
Sep 09 21:28:53.585969 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:28:53.585864    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a12320a4-fe4c-457e-9733-f9a59b4cfc32-dshm\") pod \"tokenizer-migration-path-test-kserve-7b95cf68d8-m87gf\" (UID: \"a12320a4-fe4c-457e-9733-f9a59b4cfc32\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b95cf68d8-m87gf"
Sep 09 21:28:53.686578 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:28:53.686545    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a12320a4-fe4c-457e-9733-f9a59b4cfc32-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-7b95cf68d8-m87gf\" (UID: \"a12320a4-fe4c-457e-9733-f9a59b4cfc32\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b95cf68d8-m87gf"
Sep 09 21:28:53.686578 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:28:53.686579    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a12320a4-fe4c-457e-9733-f9a59b4cfc32-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-7b95cf68d8-m87gf\" (UID: \"a12320a4-fe4c-457e-9733-f9a59b4cfc32\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b95cf68d8-m87gf"
Sep 09 21:28:53.686794 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:28:53.686598    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a12320a4-fe4c-457e-9733-f9a59b4cfc32-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-7b95cf68d8-m87gf\" (UID: \"a12320a4-fe4c-457e-9733-f9a59b4cfc32\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b95cf68d8-m87gf"
Sep 09 21:28:53.686794 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:28:53.686623    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a12320a4-fe4c-457e-9733-f9a59b4cfc32-dshm\") pod \"tokenizer-migration-path-test-kserve-7b95cf68d8-m87gf\" (UID: \"a12320a4-fe4c-457e-9733-f9a59b4cfc32\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b95cf68d8-m87gf"
Sep 09 21:28:53.686794 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:28:53.686670    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a12320a4-fe4c-457e-9733-f9a59b4cfc32-home\") pod \"tokenizer-migration-path-test-kserve-7b95cf68d8-m87gf\" (UID: \"a12320a4-fe4c-457e-9733-f9a59b4cfc32\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b95cf68d8-m87gf"
Sep 09 21:28:53.686794 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:28:53.686694    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a12320a4-fe4c-457e-9733-f9a59b4cfc32-model-cache\") pod \"tokenizer-migration-path-test-kserve-7b95cf68d8-m87gf\" (UID: \"a12320a4-fe4c-457e-9733-f9a59b4cfc32\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b95cf68d8-m87gf"
Sep 09 21:28:53.686794 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:28:53.686718    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnh9s\" (UniqueName: \"kubernetes.io/projected/a12320a4-fe4c-457e-9733-f9a59b4cfc32-kube-api-access-tnh9s\") pod \"tokenizer-migration-path-test-kserve-7b95cf68d8-m87gf\" (UID: \"a12320a4-fe4c-457e-9733-f9a59b4cfc32\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b95cf68d8-m87gf"
Sep 09 21:28:53.686794 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:28:53.686772    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a12320a4-fe4c-457e-9733-f9a59b4cfc32-tls-certs\") pod \"tokenizer-migration-path-test-kserve-7b95cf68d8-m87gf\" (UID: \"a12320a4-fe4c-457e-9733-f9a59b4cfc32\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b95cf68d8-m87gf"
Sep 09 21:28:53.687096 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:28:53.687049    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a12320a4-fe4c-457e-9733-f9a59b4cfc32-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-7b95cf68d8-m87gf\" (UID: \"a12320a4-fe4c-457e-9733-f9a59b4cfc32\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b95cf68d8-m87gf"
Sep 09 21:28:53.687167 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:28:53.687111    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a12320a4-fe4c-457e-9733-f9a59b4cfc32-home\") pod \"tokenizer-migration-path-test-kserve-7b95cf68d8-m87gf\" (UID: \"a12320a4-fe4c-457e-9733-f9a59b4cfc32\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b95cf68d8-m87gf"
Sep 09 21:28:53.687224 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:28:53.687175    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a12320a4-fe4c-457e-9733-f9a59b4cfc32-model-cache\") pod \"tokenizer-migration-path-test-kserve-7b95cf68d8-m87gf\" (UID: \"a12320a4-fe4c-457e-9733-f9a59b4cfc32\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b95cf68d8-m87gf"
Sep 09 21:28:53.687282 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:28:53.687264    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a12320a4-fe4c-457e-9733-f9a59b4cfc32-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-7b95cf68d8-m87gf\" (UID: \"a12320a4-fe4c-457e-9733-f9a59b4cfc32\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b95cf68d8-m87gf"
Sep 09 21:28:53.687476 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:28:53.687435    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a12320a4-fe4c-457e-9733-f9a59b4cfc32-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-7b95cf68d8-m87gf\" (UID: \"a12320a4-fe4c-457e-9733-f9a59b4cfc32\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b95cf68d8-m87gf"
Sep 09 21:28:53.689064 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:28:53.689046    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a12320a4-fe4c-457e-9733-f9a59b4cfc32-dshm\") pod \"tokenizer-migration-path-test-kserve-7b95cf68d8-m87gf\" (UID: \"a12320a4-fe4c-457e-9733-f9a59b4cfc32\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b95cf68d8-m87gf"
Sep 09 21:28:53.689319 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:28:53.689302    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a12320a4-fe4c-457e-9733-f9a59b4cfc32-tls-certs\") pod \"tokenizer-migration-path-test-kserve-7b95cf68d8-m87gf\" (UID: \"a12320a4-fe4c-457e-9733-f9a59b4cfc32\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b95cf68d8-m87gf"
Sep 09 21:28:53.695748 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:28:53.695715    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnh9s\" (UniqueName: \"kubernetes.io/projected/a12320a4-fe4c-457e-9733-f9a59b4cfc32-kube-api-access-tnh9s\") pod \"tokenizer-migration-path-test-kserve-7b95cf68d8-m87gf\" (UID: \"a12320a4-fe4c-457e-9733-f9a59b4cfc32\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b95cf68d8-m87gf"
Sep 09 21:28:53.815872 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:28:53.815778    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b95cf68d8-m87gf"
Sep 09 21:28:53.884531 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:28:53.883557    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-7c58c9c9b6-kcfxs"]
Sep 09 21:28:53.887394 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:28:53.887356    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-7c58c9c9b6-kcfxs"
Sep 09 21:28:53.888214 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:28:53.888194    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab94b2ff-c377-4343-8a93-3991521a1b19-tmp-dir\") pod \"tokenizer-migration-path-test-tokenizer-7c58c9c9b6-kcfxs\" (UID: \"ab94b2ff-c377-4343-8a93-3991521a1b19\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-7c58c9c9b6-kcfxs"
Sep 09 21:28:53.888433 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:28:53.888413    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab94b2ff-c377-4343-8a93-3991521a1b19-tls-certs\") pod \"tokenizer-migration-path-test-tokenizer-7c58c9c9b6-kcfxs\" (UID: \"ab94b2ff-c377-4343-8a93-3991521a1b19\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-7c58c9c9b6-kcfxs"
Sep 09 21:28:53.888600 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:28:53.888581    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ab94b2ff-c377-4343-8a93-3991521a1b19-cabundle-cert\") pod \"tokenizer-migration-path-test-tokenizer-7c58c9c9b6-kcfxs\" (UID: \"ab94b2ff-c377-4343-8a93-3991521a1b19\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-7c58c9c9b6-kcfxs"
Sep 09 21:28:53.888760 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:28:53.888745    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ab94b2ff-c377-4343-8a93-3991521a1b19-kserve-provision-location\") pod \"tokenizer-migration-path-test-tokenizer-7c58c9c9b6-kcfxs\" (UID: \"ab94b2ff-c377-4343-8a93-3991521a1b19\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-7c58c9c9b6-kcfxs"
Sep 09 21:28:53.899376 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:28:53.899340    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-7c58c9c9b6-kcfxs"]
Sep 09 21:28:53.916511 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:28:53.916485    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:28:53.989135 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:28:53.989106    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab94b2ff-c377-4343-8a93-3991521a1b19-tmp-dir\") pod \"tokenizer-migration-path-test-tokenizer-7c58c9c9b6-kcfxs\" (UID: \"ab94b2ff-c377-4343-8a93-3991521a1b19\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-7c58c9c9b6-kcfxs"
Sep 09 21:28:53.989290 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:28:53.989153    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab94b2ff-c377-4343-8a93-3991521a1b19-tls-certs\") pod \"tokenizer-migration-path-test-tokenizer-7c58c9c9b6-kcfxs\" (UID: \"ab94b2ff-c377-4343-8a93-3991521a1b19\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-7c58c9c9b6-kcfxs"
Sep 09 21:28:53.989290 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:28:53.989194    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ab94b2ff-c377-4343-8a93-3991521a1b19-cabundle-cert\") pod \"tokenizer-migration-path-test-tokenizer-7c58c9c9b6-kcfxs\" (UID: \"ab94b2ff-c377-4343-8a93-3991521a1b19\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-7c58c9c9b6-kcfxs"
Sep 09 21:28:53.989290 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:28:53.989236    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ab94b2ff-c377-4343-8a93-3991521a1b19-kserve-provision-location\") pod \"tokenizer-migration-path-test-tokenizer-7c58c9c9b6-kcfxs\" (UID: \"ab94b2ff-c377-4343-8a93-3991521a1b19\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-7c58c9c9b6-kcfxs"
Sep 09 21:28:53.989633 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:28:53.989611    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab94b2ff-c377-4343-8a93-3991521a1b19-tmp-dir\") pod \"tokenizer-migration-path-test-tokenizer-7c58c9c9b6-kcfxs\" (UID: \"ab94b2ff-c377-4343-8a93-3991521a1b19\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-7c58c9c9b6-kcfxs"
Sep 09 21:28:53.989633 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:28:53.989625    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ab94b2ff-c377-4343-8a93-3991521a1b19-kserve-provision-location\") pod \"tokenizer-migration-path-test-tokenizer-7c58c9c9b6-kcfxs\" (UID: \"ab94b2ff-c377-4343-8a93-3991521a1b19\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-7c58c9c9b6-kcfxs"
Sep 09 21:28:53.989810 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:28:53.989788    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ab94b2ff-c377-4343-8a93-3991521a1b19-cabundle-cert\") pod \"tokenizer-migration-path-test-tokenizer-7c58c9c9b6-kcfxs\" (UID: \"ab94b2ff-c377-4343-8a93-3991521a1b19\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-7c58c9c9b6-kcfxs"
Sep 09 21:28:53.991746 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:28:53.991724    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab94b2ff-c377-4343-8a93-3991521a1b19-tls-certs\") pod \"tokenizer-migration-path-test-tokenizer-7c58c9c9b6-kcfxs\" (UID: \"ab94b2ff-c377-4343-8a93-3991521a1b19\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-7c58c9c9b6-kcfxs"
Sep 09 21:28:54.007924 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:28:54.007898    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:28:54.114092 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:28:54.109754    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b95cf68d8-m87gf"]
Sep 09 21:28:54.117928 ip-10-0-134-72 kubenswrapper[2576]: W0909 21:28:54.117903    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda12320a4_fe4c_457e_9733_f9a59b4cfc32.slice/crio-be1bf82abaa255b7aa8da24da08fc0c793bb767ba87f4c86a821c37d94eda000 WatchSource:0}: Error finding container be1bf82abaa255b7aa8da24da08fc0c793bb767ba87f4c86a821c37d94eda000: Status 404 returned error can't find the container with id be1bf82abaa255b7aa8da24da08fc0c793bb767ba87f4c86a821c37d94eda000
Sep 09 21:28:54.199672 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:28:54.199650    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-7c58c9c9b6-kcfxs"
Sep 09 21:28:54.426448 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:28:54.426351    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b95cf68d8-m87gf" event={"ID":"a12320a4-fe4c-457e-9733-f9a59b4cfc32","Type":"ContainerStarted","Data":"5820327e1c27fd2111f68ed1f97adb8a9b3a8710085feb0b454aac6b2fd2aadd"}
Sep 09 21:28:54.426448 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:28:54.426412    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b95cf68d8-m87gf" event={"ID":"a12320a4-fe4c-457e-9733-f9a59b4cfc32","Type":"ContainerStarted","Data":"be1bf82abaa255b7aa8da24da08fc0c793bb767ba87f4c86a821c37d94eda000"}
Sep 09 21:28:54.483393 ip-10-0-134-72 kubenswrapper[2576]: W0909 21:28:54.483347    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab94b2ff_c377_4343_8a93_3991521a1b19.slice/crio-07186e473c44066caa4c509117d54ea49203da8ed18da50939338030a226920c WatchSource:0}: Error finding container 07186e473c44066caa4c509117d54ea49203da8ed18da50939338030a226920c: Status 404 returned error can't find the container with id 07186e473c44066caa4c509117d54ea49203da8ed18da50939338030a226920c
Sep 09 21:28:54.484061 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:28:54.483983    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-7c58c9c9b6-kcfxs"]
Sep 09 21:28:55.433051 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:28:55.433016    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-7c58c9c9b6-kcfxs" event={"ID":"ab94b2ff-c377-4343-8a93-3991521a1b19","Type":"ContainerStarted","Data":"b39ca1ab17227b0c8dddc5a6f787de1f90e25ea45bf1c8b06efb39fcef53a860"}
Sep 09 21:28:55.433560 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:28:55.433069    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-7c58c9c9b6-kcfxs" event={"ID":"ab94b2ff-c377-4343-8a93-3991521a1b19","Type":"ContainerStarted","Data":"07186e473c44066caa4c509117d54ea49203da8ed18da50939338030a226920c"}
Sep 09 21:28:55.638277 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:28:55.638244    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:28:57.444008 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:28:57.443971    2576 generic.go:356] "Generic (PLEG): container finished" podID="a12320a4-fe4c-457e-9733-f9a59b4cfc32" containerID="5820327e1c27fd2111f68ed1f97adb8a9b3a8710085feb0b454aac6b2fd2aadd" exitCode=0
Sep 09 21:28:57.444409 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:28:57.444047    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b95cf68d8-m87gf" event={"ID":"a12320a4-fe4c-457e-9733-f9a59b4cfc32","Type":"ContainerDied","Data":"5820327e1c27fd2111f68ed1f97adb8a9b3a8710085feb0b454aac6b2fd2aadd"}
Sep 09 21:28:57.445674 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:28:57.445651    2576 generic.go:356] "Generic (PLEG): container finished" podID="ab94b2ff-c377-4343-8a93-3991521a1b19" containerID="b39ca1ab17227b0c8dddc5a6f787de1f90e25ea45bf1c8b06efb39fcef53a860" exitCode=0
Sep 09 21:28:57.445792 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:28:57.445723    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-7c58c9c9b6-kcfxs" event={"ID":"ab94b2ff-c377-4343-8a93-3991521a1b19","Type":"ContainerDied","Data":"b39ca1ab17227b0c8dddc5a6f787de1f90e25ea45bf1c8b06efb39fcef53a860"}
Sep 09 21:28:58.452968 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:28:58.452932    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b95cf68d8-m87gf" event={"ID":"a12320a4-fe4c-457e-9733-f9a59b4cfc32","Type":"ContainerStarted","Data":"1352f57ac0ac962d0a931cb8a8c057b74d5b647bcd877727eb451e1141c42f3b"}
Sep 09 21:28:58.454782 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:28:58.454758    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-7c58c9c9b6-kcfxs" event={"ID":"ab94b2ff-c377-4343-8a93-3991521a1b19","Type":"ContainerStarted","Data":"0cb0ad2f606aa0e51ee48ce9e2abbf87737956afd40664713453fca048524e8f"}
Sep 09 21:28:58.454957 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:28:58.454940    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-7c58c9c9b6-kcfxs"
Sep 09 21:28:58.544701 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:28:58.544648    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-7c58c9c9b6-kcfxs" podStartSLOduration=3.132165289 podStartE2EDuration="5.544629016s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.412463727s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:28:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:28:58.544217386 +0000 UTC m=+2181.657903244" watchObservedRunningTime="2026-09-09 21:28:58.544629016 +0000 UTC m=+2181.658314874"
Sep 09 21:28:58.545906 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:28:58.545874    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b95cf68d8-m87gf" podStartSLOduration=3.079301201 podStartE2EDuration="5.545863247s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.466562046s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:28:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:28:58.502617962 +0000 UTC m=+2181.616303821" watchObservedRunningTime="2026-09-09 21:28:58.545863247 +0000 UTC m=+2181.659549105"
Sep 09 21:28:58.816910 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:28:58.816803    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b95cf68d8-m87gf"
Sep 09 21:28:59.775004 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:28:59.774946    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:29:01.955095 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:29:01.955053    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:29:02.055020 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:29:02.054972    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:29:03.815964 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:29:03.815925    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b95cf68d8-m87gf"
Sep 09 21:29:03.828757 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:29:03.828726    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b95cf68d8-m87gf"
Sep 09 21:29:04.497315 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:29:04.497283    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b95cf68d8-m87gf"
Sep 09 21:29:06.391299 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:29:06.391251    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:29:06.520986 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:29:06.520959    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:29:06.647673 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:29:06.647581    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:29:10.223887 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:29:10.223853    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:29:11.452417 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:29:11.452352    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:29:21.007165 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:29:21.007121    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:29:22.961386 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:29:22.961333    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:29:29.470823 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:29:29.470786    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-7c58c9c9b6-kcfxs"
Sep 09 21:29:36.103604 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:29:36.103572    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:29:37.590170 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:29:37.590130    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:29:49.499543 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:29:49.499500    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:29:54.711013 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:29:54.710971    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:29:54.815580 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:29:54.815539    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:30:02.623687 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:30:02.623601    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:30:05.392150 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:30:05.392114    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-312caf2d_tokenizer-migration-path-test-tokenizer-7c58c9c9b6-kcfxs_ab94b2ff-c377-4343-8a93-3991521a1b19/main/0.log"
Sep 09 21:30:22.806247 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:30:22.806203    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:30:22.818501 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:30:22.818470    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b95cf68d8-m87gf"]
Sep 09 21:30:22.828136 ip-10-0-134-72 kubenswrapper[2576]: E0909 21:30:22.828096    2576 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 09 21:30:22.828136 ip-10-0-134-72 kubenswrapper[2576]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 21:30:22.828136 ip-10-0-134-72 kubenswrapper[2576]:         , stdout: , stderr: , exit code -1
Sep 09 21:30:22.828136 ip-10-0-134-72 kubenswrapper[2576]:  > containerID="1352f57ac0ac962d0a931cb8a8c057b74d5b647bcd877727eb451e1141c42f3b" cmd=["/bin/sleep","15"]
Sep 09 21:30:22.828516 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:30:22.828152    2576 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b95cf68d8-m87gf" message="" err=<
Sep 09 21:30:22.828516 ip-10-0-134-72 kubenswrapper[2576]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 21:30:22.828516 ip-10-0-134-72 kubenswrapper[2576]:         , stdout: , stderr: , exit code -1
Sep 09 21:30:22.828516 ip-10-0-134-72 kubenswrapper[2576]:  >
Sep 09 21:30:22.828516 ip-10-0-134-72 kubenswrapper[2576]: E0909 21:30:22.828165    2576 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 09 21:30:22.828516 ip-10-0-134-72 kubenswrapper[2576]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 21:30:22.828516 ip-10-0-134-72 kubenswrapper[2576]:         , stdout: , stderr: , exit code -1
Sep 09 21:30:22.828516 ip-10-0-134-72 kubenswrapper[2576]:  > pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b95cf68d8-m87gf" podUID="a12320a4-fe4c-457e-9733-f9a59b4cfc32" containerName="main" containerID="cri-o://1352f57ac0ac962d0a931cb8a8c057b74d5b647bcd877727eb451e1141c42f3b"
Sep 09 21:30:22.828516 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:30:22.828215    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b95cf68d8-m87gf" podUID="a12320a4-fe4c-457e-9733-f9a59b4cfc32" containerName="main" containerID="cri-o://1352f57ac0ac962d0a931cb8a8c057b74d5b647bcd877727eb451e1141c42f3b" gracePeriod=60
Sep 09 21:30:22.833387 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:30:22.833347    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-7c58c9c9b6-kcfxs"]
Sep 09 21:30:22.833583 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:30:22.833564    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-7c58c9c9b6-kcfxs" podUID="ab94b2ff-c377-4343-8a93-3991521a1b19" containerName="main" containerID="cri-o://0cb0ad2f606aa0e51ee48ce9e2abbf87737956afd40664713453fca048524e8f" gracePeriod=30
Sep 09 21:30:22.895589 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:30:22.895552    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:30:23.172924 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:30:23.172896    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b95cf68d8-m87gf"
Sep 09 21:30:23.253011 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:30:23.252979    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a12320a4-fe4c-457e-9733-f9a59b4cfc32-home\") pod \"a12320a4-fe4c-457e-9733-f9a59b4cfc32\" (UID: \"a12320a4-fe4c-457e-9733-f9a59b4cfc32\") "
Sep 09 21:30:23.253219 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:30:23.253034    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a12320a4-fe4c-457e-9733-f9a59b4cfc32-cabundle-cert\") pod \"a12320a4-fe4c-457e-9733-f9a59b4cfc32\" (UID: \"a12320a4-fe4c-457e-9733-f9a59b4cfc32\") "
Sep 09 21:30:23.253219 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:30:23.253052    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a12320a4-fe4c-457e-9733-f9a59b4cfc32-kserve-provision-location\") pod \"a12320a4-fe4c-457e-9733-f9a59b4cfc32\" (UID: \"a12320a4-fe4c-457e-9733-f9a59b4cfc32\") "
Sep 09 21:30:23.253219 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:30:23.253071    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a12320a4-fe4c-457e-9733-f9a59b4cfc32-dshm\") pod \"a12320a4-fe4c-457e-9733-f9a59b4cfc32\" (UID: \"a12320a4-fe4c-457e-9733-f9a59b4cfc32\") "
Sep 09 21:30:23.253219 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:30:23.253108    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a12320a4-fe4c-457e-9733-f9a59b4cfc32-tls-certs\") pod \"a12320a4-fe4c-457e-9733-f9a59b4cfc32\" (UID: \"a12320a4-fe4c-457e-9733-f9a59b4cfc32\") "
Sep 09 21:30:23.253219 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:30:23.253134    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a12320a4-fe4c-457e-9733-f9a59b4cfc32-model-cache\") pod \"a12320a4-fe4c-457e-9733-f9a59b4cfc32\" (UID: \"a12320a4-fe4c-457e-9733-f9a59b4cfc32\") "
Sep 09 21:30:23.253219 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:30:23.253178    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tnh9s\" (UniqueName: \"kubernetes.io/projected/a12320a4-fe4c-457e-9733-f9a59b4cfc32-kube-api-access-tnh9s\") pod \"a12320a4-fe4c-457e-9733-f9a59b4cfc32\" (UID: \"a12320a4-fe4c-457e-9733-f9a59b4cfc32\") "
Sep 09 21:30:23.253576 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:30:23.253306    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a12320a4-fe4c-457e-9733-f9a59b4cfc32-home" (OuterVolumeSpecName: "home") pod "a12320a4-fe4c-457e-9733-f9a59b4cfc32" (UID: "a12320a4-fe4c-457e-9733-f9a59b4cfc32"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:30:23.253576 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:30:23.253421    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a12320a4-fe4c-457e-9733-f9a59b4cfc32-model-cache" (OuterVolumeSpecName: "model-cache") pod "a12320a4-fe4c-457e-9733-f9a59b4cfc32" (UID: "a12320a4-fe4c-457e-9733-f9a59b4cfc32"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:30:23.253576 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:30:23.253428    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a12320a4-fe4c-457e-9733-f9a59b4cfc32-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a12320a4-fe4c-457e-9733-f9a59b4cfc32" (UID: "a12320a4-fe4c-457e-9733-f9a59b4cfc32"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:30:23.253576 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:30:23.253557    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a12320a4-fe4c-457e-9733-f9a59b4cfc32-tmp-dir\") pod \"a12320a4-fe4c-457e-9733-f9a59b4cfc32\" (UID: \"a12320a4-fe4c-457e-9733-f9a59b4cfc32\") "
Sep 09 21:30:23.253801 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:30:23.253764    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a12320a4-fe4c-457e-9733-f9a59b4cfc32-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a12320a4-fe4c-457e-9733-f9a59b4cfc32" (UID: "a12320a4-fe4c-457e-9733-f9a59b4cfc32"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:30:23.253869 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:30:23.253832    2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a12320a4-fe4c-457e-9733-f9a59b4cfc32-cabundle-cert\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:30:23.253869 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:30:23.253848    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a12320a4-fe4c-457e-9733-f9a59b4cfc32-model-cache\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:30:23.253869 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:30:23.253861    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a12320a4-fe4c-457e-9733-f9a59b4cfc32-home\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:30:23.255668 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:30:23.255641    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a12320a4-fe4c-457e-9733-f9a59b4cfc32-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a12320a4-fe4c-457e-9733-f9a59b4cfc32" (UID: "a12320a4-fe4c-457e-9733-f9a59b4cfc32"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:30:23.255995 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:30:23.255972    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a12320a4-fe4c-457e-9733-f9a59b4cfc32-dshm" (OuterVolumeSpecName: "dshm") pod "a12320a4-fe4c-457e-9733-f9a59b4cfc32" (UID: "a12320a4-fe4c-457e-9733-f9a59b4cfc32"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:30:23.256087 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:30:23.256034    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a12320a4-fe4c-457e-9733-f9a59b4cfc32-kube-api-access-tnh9s" (OuterVolumeSpecName: "kube-api-access-tnh9s") pod "a12320a4-fe4c-457e-9733-f9a59b4cfc32" (UID: "a12320a4-fe4c-457e-9733-f9a59b4cfc32"). InnerVolumeSpecName "kube-api-access-tnh9s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:30:23.275510 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:30:23.275485    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a12320a4-fe4c-457e-9733-f9a59b4cfc32-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a12320a4-fe4c-457e-9733-f9a59b4cfc32" (UID: "a12320a4-fe4c-457e-9733-f9a59b4cfc32"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:30:23.355247 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:30:23.355208    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a12320a4-fe4c-457e-9733-f9a59b4cfc32-kserve-provision-location\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:30:23.355416 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:30:23.355256    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a12320a4-fe4c-457e-9733-f9a59b4cfc32-dshm\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:30:23.355416 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:30:23.355276    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a12320a4-fe4c-457e-9733-f9a59b4cfc32-tls-certs\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:30:23.355416 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:30:23.355292    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tnh9s\" (UniqueName: \"kubernetes.io/projected/a12320a4-fe4c-457e-9733-f9a59b4cfc32-kube-api-access-tnh9s\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:30:23.355416 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:30:23.355306    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a12320a4-fe4c-457e-9733-f9a59b4cfc32-tmp-dir\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:30:23.479697 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:30:23.479672    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-7c58c9c9b6-kcfxs"
Sep 09 21:30:23.557016 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:30:23.556976    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab94b2ff-c377-4343-8a93-3991521a1b19-tls-certs\") pod \"ab94b2ff-c377-4343-8a93-3991521a1b19\" (UID: \"ab94b2ff-c377-4343-8a93-3991521a1b19\") "
Sep 09 21:30:23.557181 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:30:23.557029    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ab94b2ff-c377-4343-8a93-3991521a1b19-cabundle-cert\") pod \"ab94b2ff-c377-4343-8a93-3991521a1b19\" (UID: \"ab94b2ff-c377-4343-8a93-3991521a1b19\") "
Sep 09 21:30:23.557181 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:30:23.557141    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab94b2ff-c377-4343-8a93-3991521a1b19-tmp-dir\") pod \"ab94b2ff-c377-4343-8a93-3991521a1b19\" (UID: \"ab94b2ff-c377-4343-8a93-3991521a1b19\") "
Sep 09 21:30:23.557279 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:30:23.557199    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ab94b2ff-c377-4343-8a93-3991521a1b19-kserve-provision-location\") pod \"ab94b2ff-c377-4343-8a93-3991521a1b19\" (UID: \"ab94b2ff-c377-4343-8a93-3991521a1b19\") "
Sep 09 21:30:23.557357 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:30:23.557340    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab94b2ff-c377-4343-8a93-3991521a1b19-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ab94b2ff-c377-4343-8a93-3991521a1b19" (UID: "ab94b2ff-c377-4343-8a93-3991521a1b19"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:30:23.557558 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:30:23.557543    2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ab94b2ff-c377-4343-8a93-3991521a1b19-cabundle-cert\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:30:23.557637 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:30:23.557539    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab94b2ff-c377-4343-8a93-3991521a1b19-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ab94b2ff-c377-4343-8a93-3991521a1b19" (UID: "ab94b2ff-c377-4343-8a93-3991521a1b19"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:30:23.559323 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:30:23.559297    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab94b2ff-c377-4343-8a93-3991521a1b19-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ab94b2ff-c377-4343-8a93-3991521a1b19" (UID: "ab94b2ff-c377-4343-8a93-3991521a1b19"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:30:23.579353 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:30:23.579310    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab94b2ff-c377-4343-8a93-3991521a1b19-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ab94b2ff-c377-4343-8a93-3991521a1b19" (UID: "ab94b2ff-c377-4343-8a93-3991521a1b19"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:30:23.658862 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:30:23.658776    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab94b2ff-c377-4343-8a93-3991521a1b19-tmp-dir\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:30:23.658862 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:30:23.658809    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ab94b2ff-c377-4343-8a93-3991521a1b19-kserve-provision-location\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:30:23.658862 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:30:23.658823    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab94b2ff-c377-4343-8a93-3991521a1b19-tls-certs\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:30:23.893084 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:30:23.893047    2576 generic.go:356] "Generic (PLEG): container finished" podID="a12320a4-fe4c-457e-9733-f9a59b4cfc32" containerID="1352f57ac0ac962d0a931cb8a8c057b74d5b647bcd877727eb451e1141c42f3b" exitCode=0
Sep 09 21:30:23.893573 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:30:23.893087    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b95cf68d8-m87gf" event={"ID":"a12320a4-fe4c-457e-9733-f9a59b4cfc32","Type":"ContainerDied","Data":"1352f57ac0ac962d0a931cb8a8c057b74d5b647bcd877727eb451e1141c42f3b"}
Sep 09 21:30:23.893573 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:30:23.893123    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b95cf68d8-m87gf"
Sep 09 21:30:23.893573 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:30:23.893184    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b95cf68d8-m87gf" event={"ID":"a12320a4-fe4c-457e-9733-f9a59b4cfc32","Type":"ContainerDied","Data":"be1bf82abaa255b7aa8da24da08fc0c793bb767ba87f4c86a821c37d94eda000"}
Sep 09 21:30:23.893573 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:30:23.893214    2576 scope.go:117] "RemoveContainer" containerID="1352f57ac0ac962d0a931cb8a8c057b74d5b647bcd877727eb451e1141c42f3b"
Sep 09 21:30:23.895030 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:30:23.895008    2576 generic.go:356] "Generic (PLEG): container finished" podID="ab94b2ff-c377-4343-8a93-3991521a1b19" containerID="0cb0ad2f606aa0e51ee48ce9e2abbf87737956afd40664713453fca048524e8f" exitCode=0
Sep 09 21:30:23.895162 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:30:23.895066    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-7c58c9c9b6-kcfxs"
Sep 09 21:30:23.895236 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:30:23.895157    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-7c58c9c9b6-kcfxs" event={"ID":"ab94b2ff-c377-4343-8a93-3991521a1b19","Type":"ContainerDied","Data":"0cb0ad2f606aa0e51ee48ce9e2abbf87737956afd40664713453fca048524e8f"}
Sep 09 21:30:23.895236 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:30:23.895194    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-7c58c9c9b6-kcfxs" event={"ID":"ab94b2ff-c377-4343-8a93-3991521a1b19","Type":"ContainerDied","Data":"07186e473c44066caa4c509117d54ea49203da8ed18da50939338030a226920c"}
Sep 09 21:30:23.903129 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:30:23.903113    2576 scope.go:117] "RemoveContainer" containerID="5820327e1c27fd2111f68ed1f97adb8a9b3a8710085feb0b454aac6b2fd2aadd"
Sep 09 21:30:23.910836 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:30:23.910819    2576 scope.go:117] "RemoveContainer" containerID="1352f57ac0ac962d0a931cb8a8c057b74d5b647bcd877727eb451e1141c42f3b"
Sep 09 21:30:23.911106 ip-10-0-134-72 kubenswrapper[2576]: E0909 21:30:23.911087    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1352f57ac0ac962d0a931cb8a8c057b74d5b647bcd877727eb451e1141c42f3b\": container with ID starting with 1352f57ac0ac962d0a931cb8a8c057b74d5b647bcd877727eb451e1141c42f3b not found: ID does not exist" containerID="1352f57ac0ac962d0a931cb8a8c057b74d5b647bcd877727eb451e1141c42f3b"
Sep 09 21:30:23.911149 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:30:23.911115    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1352f57ac0ac962d0a931cb8a8c057b74d5b647bcd877727eb451e1141c42f3b"} err="failed to get container status \"1352f57ac0ac962d0a931cb8a8c057b74d5b647bcd877727eb451e1141c42f3b\": rpc error: code = NotFound desc = could not find container \"1352f57ac0ac962d0a931cb8a8c057b74d5b647bcd877727eb451e1141c42f3b\": container with ID starting with 1352f57ac0ac962d0a931cb8a8c057b74d5b647bcd877727eb451e1141c42f3b not found: ID does not exist"
Sep 09 21:30:23.911149 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:30:23.911131    2576 scope.go:117] "RemoveContainer" containerID="5820327e1c27fd2111f68ed1f97adb8a9b3a8710085feb0b454aac6b2fd2aadd"
Sep 09 21:30:23.911421 ip-10-0-134-72 kubenswrapper[2576]: E0909 21:30:23.911401    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5820327e1c27fd2111f68ed1f97adb8a9b3a8710085feb0b454aac6b2fd2aadd\": container with ID starting with 5820327e1c27fd2111f68ed1f97adb8a9b3a8710085feb0b454aac6b2fd2aadd not found: ID does not exist" containerID="5820327e1c27fd2111f68ed1f97adb8a9b3a8710085feb0b454aac6b2fd2aadd"
Sep 09 21:30:23.911489 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:30:23.911424    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5820327e1c27fd2111f68ed1f97adb8a9b3a8710085feb0b454aac6b2fd2aadd"} err="failed to get container status \"5820327e1c27fd2111f68ed1f97adb8a9b3a8710085feb0b454aac6b2fd2aadd\": rpc error: code = NotFound desc = could not find container \"5820327e1c27fd2111f68ed1f97adb8a9b3a8710085feb0b454aac6b2fd2aadd\": container with ID starting with 5820327e1c27fd2111f68ed1f97adb8a9b3a8710085feb0b454aac6b2fd2aadd not found: ID does not exist"
Sep 09 21:30:23.911489 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:30:23.911438    2576 scope.go:117] "RemoveContainer" containerID="0cb0ad2f606aa0e51ee48ce9e2abbf87737956afd40664713453fca048524e8f"
Sep 09 21:30:23.919273 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:30:23.917307    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b95cf68d8-m87gf"]
Sep 09 21:30:23.921618 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:30:23.921565    2576 scope.go:117] "RemoveContainer" containerID="b39ca1ab17227b0c8dddc5a6f787de1f90e25ea45bf1c8b06efb39fcef53a860"
Sep 09 21:30:23.924821 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:30:23.924799    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b95cf68d8-m87gf"]
Sep 09 21:30:23.931099 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:30:23.931083    2576 scope.go:117] "RemoveContainer" containerID="0cb0ad2f606aa0e51ee48ce9e2abbf87737956afd40664713453fca048524e8f"
Sep 09 21:30:23.931404 ip-10-0-134-72 kubenswrapper[2576]: E0909 21:30:23.931386    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0cb0ad2f606aa0e51ee48ce9e2abbf87737956afd40664713453fca048524e8f\": container with ID starting with 0cb0ad2f606aa0e51ee48ce9e2abbf87737956afd40664713453fca048524e8f not found: ID does not exist" containerID="0cb0ad2f606aa0e51ee48ce9e2abbf87737956afd40664713453fca048524e8f"
Sep 09 21:30:23.931464 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:30:23.931412    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0cb0ad2f606aa0e51ee48ce9e2abbf87737956afd40664713453fca048524e8f"} err="failed to get container status \"0cb0ad2f606aa0e51ee48ce9e2abbf87737956afd40664713453fca048524e8f\": rpc error: code = NotFound desc = could not find container \"0cb0ad2f606aa0e51ee48ce9e2abbf87737956afd40664713453fca048524e8f\": container with ID starting with 0cb0ad2f606aa0e51ee48ce9e2abbf87737956afd40664713453fca048524e8f not found: ID does not exist"
Sep 09 21:30:23.931464 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:30:23.931430    2576 scope.go:117] "RemoveContainer" containerID="b39ca1ab17227b0c8dddc5a6f787de1f90e25ea45bf1c8b06efb39fcef53a860"
Sep 09 21:30:23.931667 ip-10-0-134-72 kubenswrapper[2576]: E0909 21:30:23.931648    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b39ca1ab17227b0c8dddc5a6f787de1f90e25ea45bf1c8b06efb39fcef53a860\": container with ID starting with b39ca1ab17227b0c8dddc5a6f787de1f90e25ea45bf1c8b06efb39fcef53a860 not found: ID does not exist" containerID="b39ca1ab17227b0c8dddc5a6f787de1f90e25ea45bf1c8b06efb39fcef53a860"
Sep 09 21:30:23.931723 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:30:23.931676    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b39ca1ab17227b0c8dddc5a6f787de1f90e25ea45bf1c8b06efb39fcef53a860"} err="failed to get container status \"b39ca1ab17227b0c8dddc5a6f787de1f90e25ea45bf1c8b06efb39fcef53a860\": rpc error: code = NotFound desc = could not find container \"b39ca1ab17227b0c8dddc5a6f787de1f90e25ea45bf1c8b06efb39fcef53a860\": container with ID starting with b39ca1ab17227b0c8dddc5a6f787de1f90e25ea45bf1c8b06efb39fcef53a860 not found: ID does not exist"
Sep 09 21:30:23.938256 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:30:23.938232    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-7c58c9c9b6-kcfxs"]
Sep 09 21:30:23.942063 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:30:23.942042    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-7c58c9c9b6-kcfxs"]
Sep 09 21:30:25.409253 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:30:25.409215    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a12320a4-fe4c-457e-9733-f9a59b4cfc32" path="/var/lib/kubelet/pods/a12320a4-fe4c-457e-9733-f9a59b4cfc32/volumes"
Sep 09 21:30:25.409717 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:30:25.409699    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ab94b2ff-c377-4343-8a93-3991521a1b19" path="/var/lib/kubelet/pods/ab94b2ff-c377-4343-8a93-3991521a1b19/volumes"
Sep 09 21:30:36.557911 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:30:36.557879    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:31:02.330302 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:31:02.330265    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:31:02.452729 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:31:02.452690    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:31:02.560048 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:31:02.560010    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:31:02.853914 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:31:02.853872    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:31:03.710670 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:31:03.710624    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:31:04.287943 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:31:04.287893    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:31:07.621346 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:31:07.621307    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:31:11.427936 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:31:11.427900    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:31:11.547921 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:31:11.547886    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:31:12.808667 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:31:12.808625    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:31:18.543356 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:31:18.543307    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:31:21.297848 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:31:21.297811    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:31:21.601914 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:31:21.601882    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5fb8d4448-tdq8p"]
Sep 09 21:31:21.602252 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:31:21.602239    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab94b2ff-c377-4343-8a93-3991521a1b19" containerName="main"
Sep 09 21:31:21.602298 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:31:21.602255    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab94b2ff-c377-4343-8a93-3991521a1b19" containerName="main"
Sep 09 21:31:21.602298 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:31:21.602265    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a12320a4-fe4c-457e-9733-f9a59b4cfc32" containerName="storage-initializer"
Sep 09 21:31:21.602298 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:31:21.602272    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="a12320a4-fe4c-457e-9733-f9a59b4cfc32" containerName="storage-initializer"
Sep 09 21:31:21.602298 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:31:21.602289    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab94b2ff-c377-4343-8a93-3991521a1b19" containerName="storage-initializer"
Sep 09 21:31:21.602298 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:31:21.602295    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab94b2ff-c377-4343-8a93-3991521a1b19" containerName="storage-initializer"
Sep 09 21:31:21.602484 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:31:21.602309    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a12320a4-fe4c-457e-9733-f9a59b4cfc32" containerName="main"
Sep 09 21:31:21.602484 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:31:21.602314    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="a12320a4-fe4c-457e-9733-f9a59b4cfc32" containerName="main"
Sep 09 21:31:21.602484 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:31:21.602409    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="a12320a4-fe4c-457e-9733-f9a59b4cfc32" containerName="main"
Sep 09 21:31:21.602484 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:31:21.602417    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab94b2ff-c377-4343-8a93-3991521a1b19" containerName="main"
Sep 09 21:31:21.604969 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:31:21.604953    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5fb8d4448-tdq8p"
Sep 09 21:31:21.608844 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:31:21.608819    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"openshift-service-ca.crt\""
Sep 09 21:31:21.608844 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:31:21.608839    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"default-dockercfg-v9wtf\""
Sep 09 21:31:21.609048 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:31:21.608824    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"seaweedfs-s3-creds\""
Sep 09 21:31:21.609048 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:31:21.608875    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"odh-kserve-custom-ca-bundle\""
Sep 09 21:31:21.609048 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:31:21.608962    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"kube-root-ca.crt\""
Sep 09 21:31:21.609048 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:31:21.608986    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"tokenizer-template-args-test-kserve-self-signed-certs\""
Sep 09 21:31:21.619721 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:31:21.619698    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5fb8d4448-tdq8p"]
Sep 09 21:31:21.633361 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:31:21.633332    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:31:21.763061 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:31:21.763018    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/230233f1-8816-47af-b55b-8ad56d29167e-home\") pod \"tokenizer-template-args-test-kserve-5fb8d4448-tdq8p\" (UID: \"230233f1-8816-47af-b55b-8ad56d29167e\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5fb8d4448-tdq8p"
Sep 09 21:31:21.763061 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:31:21.763072    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/230233f1-8816-47af-b55b-8ad56d29167e-model-cache\") pod \"tokenizer-template-args-test-kserve-5fb8d4448-tdq8p\" (UID: \"230233f1-8816-47af-b55b-8ad56d29167e\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5fb8d4448-tdq8p"
Sep 09 21:31:21.763265 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:31:21.763143    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/230233f1-8816-47af-b55b-8ad56d29167e-dshm\") pod \"tokenizer-template-args-test-kserve-5fb8d4448-tdq8p\" (UID: \"230233f1-8816-47af-b55b-8ad56d29167e\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5fb8d4448-tdq8p"
Sep 09 21:31:21.763265 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:31:21.763174    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/230233f1-8816-47af-b55b-8ad56d29167e-tls-certs\") pod \"tokenizer-template-args-test-kserve-5fb8d4448-tdq8p\" (UID: \"230233f1-8816-47af-b55b-8ad56d29167e\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5fb8d4448-tdq8p"
Sep 09 21:31:21.763265 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:31:21.763194    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/230233f1-8816-47af-b55b-8ad56d29167e-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-5fb8d4448-tdq8p\" (UID: \"230233f1-8816-47af-b55b-8ad56d29167e\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5fb8d4448-tdq8p"
Sep 09 21:31:21.763265 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:31:21.763245    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/230233f1-8816-47af-b55b-8ad56d29167e-tmp-dir\") pod \"tokenizer-template-args-test-kserve-5fb8d4448-tdq8p\" (UID: \"230233f1-8816-47af-b55b-8ad56d29167e\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5fb8d4448-tdq8p"
Sep 09 21:31:21.763456 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:31:21.763279    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmszk\" (UniqueName: \"kubernetes.io/projected/230233f1-8816-47af-b55b-8ad56d29167e-kube-api-access-hmszk\") pod \"tokenizer-template-args-test-kserve-5fb8d4448-tdq8p\" (UID: \"230233f1-8816-47af-b55b-8ad56d29167e\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5fb8d4448-tdq8p"
Sep 09 21:31:21.763456 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:31:21.763316    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/230233f1-8816-47af-b55b-8ad56d29167e-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-5fb8d4448-tdq8p\" (UID: \"230233f1-8816-47af-b55b-8ad56d29167e\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5fb8d4448-tdq8p"
Sep 09 21:31:21.865072 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:31:21.864984    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-765d4d7894-w58r6"]
Sep 09 21:31:21.868890 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:31:21.868846    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/230233f1-8816-47af-b55b-8ad56d29167e-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-5fb8d4448-tdq8p\" (UID: \"230233f1-8816-47af-b55b-8ad56d29167e\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5fb8d4448-tdq8p"
Sep 09 21:31:21.869102 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:31:21.868903    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/230233f1-8816-47af-b55b-8ad56d29167e-home\") pod \"tokenizer-template-args-test-kserve-5fb8d4448-tdq8p\" (UID: \"230233f1-8816-47af-b55b-8ad56d29167e\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5fb8d4448-tdq8p"
Sep 09 21:31:21.869196 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:31:21.869176    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/230233f1-8816-47af-b55b-8ad56d29167e-model-cache\") pod \"tokenizer-template-args-test-kserve-5fb8d4448-tdq8p\" (UID: \"230233f1-8816-47af-b55b-8ad56d29167e\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5fb8d4448-tdq8p"
Sep 09 21:31:21.869263 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:31:21.869241    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/230233f1-8816-47af-b55b-8ad56d29167e-dshm\") pod \"tokenizer-template-args-test-kserve-5fb8d4448-tdq8p\" (UID: \"230233f1-8816-47af-b55b-8ad56d29167e\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5fb8d4448-tdq8p"
Sep 09 21:31:21.869315 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:31:21.869275    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/230233f1-8816-47af-b55b-8ad56d29167e-tls-certs\") pod \"tokenizer-template-args-test-kserve-5fb8d4448-tdq8p\" (UID: \"230233f1-8816-47af-b55b-8ad56d29167e\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5fb8d4448-tdq8p"
Sep 09 21:31:21.869315 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:31:21.869301    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/230233f1-8816-47af-b55b-8ad56d29167e-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-5fb8d4448-tdq8p\" (UID: \"230233f1-8816-47af-b55b-8ad56d29167e\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5fb8d4448-tdq8p"
Sep 09 21:31:21.869458 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:31:21.869350    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/230233f1-8816-47af-b55b-8ad56d29167e-tmp-dir\") pod \"tokenizer-template-args-test-kserve-5fb8d4448-tdq8p\" (UID: \"230233f1-8816-47af-b55b-8ad56d29167e\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5fb8d4448-tdq8p"
Sep 09 21:31:21.869458 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:31:21.869414    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmszk\" (UniqueName: \"kubernetes.io/projected/230233f1-8816-47af-b55b-8ad56d29167e-kube-api-access-hmszk\") pod \"tokenizer-template-args-test-kserve-5fb8d4448-tdq8p\" (UID: \"230233f1-8816-47af-b55b-8ad56d29167e\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5fb8d4448-tdq8p"
Sep 09 21:31:21.869458 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:31:21.869432    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/230233f1-8816-47af-b55b-8ad56d29167e-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-5fb8d4448-tdq8p\" (UID: \"230233f1-8816-47af-b55b-8ad56d29167e\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5fb8d4448-tdq8p"
Sep 09 21:31:21.869458 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:31:21.869441    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/230233f1-8816-47af-b55b-8ad56d29167e-home\") pod \"tokenizer-template-args-test-kserve-5fb8d4448-tdq8p\" (UID: \"230233f1-8816-47af-b55b-8ad56d29167e\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5fb8d4448-tdq8p"
Sep 09 21:31:21.869740 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:31:21.869713    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/230233f1-8816-47af-b55b-8ad56d29167e-model-cache\") pod \"tokenizer-template-args-test-kserve-5fb8d4448-tdq8p\" (UID: \"230233f1-8816-47af-b55b-8ad56d29167e\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5fb8d4448-tdq8p"
Sep 09 21:31:21.870054 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:31:21.870032    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/230233f1-8816-47af-b55b-8ad56d29167e-tmp-dir\") pod \"tokenizer-template-args-test-kserve-5fb8d4448-tdq8p\" (UID: \"230233f1-8816-47af-b55b-8ad56d29167e\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5fb8d4448-tdq8p"
Sep 09 21:31:21.870328 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:31:21.870306    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/230233f1-8816-47af-b55b-8ad56d29167e-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-5fb8d4448-tdq8p\" (UID: \"230233f1-8816-47af-b55b-8ad56d29167e\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5fb8d4448-tdq8p"
Sep 09 21:31:21.871502 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:31:21.871483    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-765d4d7894-w58r6"
Sep 09 21:31:21.871836 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:31:21.871797    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/230233f1-8816-47af-b55b-8ad56d29167e-dshm\") pod \"tokenizer-template-args-test-kserve-5fb8d4448-tdq8p\" (UID: \"230233f1-8816-47af-b55b-8ad56d29167e\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5fb8d4448-tdq8p"
Sep 09 21:31:21.872787 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:31:21.872762    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/230233f1-8816-47af-b55b-8ad56d29167e-tls-certs\") pod \"tokenizer-template-args-test-kserve-5fb8d4448-tdq8p\" (UID: \"230233f1-8816-47af-b55b-8ad56d29167e\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5fb8d4448-tdq8p"
Sep 09 21:31:21.880661 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:31:21.880643    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-765d4d7894-w58r6"]
Sep 09 21:31:21.885471 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:31:21.885450    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmszk\" (UniqueName: \"kubernetes.io/projected/230233f1-8816-47af-b55b-8ad56d29167e-kube-api-access-hmszk\") pod \"tokenizer-template-args-test-kserve-5fb8d4448-tdq8p\" (UID: \"230233f1-8816-47af-b55b-8ad56d29167e\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5fb8d4448-tdq8p"
Sep 09 21:31:21.914851 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:31:21.914829    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5fb8d4448-tdq8p"
Sep 09 21:31:21.919821 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:31:21.919793    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:31:21.969934 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:31:21.969901    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ffd291b3-aa2f-4e69-b584-f1a614d9c2e1-kserve-provision-location\") pod \"tokenizer-template-args-test-tokenizer-765d4d7894-w58r6\" (UID: \"ffd291b3-aa2f-4e69-b584-f1a614d9c2e1\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-765d4d7894-w58r6"
Sep 09 21:31:21.970040 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:31:21.969996    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ffd291b3-aa2f-4e69-b584-f1a614d9c2e1-cabundle-cert\") pod \"tokenizer-template-args-test-tokenizer-765d4d7894-w58r6\" (UID: \"ffd291b3-aa2f-4e69-b584-f1a614d9c2e1\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-765d4d7894-w58r6"
Sep 09 21:31:21.970087 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:31:21.970042    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffd291b3-aa2f-4e69-b584-f1a614d9c2e1-tls-certs\") pod \"tokenizer-template-args-test-tokenizer-765d4d7894-w58r6\" (UID: \"ffd291b3-aa2f-4e69-b584-f1a614d9c2e1\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-765d4d7894-w58r6"
Sep 09 21:31:21.970127 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:31:21.970100    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffd291b3-aa2f-4e69-b584-f1a614d9c2e1-tmp-dir\") pod \"tokenizer-template-args-test-tokenizer-765d4d7894-w58r6\" (UID: \"ffd291b3-aa2f-4e69-b584-f1a614d9c2e1\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-765d4d7894-w58r6"
Sep 09 21:31:22.011631 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:31:22.011591    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:31:22.070786 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:31:22.070756    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ffd291b3-aa2f-4e69-b584-f1a614d9c2e1-kserve-provision-location\") pod \"tokenizer-template-args-test-tokenizer-765d4d7894-w58r6\" (UID: \"ffd291b3-aa2f-4e69-b584-f1a614d9c2e1\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-765d4d7894-w58r6"
Sep 09 21:31:22.070915 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:31:22.070791    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ffd291b3-aa2f-4e69-b584-f1a614d9c2e1-cabundle-cert\") pod \"tokenizer-template-args-test-tokenizer-765d4d7894-w58r6\" (UID: \"ffd291b3-aa2f-4e69-b584-f1a614d9c2e1\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-765d4d7894-w58r6"
Sep 09 21:31:22.070915 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:31:22.070819    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffd291b3-aa2f-4e69-b584-f1a614d9c2e1-tls-certs\") pod \"tokenizer-template-args-test-tokenizer-765d4d7894-w58r6\" (UID: \"ffd291b3-aa2f-4e69-b584-f1a614d9c2e1\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-765d4d7894-w58r6"
Sep 09 21:31:22.070915 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:31:22.070865    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffd291b3-aa2f-4e69-b584-f1a614d9c2e1-tmp-dir\") pod \"tokenizer-template-args-test-tokenizer-765d4d7894-w58r6\" (UID: \"ffd291b3-aa2f-4e69-b584-f1a614d9c2e1\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-765d4d7894-w58r6"
Sep 09 21:31:22.071220 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:31:22.071199    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ffd291b3-aa2f-4e69-b584-f1a614d9c2e1-kserve-provision-location\") pod \"tokenizer-template-args-test-tokenizer-765d4d7894-w58r6\" (UID: \"ffd291b3-aa2f-4e69-b584-f1a614d9c2e1\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-765d4d7894-w58r6"
Sep 09 21:31:22.071264 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:31:22.071243    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffd291b3-aa2f-4e69-b584-f1a614d9c2e1-tmp-dir\") pod \"tokenizer-template-args-test-tokenizer-765d4d7894-w58r6\" (UID: \"ffd291b3-aa2f-4e69-b584-f1a614d9c2e1\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-765d4d7894-w58r6"
Sep 09 21:31:22.071489 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:31:22.071470    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ffd291b3-aa2f-4e69-b584-f1a614d9c2e1-cabundle-cert\") pod \"tokenizer-template-args-test-tokenizer-765d4d7894-w58r6\" (UID: \"ffd291b3-aa2f-4e69-b584-f1a614d9c2e1\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-765d4d7894-w58r6"
Sep 09 21:31:22.073469 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:31:22.073446    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffd291b3-aa2f-4e69-b584-f1a614d9c2e1-tls-certs\") pod \"tokenizer-template-args-test-tokenizer-765d4d7894-w58r6\" (UID: \"ffd291b3-aa2f-4e69-b584-f1a614d9c2e1\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-765d4d7894-w58r6"
Sep 09 21:31:22.193515 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:31:22.193431    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5fb8d4448-tdq8p"]
Sep 09 21:31:22.196084 ip-10-0-134-72 kubenswrapper[2576]: W0909 21:31:22.196056    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod230233f1_8816_47af_b55b_8ad56d29167e.slice/crio-3b28231dd4078edeb9e8e799c61083c3c206d49e71cd812b1d3754b893af2edb WatchSource:0}: Error finding container 3b28231dd4078edeb9e8e799c61083c3c206d49e71cd812b1d3754b893af2edb: Status 404 returned error can't find the container with id 3b28231dd4078edeb9e8e799c61083c3c206d49e71cd812b1d3754b893af2edb
Sep 09 21:31:22.207173 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:31:22.207147    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-765d4d7894-w58r6"
Sep 09 21:31:22.494281 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:31:22.494225    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-765d4d7894-w58r6"]
Sep 09 21:31:22.495042 ip-10-0-134-72 kubenswrapper[2576]: W0909 21:31:22.495016    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffd291b3_aa2f_4e69_b584_f1a614d9c2e1.slice/crio-cee0a03e8da69eff58a3080f8e31e931416651ce917d12bc92f8de0703782428 WatchSource:0}: Error finding container cee0a03e8da69eff58a3080f8e31e931416651ce917d12bc92f8de0703782428: Status 404 returned error can't find the container with id cee0a03e8da69eff58a3080f8e31e931416651ce917d12bc92f8de0703782428
Sep 09 21:31:23.198298 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:31:23.198257    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-765d4d7894-w58r6" event={"ID":"ffd291b3-aa2f-4e69-b584-f1a614d9c2e1","Type":"ContainerStarted","Data":"bd6661b9699877bcbe2adb7c3ea9ef54d0ade130b6f518cdef8779e707f0c80a"}
Sep 09 21:31:23.198534 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:31:23.198331    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-765d4d7894-w58r6" event={"ID":"ffd291b3-aa2f-4e69-b584-f1a614d9c2e1","Type":"ContainerStarted","Data":"cee0a03e8da69eff58a3080f8e31e931416651ce917d12bc92f8de0703782428"}
Sep 09 21:31:23.199908 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:31:23.199886    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5fb8d4448-tdq8p" event={"ID":"230233f1-8816-47af-b55b-8ad56d29167e","Type":"ContainerStarted","Data":"67ace5f94559f63da620c92823d953b4bc92ea3f47594e929d6fe936b21efb50"}
Sep 09 21:31:23.200052 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:31:23.199923    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5fb8d4448-tdq8p" event={"ID":"230233f1-8816-47af-b55b-8ad56d29167e","Type":"ContainerStarted","Data":"3b28231dd4078edeb9e8e799c61083c3c206d49e71cd812b1d3754b893af2edb"}
Sep 09 21:31:25.213158 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:31:25.213122    2576 generic.go:356] "Generic (PLEG): container finished" podID="ffd291b3-aa2f-4e69-b584-f1a614d9c2e1" containerID="bd6661b9699877bcbe2adb7c3ea9ef54d0ade130b6f518cdef8779e707f0c80a" exitCode=0
Sep 09 21:31:25.213681 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:31:25.213200    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-765d4d7894-w58r6" event={"ID":"ffd291b3-aa2f-4e69-b584-f1a614d9c2e1","Type":"ContainerDied","Data":"bd6661b9699877bcbe2adb7c3ea9ef54d0ade130b6f518cdef8779e707f0c80a"}
Sep 09 21:31:25.214906 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:31:25.214883    2576 generic.go:356] "Generic (PLEG): container finished" podID="230233f1-8816-47af-b55b-8ad56d29167e" containerID="67ace5f94559f63da620c92823d953b4bc92ea3f47594e929d6fe936b21efb50" exitCode=0
Sep 09 21:31:25.215001 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:31:25.214956    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5fb8d4448-tdq8p" event={"ID":"230233f1-8816-47af-b55b-8ad56d29167e","Type":"ContainerDied","Data":"67ace5f94559f63da620c92823d953b4bc92ea3f47594e929d6fe936b21efb50"}
Sep 09 21:31:25.638769 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:31:25.634625    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:31:25.718639 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:31:25.718607    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:31:26.222945 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:31:26.222906    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-765d4d7894-w58r6" event={"ID":"ffd291b3-aa2f-4e69-b584-f1a614d9c2e1","Type":"ContainerStarted","Data":"4a99f1ef4576682642875e996b10f925f08a7871a1356f23ae9ce3c422e57ddf"}
Sep 09 21:31:26.223443 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:31:26.223266    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-765d4d7894-w58r6"
Sep 09 21:31:26.225119 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:31:26.225096    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5fb8d4448-tdq8p" event={"ID":"230233f1-8816-47af-b55b-8ad56d29167e","Type":"ContainerStarted","Data":"e19367aa484a1495450978f16d92c2878e0c531cda2ff64d568b7d30d1a6c883"}
Sep 09 21:31:26.246598 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:31:26.246534    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-765d4d7894-w58r6" podStartSLOduration=2.769248608 podStartE2EDuration="5.246514712s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.477266104s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:31:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:31:26.24438907 +0000 UTC m=+2329.358074932" watchObservedRunningTime="2026-09-09 21:31:26.246514712 +0000 UTC m=+2329.360200572"
Sep 09 21:31:26.268445 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:31:26.268400    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5fb8d4448-tdq8p" podStartSLOduration=2.724853299 podStartE2EDuration="5.268385448s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.543532149s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:31:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:31:26.26699361 +0000 UTC m=+2329.380679496" watchObservedRunningTime="2026-09-09 21:31:26.268385448 +0000 UTC m=+2329.382071299"
Sep 09 21:31:26.915762 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:31:26.915729    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5fb8d4448-tdq8p"
Sep 09 21:31:31.915561 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:31:31.915472    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5fb8d4448-tdq8p"
Sep 09 21:31:31.934593 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:31:31.934559    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5fb8d4448-tdq8p"
Sep 09 21:31:32.269879 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:31:32.269789    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5fb8d4448-tdq8p"
Sep 09 21:31:36.037280 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:31:36.037236    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:31:41.782763 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:31:41.782717    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:31:45.377083 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:31:45.377045    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:31:51.872406 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:31:51.872347    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:31:51.972830 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:31:51.972780    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:31:57.241439 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:31:57.241405    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-765d4d7894-w58r6"
Sep 09 21:32:00.057968 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:32:00.057934    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-0eb24714_tokenizer-template-args-test-tokenizer-765d4d7894-w58r6_ffd291b3-aa2f-4e69-b584-f1a614d9c2e1/main/0.log"
Sep 09 21:32:02.324861 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:32:02.324816    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:32:02.392285 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:32:02.392246    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5fb8d4448-tdq8p"]
Sep 09 21:32:02.394306 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:32:02.394280    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-765d4d7894-w58r6"]
Sep 09 21:32:02.394489 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:32:02.394469    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-765d4d7894-w58r6" podUID="ffd291b3-aa2f-4e69-b584-f1a614d9c2e1" containerName="main" containerID="cri-o://4a99f1ef4576682642875e996b10f925f08a7871a1356f23ae9ce3c422e57ddf" gracePeriod=30
Sep 09 21:32:02.401607 ip-10-0-134-72 kubenswrapper[2576]: E0909 21:32:02.401571    2576 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 09 21:32:02.401607 ip-10-0-134-72 kubenswrapper[2576]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 21:32:02.401607 ip-10-0-134-72 kubenswrapper[2576]:         , stdout: , stderr: , exit code -1
Sep 09 21:32:02.401607 ip-10-0-134-72 kubenswrapper[2576]:  > containerID="e19367aa484a1495450978f16d92c2878e0c531cda2ff64d568b7d30d1a6c883" cmd=["/bin/sleep","15"]
Sep 09 21:32:02.401840 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:32:02.401632    2576 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5fb8d4448-tdq8p" message="" err=<
Sep 09 21:32:02.401840 ip-10-0-134-72 kubenswrapper[2576]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 21:32:02.401840 ip-10-0-134-72 kubenswrapper[2576]:         , stdout: , stderr: , exit code -1
Sep 09 21:32:02.401840 ip-10-0-134-72 kubenswrapper[2576]:  >
Sep 09 21:32:02.401840 ip-10-0-134-72 kubenswrapper[2576]: E0909 21:32:02.401648    2576 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 09 21:32:02.401840 ip-10-0-134-72 kubenswrapper[2576]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 21:32:02.401840 ip-10-0-134-72 kubenswrapper[2576]:         , stdout: , stderr: , exit code -1
Sep 09 21:32:02.401840 ip-10-0-134-72 kubenswrapper[2576]:  > pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5fb8d4448-tdq8p" podUID="230233f1-8816-47af-b55b-8ad56d29167e" containerName="main" containerID="cri-o://e19367aa484a1495450978f16d92c2878e0c531cda2ff64d568b7d30d1a6c883"
Sep 09 21:32:02.401840 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:32:02.401689    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5fb8d4448-tdq8p" podUID="230233f1-8816-47af-b55b-8ad56d29167e" containerName="main" containerID="cri-o://e19367aa484a1495450978f16d92c2878e0c531cda2ff64d568b7d30d1a6c883" gracePeriod=60
Sep 09 21:32:02.494479 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:32:02.494442    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:32:02.653606 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:32:02.653566    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:32:02.764318 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:32:02.764291    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5fb8d4448-tdq8p"
Sep 09 21:32:02.836542 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:32:02.836506    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/230233f1-8816-47af-b55b-8ad56d29167e-home\") pod \"230233f1-8816-47af-b55b-8ad56d29167e\" (UID: \"230233f1-8816-47af-b55b-8ad56d29167e\") "
Sep 09 21:32:02.836741 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:32:02.836583    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/230233f1-8816-47af-b55b-8ad56d29167e-model-cache\") pod \"230233f1-8816-47af-b55b-8ad56d29167e\" (UID: \"230233f1-8816-47af-b55b-8ad56d29167e\") "
Sep 09 21:32:02.836741 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:32:02.836617    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/230233f1-8816-47af-b55b-8ad56d29167e-kserve-provision-location\") pod \"230233f1-8816-47af-b55b-8ad56d29167e\" (UID: \"230233f1-8816-47af-b55b-8ad56d29167e\") "
Sep 09 21:32:02.836741 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:32:02.836658    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/230233f1-8816-47af-b55b-8ad56d29167e-dshm\") pod \"230233f1-8816-47af-b55b-8ad56d29167e\" (UID: \"230233f1-8816-47af-b55b-8ad56d29167e\") "
Sep 09 21:32:02.836741 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:32:02.836687    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/230233f1-8816-47af-b55b-8ad56d29167e-tls-certs\") pod \"230233f1-8816-47af-b55b-8ad56d29167e\" (UID: \"230233f1-8816-47af-b55b-8ad56d29167e\") "
Sep 09 21:32:02.836741 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:32:02.836716    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/230233f1-8816-47af-b55b-8ad56d29167e-cabundle-cert\") pod \"230233f1-8816-47af-b55b-8ad56d29167e\" (UID: \"230233f1-8816-47af-b55b-8ad56d29167e\") "
Sep 09 21:32:02.837003 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:32:02.836756    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hmszk\" (UniqueName: \"kubernetes.io/projected/230233f1-8816-47af-b55b-8ad56d29167e-kube-api-access-hmszk\") pod \"230233f1-8816-47af-b55b-8ad56d29167e\" (UID: \"230233f1-8816-47af-b55b-8ad56d29167e\") "
Sep 09 21:32:02.837003 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:32:02.836770    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/230233f1-8816-47af-b55b-8ad56d29167e-home" (OuterVolumeSpecName: "home") pod "230233f1-8816-47af-b55b-8ad56d29167e" (UID: "230233f1-8816-47af-b55b-8ad56d29167e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:32:02.837003 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:32:02.836822    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/230233f1-8816-47af-b55b-8ad56d29167e-tmp-dir\") pod \"230233f1-8816-47af-b55b-8ad56d29167e\" (UID: \"230233f1-8816-47af-b55b-8ad56d29167e\") "
Sep 09 21:32:02.837003 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:32:02.836851    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/230233f1-8816-47af-b55b-8ad56d29167e-model-cache" (OuterVolumeSpecName: "model-cache") pod "230233f1-8816-47af-b55b-8ad56d29167e" (UID: "230233f1-8816-47af-b55b-8ad56d29167e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:32:02.837213 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:32:02.837070    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/230233f1-8816-47af-b55b-8ad56d29167e-home\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:32:02.837213 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:32:02.837090    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/230233f1-8816-47af-b55b-8ad56d29167e-model-cache\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:32:02.837331 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:32:02.837242    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/230233f1-8816-47af-b55b-8ad56d29167e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "230233f1-8816-47af-b55b-8ad56d29167e" (UID: "230233f1-8816-47af-b55b-8ad56d29167e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:32:02.837331 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:32:02.837273    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/230233f1-8816-47af-b55b-8ad56d29167e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "230233f1-8816-47af-b55b-8ad56d29167e" (UID: "230233f1-8816-47af-b55b-8ad56d29167e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:32:02.841710 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:32:02.841678    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/230233f1-8816-47af-b55b-8ad56d29167e-dshm" (OuterVolumeSpecName: "dshm") pod "230233f1-8816-47af-b55b-8ad56d29167e" (UID: "230233f1-8816-47af-b55b-8ad56d29167e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:32:02.843523 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:32:02.843497    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/230233f1-8816-47af-b55b-8ad56d29167e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "230233f1-8816-47af-b55b-8ad56d29167e" (UID: "230233f1-8816-47af-b55b-8ad56d29167e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:32:02.843523 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:32:02.843504    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/230233f1-8816-47af-b55b-8ad56d29167e-kube-api-access-hmszk" (OuterVolumeSpecName: "kube-api-access-hmszk") pod "230233f1-8816-47af-b55b-8ad56d29167e" (UID: "230233f1-8816-47af-b55b-8ad56d29167e"). InnerVolumeSpecName "kube-api-access-hmszk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:32:02.860398 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:32:02.860342    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/230233f1-8816-47af-b55b-8ad56d29167e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "230233f1-8816-47af-b55b-8ad56d29167e" (UID: "230233f1-8816-47af-b55b-8ad56d29167e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:32:02.937976 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:32:02.937932    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/230233f1-8816-47af-b55b-8ad56d29167e-dshm\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:32:02.938219 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:32:02.938202    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/230233f1-8816-47af-b55b-8ad56d29167e-tls-certs\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:32:02.938324 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:32:02.938303    2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/230233f1-8816-47af-b55b-8ad56d29167e-cabundle-cert\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:32:02.938324 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:32:02.938324    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmszk\" (UniqueName: \"kubernetes.io/projected/230233f1-8816-47af-b55b-8ad56d29167e-kube-api-access-hmszk\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:32:02.938553 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:32:02.938337    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/230233f1-8816-47af-b55b-8ad56d29167e-tmp-dir\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:32:02.938553 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:32:02.938351    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/230233f1-8816-47af-b55b-8ad56d29167e-kserve-provision-location\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:32:03.035239 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:32:03.035214    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-765d4d7894-w58r6"
Sep 09 21:32:03.038799 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:32:03.038779    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ffd291b3-aa2f-4e69-b584-f1a614d9c2e1-cabundle-cert\") pod \"ffd291b3-aa2f-4e69-b584-f1a614d9c2e1\" (UID: \"ffd291b3-aa2f-4e69-b584-f1a614d9c2e1\") "
Sep 09 21:32:03.038909 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:32:03.038810    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ffd291b3-aa2f-4e69-b584-f1a614d9c2e1-kserve-provision-location\") pod \"ffd291b3-aa2f-4e69-b584-f1a614d9c2e1\" (UID: \"ffd291b3-aa2f-4e69-b584-f1a614d9c2e1\") "
Sep 09 21:32:03.038909 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:32:03.038832    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffd291b3-aa2f-4e69-b584-f1a614d9c2e1-tmp-dir\") pod \"ffd291b3-aa2f-4e69-b584-f1a614d9c2e1\" (UID: \"ffd291b3-aa2f-4e69-b584-f1a614d9c2e1\") "
Sep 09 21:32:03.038909 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:32:03.038892    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffd291b3-aa2f-4e69-b584-f1a614d9c2e1-tls-certs\") pod \"ffd291b3-aa2f-4e69-b584-f1a614d9c2e1\" (UID: \"ffd291b3-aa2f-4e69-b584-f1a614d9c2e1\") "
Sep 09 21:32:03.039201 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:32:03.039171    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffd291b3-aa2f-4e69-b584-f1a614d9c2e1-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ffd291b3-aa2f-4e69-b584-f1a614d9c2e1" (UID: "ffd291b3-aa2f-4e69-b584-f1a614d9c2e1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:32:03.039308 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:32:03.039264    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffd291b3-aa2f-4e69-b584-f1a614d9c2e1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ffd291b3-aa2f-4e69-b584-f1a614d9c2e1" (UID: "ffd291b3-aa2f-4e69-b584-f1a614d9c2e1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:32:03.041004 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:32:03.040987    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffd291b3-aa2f-4e69-b584-f1a614d9c2e1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ffd291b3-aa2f-4e69-b584-f1a614d9c2e1" (UID: "ffd291b3-aa2f-4e69-b584-f1a614d9c2e1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:32:03.061117 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:32:03.061083    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffd291b3-aa2f-4e69-b584-f1a614d9c2e1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ffd291b3-aa2f-4e69-b584-f1a614d9c2e1" (UID: "ffd291b3-aa2f-4e69-b584-f1a614d9c2e1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:32:03.140173 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:32:03.140135    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffd291b3-aa2f-4e69-b584-f1a614d9c2e1-tls-certs\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:32:03.140173 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:32:03.140165    2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ffd291b3-aa2f-4e69-b584-f1a614d9c2e1-cabundle-cert\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:32:03.140173 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:32:03.140174    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ffd291b3-aa2f-4e69-b584-f1a614d9c2e1-kserve-provision-location\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:32:03.140173 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:32:03.140182    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffd291b3-aa2f-4e69-b584-f1a614d9c2e1-tmp-dir\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:32:03.423145 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:32:03.423112    2576 generic.go:356] "Generic (PLEG): container finished" podID="ffd291b3-aa2f-4e69-b584-f1a614d9c2e1" containerID="4a99f1ef4576682642875e996b10f925f08a7871a1356f23ae9ce3c422e57ddf" exitCode=0
Sep 09 21:32:03.423617 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:32:03.423180    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-765d4d7894-w58r6"
Sep 09 21:32:03.423617 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:32:03.423205    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-765d4d7894-w58r6" event={"ID":"ffd291b3-aa2f-4e69-b584-f1a614d9c2e1","Type":"ContainerDied","Data":"4a99f1ef4576682642875e996b10f925f08a7871a1356f23ae9ce3c422e57ddf"}
Sep 09 21:32:03.423617 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:32:03.423241    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-765d4d7894-w58r6" event={"ID":"ffd291b3-aa2f-4e69-b584-f1a614d9c2e1","Type":"ContainerDied","Data":"cee0a03e8da69eff58a3080f8e31e931416651ce917d12bc92f8de0703782428"}
Sep 09 21:32:03.423617 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:32:03.423270    2576 scope.go:117] "RemoveContainer" containerID="4a99f1ef4576682642875e996b10f925f08a7871a1356f23ae9ce3c422e57ddf"
Sep 09 21:32:03.425796 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:32:03.425768    2576 generic.go:356] "Generic (PLEG): container finished" podID="230233f1-8816-47af-b55b-8ad56d29167e" containerID="e19367aa484a1495450978f16d92c2878e0c531cda2ff64d568b7d30d1a6c883" exitCode=0
Sep 09 21:32:03.425896 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:32:03.425820    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5fb8d4448-tdq8p"
Sep 09 21:32:03.425896 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:32:03.425824    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5fb8d4448-tdq8p" event={"ID":"230233f1-8816-47af-b55b-8ad56d29167e","Type":"ContainerDied","Data":"e19367aa484a1495450978f16d92c2878e0c531cda2ff64d568b7d30d1a6c883"}
Sep 09 21:32:03.425896 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:32:03.425882    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5fb8d4448-tdq8p" event={"ID":"230233f1-8816-47af-b55b-8ad56d29167e","Type":"ContainerDied","Data":"3b28231dd4078edeb9e8e799c61083c3c206d49e71cd812b1d3754b893af2edb"}
Sep 09 21:32:03.433823 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:32:03.433807    2576 scope.go:117] "RemoveContainer" containerID="bd6661b9699877bcbe2adb7c3ea9ef54d0ade130b6f518cdef8779e707f0c80a"
Sep 09 21:32:03.442204 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:32:03.442186    2576 scope.go:117] "RemoveContainer" containerID="4a99f1ef4576682642875e996b10f925f08a7871a1356f23ae9ce3c422e57ddf"
Sep 09 21:32:03.442497 ip-10-0-134-72 kubenswrapper[2576]: E0909 21:32:03.442476    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a99f1ef4576682642875e996b10f925f08a7871a1356f23ae9ce3c422e57ddf\": container with ID starting with 4a99f1ef4576682642875e996b10f925f08a7871a1356f23ae9ce3c422e57ddf not found: ID does not exist" containerID="4a99f1ef4576682642875e996b10f925f08a7871a1356f23ae9ce3c422e57ddf"
Sep 09 21:32:03.442594 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:32:03.442507    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a99f1ef4576682642875e996b10f925f08a7871a1356f23ae9ce3c422e57ddf"} err="failed to get container status \"4a99f1ef4576682642875e996b10f925f08a7871a1356f23ae9ce3c422e57ddf\": rpc error: code = NotFound desc = could not find container \"4a99f1ef4576682642875e996b10f925f08a7871a1356f23ae9ce3c422e57ddf\": container with ID starting with 4a99f1ef4576682642875e996b10f925f08a7871a1356f23ae9ce3c422e57ddf not found: ID does not exist"
Sep 09 21:32:03.442594 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:32:03.442531    2576 scope.go:117] "RemoveContainer" containerID="bd6661b9699877bcbe2adb7c3ea9ef54d0ade130b6f518cdef8779e707f0c80a"
Sep 09 21:32:03.442859 ip-10-0-134-72 kubenswrapper[2576]: E0909 21:32:03.442829    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd6661b9699877bcbe2adb7c3ea9ef54d0ade130b6f518cdef8779e707f0c80a\": container with ID starting with bd6661b9699877bcbe2adb7c3ea9ef54d0ade130b6f518cdef8779e707f0c80a not found: ID does not exist" containerID="bd6661b9699877bcbe2adb7c3ea9ef54d0ade130b6f518cdef8779e707f0c80a"
Sep 09 21:32:03.442943 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:32:03.442860    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd6661b9699877bcbe2adb7c3ea9ef54d0ade130b6f518cdef8779e707f0c80a"} err="failed to get container status \"bd6661b9699877bcbe2adb7c3ea9ef54d0ade130b6f518cdef8779e707f0c80a\": rpc error: code = NotFound desc = could not find container \"bd6661b9699877bcbe2adb7c3ea9ef54d0ade130b6f518cdef8779e707f0c80a\": container with ID starting with bd6661b9699877bcbe2adb7c3ea9ef54d0ade130b6f518cdef8779e707f0c80a not found: ID does not exist"
Sep 09 21:32:03.442943 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:32:03.442881    2576 scope.go:117] "RemoveContainer" containerID="e19367aa484a1495450978f16d92c2878e0c531cda2ff64d568b7d30d1a6c883"
Sep 09 21:32:03.451607 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:32:03.451583    2576 scope.go:117] "RemoveContainer" containerID="67ace5f94559f63da620c92823d953b4bc92ea3f47594e929d6fe936b21efb50"
Sep 09 21:32:03.459856 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:32:03.459829    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-765d4d7894-w58r6"]
Sep 09 21:32:03.460537 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:32:03.460521    2576 scope.go:117] "RemoveContainer" containerID="e19367aa484a1495450978f16d92c2878e0c531cda2ff64d568b7d30d1a6c883"
Sep 09 21:32:03.460789 ip-10-0-134-72 kubenswrapper[2576]: E0909 21:32:03.460770    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e19367aa484a1495450978f16d92c2878e0c531cda2ff64d568b7d30d1a6c883\": container with ID starting with e19367aa484a1495450978f16d92c2878e0c531cda2ff64d568b7d30d1a6c883 not found: ID does not exist" containerID="e19367aa484a1495450978f16d92c2878e0c531cda2ff64d568b7d30d1a6c883"
Sep 09 21:32:03.460845 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:32:03.460796    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e19367aa484a1495450978f16d92c2878e0c531cda2ff64d568b7d30d1a6c883"} err="failed to get container status \"e19367aa484a1495450978f16d92c2878e0c531cda2ff64d568b7d30d1a6c883\": rpc error: code = NotFound desc = could not find container \"e19367aa484a1495450978f16d92c2878e0c531cda2ff64d568b7d30d1a6c883\": container with ID starting with e19367aa484a1495450978f16d92c2878e0c531cda2ff64d568b7d30d1a6c883 not found: ID does not exist"
Sep 09 21:32:03.460845 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:32:03.460813    2576 scope.go:117] "RemoveContainer" containerID="67ace5f94559f63da620c92823d953b4bc92ea3f47594e929d6fe936b21efb50"
Sep 09 21:32:03.461054 ip-10-0-134-72 kubenswrapper[2576]: E0909 21:32:03.461037    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67ace5f94559f63da620c92823d953b4bc92ea3f47594e929d6fe936b21efb50\": container with ID starting with 67ace5f94559f63da620c92823d953b4bc92ea3f47594e929d6fe936b21efb50 not found: ID does not exist" containerID="67ace5f94559f63da620c92823d953b4bc92ea3f47594e929d6fe936b21efb50"
Sep 09 21:32:03.461096 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:32:03.461062    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67ace5f94559f63da620c92823d953b4bc92ea3f47594e929d6fe936b21efb50"} err="failed to get container status \"67ace5f94559f63da620c92823d953b4bc92ea3f47594e929d6fe936b21efb50\": rpc error: code = NotFound desc = could not find container \"67ace5f94559f63da620c92823d953b4bc92ea3f47594e929d6fe936b21efb50\": container with ID starting with 67ace5f94559f63da620c92823d953b4bc92ea3f47594e929d6fe936b21efb50 not found: ID does not exist"
Sep 09 21:32:03.466132 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:32:03.466111    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-765d4d7894-w58r6"]
Sep 09 21:32:03.493590 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:32:03.493553    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5fb8d4448-tdq8p"]
Sep 09 21:32:03.497753 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:32:03.497730    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5fb8d4448-tdq8p"]
Sep 09 21:32:05.409311 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:32:05.409279    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="230233f1-8816-47af-b55b-8ad56d29167e" path="/var/lib/kubelet/pods/230233f1-8816-47af-b55b-8ad56d29167e/volumes"
Sep 09 21:32:05.409723 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:32:05.409709    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ffd291b3-aa2f-4e69-b584-f1a614d9c2e1" path="/var/lib/kubelet/pods/ffd291b3-aa2f-4e69-b584-f1a614d9c2e1/volumes"
Sep 09 21:32:06.978628 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:32:06.978598    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:32:18.157832 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:32:18.157798    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:32:20.199903 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:32:20.199872    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:32:20.412297 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:32:20.412255    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:32:20.510637 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:32:20.510550    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:32:20.641612 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:32:20.641579    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:32:25.238764 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:32:25.238722    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:32:27.703410 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:32:27.703346    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:32:39.933253 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:32:39.933210    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:32:40.033084 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:32:40.032996    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:32:50.525593 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:32:50.525549    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:32:50.623030 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:32:50.622995    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:32:53.608101 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:32:53.608069    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:32:53.711555 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:32:53.711520    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:32:57.849793 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:32:57.849748    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:33:03.511883 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:33:03.511795    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:33:04.423851 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:33:04.423813    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:33:04.534402 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:33:04.534330    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:33:07.608719 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:33:07.608685    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-6b4x4"]
Sep 09 21:33:07.609074 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:33:07.609009    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="230233f1-8816-47af-b55b-8ad56d29167e" containerName="storage-initializer"
Sep 09 21:33:07.609074 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:33:07.609021    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="230233f1-8816-47af-b55b-8ad56d29167e" containerName="storage-initializer"
Sep 09 21:33:07.609074 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:33:07.609035    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ffd291b3-aa2f-4e69-b584-f1a614d9c2e1" containerName="storage-initializer"
Sep 09 21:33:07.609074 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:33:07.609041    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="ffd291b3-aa2f-4e69-b584-f1a614d9c2e1" containerName="storage-initializer"
Sep 09 21:33:07.609074 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:33:07.609050    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ffd291b3-aa2f-4e69-b584-f1a614d9c2e1" containerName="main"
Sep 09 21:33:07.609074 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:33:07.609055    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="ffd291b3-aa2f-4e69-b584-f1a614d9c2e1" containerName="main"
Sep 09 21:33:07.609074 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:33:07.609062    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="230233f1-8816-47af-b55b-8ad56d29167e" containerName="main"
Sep 09 21:33:07.609074 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:33:07.609067    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="230233f1-8816-47af-b55b-8ad56d29167e" containerName="main"
Sep 09 21:33:07.609351 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:33:07.609121    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="ffd291b3-aa2f-4e69-b584-f1a614d9c2e1" containerName="main"
Sep 09 21:33:07.609351 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:33:07.609128    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="230233f1-8816-47af-b55b-8ad56d29167e" containerName="main"
Sep 09 21:33:07.612132 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:33:07.612115    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-6b4x4"
Sep 09 21:33:07.616456 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:33:07.616421    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"llmisv5c7e67b6c51568d1d6d13829a9337f2a-kserve-self-signed-certs\""
Sep 09 21:33:07.616456 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:33:07.616435    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"kube-root-ca.crt\""
Sep 09 21:33:07.616655 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:33:07.616468    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"openshift-service-ca.crt\""
Sep 09 21:33:07.616655 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:33:07.616422    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"default-dockercfg-lxc9f\""
Sep 09 21:33:07.626897 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:33:07.626877    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-6b4x4"]
Sep 09 21:33:07.652007 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:33:07.651978    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:33:07.667025 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:33:07.666998    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0ddcf09-04f0-4bfe-9b75-e850e0ec4e51-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-6b4x4\" (UID: \"a0ddcf09-04f0-4bfe-9b75-e850e0ec4e51\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-6b4x4"
Sep 09 21:33:07.667130 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:33:07.667042    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0ddcf09-04f0-4bfe-9b75-e850e0ec4e51-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-6b4x4\" (UID: \"a0ddcf09-04f0-4bfe-9b75-e850e0ec4e51\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-6b4x4"
Sep 09 21:33:07.667130 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:33:07.667063    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jk98m\" (UniqueName: \"kubernetes.io/projected/a0ddcf09-04f0-4bfe-9b75-e850e0ec4e51-kube-api-access-jk98m\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-6b4x4\" (UID: \"a0ddcf09-04f0-4bfe-9b75-e850e0ec4e51\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-6b4x4"
Sep 09 21:33:07.667206 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:33:07.667190    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0ddcf09-04f0-4bfe-9b75-e850e0ec4e51-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-6b4x4\" (UID: \"a0ddcf09-04f0-4bfe-9b75-e850e0ec4e51\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-6b4x4"
Sep 09 21:33:07.667245 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:33:07.667220    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0ddcf09-04f0-4bfe-9b75-e850e0ec4e51-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-6b4x4\" (UID: \"a0ddcf09-04f0-4bfe-9b75-e850e0ec4e51\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-6b4x4"
Sep 09 21:33:07.667245 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:33:07.667240    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0ddcf09-04f0-4bfe-9b75-e850e0ec4e51-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-6b4x4\" (UID: \"a0ddcf09-04f0-4bfe-9b75-e850e0ec4e51\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-6b4x4"
Sep 09 21:33:07.768166 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:33:07.768138    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0ddcf09-04f0-4bfe-9b75-e850e0ec4e51-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-6b4x4\" (UID: \"a0ddcf09-04f0-4bfe-9b75-e850e0ec4e51\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-6b4x4"
Sep 09 21:33:07.768405 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:33:07.768180    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0ddcf09-04f0-4bfe-9b75-e850e0ec4e51-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-6b4x4\" (UID: \"a0ddcf09-04f0-4bfe-9b75-e850e0ec4e51\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-6b4x4"
Sep 09 21:33:07.768405 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:33:07.768202    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jk98m\" (UniqueName: \"kubernetes.io/projected/a0ddcf09-04f0-4bfe-9b75-e850e0ec4e51-kube-api-access-jk98m\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-6b4x4\" (UID: \"a0ddcf09-04f0-4bfe-9b75-e850e0ec4e51\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-6b4x4"
Sep 09 21:33:07.768405 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:33:07.768261    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0ddcf09-04f0-4bfe-9b75-e850e0ec4e51-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-6b4x4\" (UID: \"a0ddcf09-04f0-4bfe-9b75-e850e0ec4e51\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-6b4x4"
Sep 09 21:33:07.768405 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:33:07.768285    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0ddcf09-04f0-4bfe-9b75-e850e0ec4e51-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-6b4x4\" (UID: \"a0ddcf09-04f0-4bfe-9b75-e850e0ec4e51\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-6b4x4"
Sep 09 21:33:07.768405 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:33:07.768312    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0ddcf09-04f0-4bfe-9b75-e850e0ec4e51-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-6b4x4\" (UID: \"a0ddcf09-04f0-4bfe-9b75-e850e0ec4e51\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-6b4x4"
Sep 09 21:33:07.768702 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:33:07.768583    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0ddcf09-04f0-4bfe-9b75-e850e0ec4e51-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-6b4x4\" (UID: \"a0ddcf09-04f0-4bfe-9b75-e850e0ec4e51\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-6b4x4"
Sep 09 21:33:07.768763 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:33:07.768746    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0ddcf09-04f0-4bfe-9b75-e850e0ec4e51-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-6b4x4\" (UID: \"a0ddcf09-04f0-4bfe-9b75-e850e0ec4e51\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-6b4x4"
Sep 09 21:33:07.768763 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:33:07.768756    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0ddcf09-04f0-4bfe-9b75-e850e0ec4e51-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-6b4x4\" (UID: \"a0ddcf09-04f0-4bfe-9b75-e850e0ec4e51\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-6b4x4"
Sep 09 21:33:07.770778 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:33:07.770746    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0ddcf09-04f0-4bfe-9b75-e850e0ec4e51-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-6b4x4\" (UID: \"a0ddcf09-04f0-4bfe-9b75-e850e0ec4e51\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-6b4x4"
Sep 09 21:33:07.771011 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:33:07.770992    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0ddcf09-04f0-4bfe-9b75-e850e0ec4e51-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-6b4x4\" (UID: \"a0ddcf09-04f0-4bfe-9b75-e850e0ec4e51\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-6b4x4"
Sep 09 21:33:07.779139 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:33:07.779119    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jk98m\" (UniqueName: \"kubernetes.io/projected/a0ddcf09-04f0-4bfe-9b75-e850e0ec4e51-kube-api-access-jk98m\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-6b4x4\" (UID: \"a0ddcf09-04f0-4bfe-9b75-e850e0ec4e51\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-6b4x4"
Sep 09 21:33:07.912668 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:33:07.912575    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:33:07.922500 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:33:07.922474    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-6b4x4"
Sep 09 21:33:08.209420 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:33:08.209398    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-6b4x4"]
Sep 09 21:33:08.210232 ip-10-0-134-72 kubenswrapper[2576]: W0909 21:33:08.210209    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0ddcf09_04f0_4bfe_9b75_e850e0ec4e51.slice/crio-4097cd2356a058e754a77c71e2082097222fe5651e513b9164fecd2110bde434 WatchSource:0}: Error finding container 4097cd2356a058e754a77c71e2082097222fe5651e513b9164fecd2110bde434: Status 404 returned error can't find the container with id 4097cd2356a058e754a77c71e2082097222fe5651e513b9164fecd2110bde434
Sep 09 21:33:08.764539 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:33:08.763765    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-6b4x4" event={"ID":"a0ddcf09-04f0-4bfe-9b75-e850e0ec4e51","Type":"ContainerStarted","Data":"e7b97698102c685bf1d19273f27b022dcec2e78e3df15c9d518e93ece63f9062"}
Sep 09 21:33:08.764539 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:33:08.763831    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-6b4x4" event={"ID":"a0ddcf09-04f0-4bfe-9b75-e850e0ec4e51","Type":"ContainerStarted","Data":"4097cd2356a058e754a77c71e2082097222fe5651e513b9164fecd2110bde434"}
Sep 09 21:33:08.816438 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:33:08.816344    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-6b4x4" podStartSLOduration=1.816322847 podStartE2EDuration="1.816322847s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:33:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:33:08.812407273 +0000 UTC m=+2431.926093131" watchObservedRunningTime="2026-09-09 21:33:08.816322847 +0000 UTC m=+2431.930008707"
Sep 09 21:33:08.922899 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:33:08.922866    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-6b4x4"
Sep 09 21:33:10.837254 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:33:10.837219    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:33:10.930502 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:33:10.930466    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:33:17.923387 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:33:17.923333    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-6b4x4"
Sep 09 21:33:17.936056 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:33:17.936028    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-6b4x4"
Sep 09 21:33:18.825437 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:33:18.825409    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-6b4x4"
Sep 09 21:33:28.334750 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:33:28.334718    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:33:30.319249 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:33:30.319217    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:33:41.894404 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:33:41.894332    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-6b4x4"]
Sep 09 21:33:41.905208 ip-10-0-134-72 kubenswrapper[2576]: E0909 21:33:41.905173    2576 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 09 21:33:41.905208 ip-10-0-134-72 kubenswrapper[2576]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 21:33:41.905208 ip-10-0-134-72 kubenswrapper[2576]:         , stdout: , stderr: , exit code -1
Sep 09 21:33:41.905208 ip-10-0-134-72 kubenswrapper[2576]:  > containerID="e7b97698102c685bf1d19273f27b022dcec2e78e3df15c9d518e93ece63f9062" cmd=["/bin/sleep","15"]
Sep 09 21:33:41.905501 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:33:41.905226    2576 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-6b4x4" message="" err=<
Sep 09 21:33:41.905501 ip-10-0-134-72 kubenswrapper[2576]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 21:33:41.905501 ip-10-0-134-72 kubenswrapper[2576]:         , stdout: , stderr: , exit code -1
Sep 09 21:33:41.905501 ip-10-0-134-72 kubenswrapper[2576]:  >
Sep 09 21:33:41.905501 ip-10-0-134-72 kubenswrapper[2576]: E0909 21:33:41.905238    2576 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 09 21:33:41.905501 ip-10-0-134-72 kubenswrapper[2576]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 21:33:41.905501 ip-10-0-134-72 kubenswrapper[2576]:         , stdout: , stderr: , exit code -1
Sep 09 21:33:41.905501 ip-10-0-134-72 kubenswrapper[2576]:  > pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-6b4x4" podUID="a0ddcf09-04f0-4bfe-9b75-e850e0ec4e51" containerName="main" containerID="cri-o://e7b97698102c685bf1d19273f27b022dcec2e78e3df15c9d518e93ece63f9062"
Sep 09 21:33:41.905501 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:33:41.905288    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-6b4x4" podUID="a0ddcf09-04f0-4bfe-9b75-e850e0ec4e51" containerName="main" containerID="cri-o://e7b97698102c685bf1d19273f27b022dcec2e78e3df15c9d518e93ece63f9062" gracePeriod=60
Sep 09 21:33:41.932300 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:33:41.932266    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:33:42.040413 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:33:42.040387    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:33:42.255701 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:33:42.255674    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-6b4x4"
Sep 09 21:33:42.361272 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:33:42.361243    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0ddcf09-04f0-4bfe-9b75-e850e0ec4e51-tmp-dir\") pod \"a0ddcf09-04f0-4bfe-9b75-e850e0ec4e51\" (UID: \"a0ddcf09-04f0-4bfe-9b75-e850e0ec4e51\") "
Sep 09 21:33:42.361497 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:33:42.361279    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jk98m\" (UniqueName: \"kubernetes.io/projected/a0ddcf09-04f0-4bfe-9b75-e850e0ec4e51-kube-api-access-jk98m\") pod \"a0ddcf09-04f0-4bfe-9b75-e850e0ec4e51\" (UID: \"a0ddcf09-04f0-4bfe-9b75-e850e0ec4e51\") "
Sep 09 21:33:42.361497 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:33:42.361304    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0ddcf09-04f0-4bfe-9b75-e850e0ec4e51-home\") pod \"a0ddcf09-04f0-4bfe-9b75-e850e0ec4e51\" (UID: \"a0ddcf09-04f0-4bfe-9b75-e850e0ec4e51\") "
Sep 09 21:33:42.361497 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:33:42.361325    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0ddcf09-04f0-4bfe-9b75-e850e0ec4e51-model-cache\") pod \"a0ddcf09-04f0-4bfe-9b75-e850e0ec4e51\" (UID: \"a0ddcf09-04f0-4bfe-9b75-e850e0ec4e51\") "
Sep 09 21:33:42.361497 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:33:42.361436    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0ddcf09-04f0-4bfe-9b75-e850e0ec4e51-tls-certs\") pod \"a0ddcf09-04f0-4bfe-9b75-e850e0ec4e51\" (UID: \"a0ddcf09-04f0-4bfe-9b75-e850e0ec4e51\") "
Sep 09 21:33:42.361703 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:33:42.361504    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0ddcf09-04f0-4bfe-9b75-e850e0ec4e51-dshm\") pod \"a0ddcf09-04f0-4bfe-9b75-e850e0ec4e51\" (UID: \"a0ddcf09-04f0-4bfe-9b75-e850e0ec4e51\") "
Sep 09 21:33:42.361703 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:33:42.361528    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0ddcf09-04f0-4bfe-9b75-e850e0ec4e51-model-cache" (OuterVolumeSpecName: "model-cache") pod "a0ddcf09-04f0-4bfe-9b75-e850e0ec4e51" (UID: "a0ddcf09-04f0-4bfe-9b75-e850e0ec4e51"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:33:42.361703 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:33:42.361541    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0ddcf09-04f0-4bfe-9b75-e850e0ec4e51-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a0ddcf09-04f0-4bfe-9b75-e850e0ec4e51" (UID: "a0ddcf09-04f0-4bfe-9b75-e850e0ec4e51"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:33:42.361703 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:33:42.361555    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0ddcf09-04f0-4bfe-9b75-e850e0ec4e51-home" (OuterVolumeSpecName: "home") pod "a0ddcf09-04f0-4bfe-9b75-e850e0ec4e51" (UID: "a0ddcf09-04f0-4bfe-9b75-e850e0ec4e51"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:33:42.361839 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:33:42.361760    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0ddcf09-04f0-4bfe-9b75-e850e0ec4e51-tmp-dir\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:33:42.361839 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:33:42.361778    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0ddcf09-04f0-4bfe-9b75-e850e0ec4e51-home\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:33:42.361839 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:33:42.361787    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0ddcf09-04f0-4bfe-9b75-e850e0ec4e51-model-cache\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:33:42.363683 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:33:42.363657    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0ddcf09-04f0-4bfe-9b75-e850e0ec4e51-dshm" (OuterVolumeSpecName: "dshm") pod "a0ddcf09-04f0-4bfe-9b75-e850e0ec4e51" (UID: "a0ddcf09-04f0-4bfe-9b75-e850e0ec4e51"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:33:42.363807 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:33:42.363723    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0ddcf09-04f0-4bfe-9b75-e850e0ec4e51-kube-api-access-jk98m" (OuterVolumeSpecName: "kube-api-access-jk98m") pod "a0ddcf09-04f0-4bfe-9b75-e850e0ec4e51" (UID: "a0ddcf09-04f0-4bfe-9b75-e850e0ec4e51"). InnerVolumeSpecName "kube-api-access-jk98m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:33:42.363807 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:33:42.363735    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0ddcf09-04f0-4bfe-9b75-e850e0ec4e51-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a0ddcf09-04f0-4bfe-9b75-e850e0ec4e51" (UID: "a0ddcf09-04f0-4bfe-9b75-e850e0ec4e51"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:33:42.463090 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:33:42.463014    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jk98m\" (UniqueName: \"kubernetes.io/projected/a0ddcf09-04f0-4bfe-9b75-e850e0ec4e51-kube-api-access-jk98m\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:33:42.463090 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:33:42.463041    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0ddcf09-04f0-4bfe-9b75-e850e0ec4e51-tls-certs\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:33:42.463090 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:33:42.463052    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0ddcf09-04f0-4bfe-9b75-e850e0ec4e51-dshm\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:33:42.939828 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:33:42.939798    2576 generic.go:356] "Generic (PLEG): container finished" podID="a0ddcf09-04f0-4bfe-9b75-e850e0ec4e51" containerID="e7b97698102c685bf1d19273f27b022dcec2e78e3df15c9d518e93ece63f9062" exitCode=0
Sep 09 21:33:42.940206 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:33:42.939841    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-6b4x4"
Sep 09 21:33:42.940206 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:33:42.939886    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-6b4x4" event={"ID":"a0ddcf09-04f0-4bfe-9b75-e850e0ec4e51","Type":"ContainerDied","Data":"e7b97698102c685bf1d19273f27b022dcec2e78e3df15c9d518e93ece63f9062"}
Sep 09 21:33:42.940206 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:33:42.939939    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-6b4x4" event={"ID":"a0ddcf09-04f0-4bfe-9b75-e850e0ec4e51","Type":"ContainerDied","Data":"4097cd2356a058e754a77c71e2082097222fe5651e513b9164fecd2110bde434"}
Sep 09 21:33:42.940206 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:33:42.939962    2576 scope.go:117] "RemoveContainer" containerID="e7b97698102c685bf1d19273f27b022dcec2e78e3df15c9d518e93ece63f9062"
Sep 09 21:33:42.949312 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:33:42.949295    2576 scope.go:117] "RemoveContainer" containerID="e7b97698102c685bf1d19273f27b022dcec2e78e3df15c9d518e93ece63f9062"
Sep 09 21:33:42.949625 ip-10-0-134-72 kubenswrapper[2576]: E0909 21:33:42.949607    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7b97698102c685bf1d19273f27b022dcec2e78e3df15c9d518e93ece63f9062\": container with ID starting with e7b97698102c685bf1d19273f27b022dcec2e78e3df15c9d518e93ece63f9062 not found: ID does not exist" containerID="e7b97698102c685bf1d19273f27b022dcec2e78e3df15c9d518e93ece63f9062"
Sep 09 21:33:42.949699 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:33:42.949636    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7b97698102c685bf1d19273f27b022dcec2e78e3df15c9d518e93ece63f9062"} err="failed to get container status \"e7b97698102c685bf1d19273f27b022dcec2e78e3df15c9d518e93ece63f9062\": rpc error: code = NotFound desc = could not find container \"e7b97698102c685bf1d19273f27b022dcec2e78e3df15c9d518e93ece63f9062\": container with ID starting with e7b97698102c685bf1d19273f27b022dcec2e78e3df15c9d518e93ece63f9062 not found: ID does not exist"
Sep 09 21:33:43.003696 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:33:43.003665    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-6b4x4"]
Sep 09 21:33:43.012974 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:33:43.012947    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-6b4x4"]
Sep 09 21:33:43.409185 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:33:43.409149    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a0ddcf09-04f0-4bfe-9b75-e850e0ec4e51" path="/var/lib/kubelet/pods/a0ddcf09-04f0-4bfe-9b75-e850e0ec4e51/volumes"
Sep 09 21:33:46.295687 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:33:46.295651    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:33:46.422014 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:33:46.421979    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:33:46.529830 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:33:46.529794    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:33:51.431823 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:33:51.431790    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:33:56.511735 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:33:56.511699    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:34:04.041666 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:34:04.041621    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:34:30.009456 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:34:30.009356    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:34:31.882966 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:34:31.882936    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:34:31.996802 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:34:31.996766    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:34:36.415837 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:34:36.415797    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:34:36.515732 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:34:36.515658    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:34:40.094564 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:34:40.094522    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:34:42.643554 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:34:42.643523    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-m68qh"]
Sep 09 21:34:42.643997 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:34:42.643889    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0ddcf09-04f0-4bfe-9b75-e850e0ec4e51" containerName="main"
Sep 09 21:34:42.643997 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:34:42.643901    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0ddcf09-04f0-4bfe-9b75-e850e0ec4e51" containerName="main"
Sep 09 21:34:42.643997 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:34:42.643959    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0ddcf09-04f0-4bfe-9b75-e850e0ec4e51" containerName="main"
Sep 09 21:34:42.648157 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:34:42.648131    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-m68qh"
Sep 09 21:34:42.662808 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:34:42.662786    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"openshift-service-ca.crt\""
Sep 09 21:34:42.662808 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:34:42.662797    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"kube-root-ca.crt\""
Sep 09 21:34:42.662972 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:34:42.662797    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs\""
Sep 09 21:34:42.662972 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:34:42.662797    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"default-dockercfg-2grph\""
Sep 09 21:34:42.683154 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:34:42.683134    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-m68qh"]
Sep 09 21:34:42.709980 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:34:42.709948    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:34:42.766275 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:34:42.766248    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2573d04f-f7a6-43fb-8e40-ebabc82a8a32-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-m68qh\" (UID: \"2573d04f-f7a6-43fb-8e40-ebabc82a8a32\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-m68qh"
Sep 09 21:34:42.766453 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:34:42.766291    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2573d04f-f7a6-43fb-8e40-ebabc82a8a32-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-m68qh\" (UID: \"2573d04f-f7a6-43fb-8e40-ebabc82a8a32\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-m68qh"
Sep 09 21:34:42.766453 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:34:42.766403    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2573d04f-f7a6-43fb-8e40-ebabc82a8a32-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-m68qh\" (UID: \"2573d04f-f7a6-43fb-8e40-ebabc82a8a32\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-m68qh"
Sep 09 21:34:42.766453 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:34:42.766428    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2573d04f-f7a6-43fb-8e40-ebabc82a8a32-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-m68qh\" (UID: \"2573d04f-f7a6-43fb-8e40-ebabc82a8a32\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-m68qh"
Sep 09 21:34:42.766570 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:34:42.766452    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bk6c\" (UniqueName: \"kubernetes.io/projected/2573d04f-f7a6-43fb-8e40-ebabc82a8a32-kube-api-access-4bk6c\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-m68qh\" (UID: \"2573d04f-f7a6-43fb-8e40-ebabc82a8a32\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-m68qh"
Sep 09 21:34:42.766570 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:34:42.766482    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2573d04f-f7a6-43fb-8e40-ebabc82a8a32-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-m68qh\" (UID: \"2573d04f-f7a6-43fb-8e40-ebabc82a8a32\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-m68qh"
Sep 09 21:34:42.867470 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:34:42.867434    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2573d04f-f7a6-43fb-8e40-ebabc82a8a32-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-m68qh\" (UID: \"2573d04f-f7a6-43fb-8e40-ebabc82a8a32\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-m68qh"
Sep 09 21:34:42.867470 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:34:42.867470    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2573d04f-f7a6-43fb-8e40-ebabc82a8a32-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-m68qh\" (UID: \"2573d04f-f7a6-43fb-8e40-ebabc82a8a32\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-m68qh"
Sep 09 21:34:42.867728 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:34:42.867492    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bk6c\" (UniqueName: \"kubernetes.io/projected/2573d04f-f7a6-43fb-8e40-ebabc82a8a32-kube-api-access-4bk6c\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-m68qh\" (UID: \"2573d04f-f7a6-43fb-8e40-ebabc82a8a32\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-m68qh"
Sep 09 21:34:42.867728 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:34:42.867542    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2573d04f-f7a6-43fb-8e40-ebabc82a8a32-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-m68qh\" (UID: \"2573d04f-f7a6-43fb-8e40-ebabc82a8a32\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-m68qh"
Sep 09 21:34:42.867728 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:34:42.867612    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2573d04f-f7a6-43fb-8e40-ebabc82a8a32-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-m68qh\" (UID: \"2573d04f-f7a6-43fb-8e40-ebabc82a8a32\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-m68qh"
Sep 09 21:34:42.867728 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:34:42.867658    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2573d04f-f7a6-43fb-8e40-ebabc82a8a32-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-m68qh\" (UID: \"2573d04f-f7a6-43fb-8e40-ebabc82a8a32\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-m68qh"
Sep 09 21:34:42.867972 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:34:42.867885    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2573d04f-f7a6-43fb-8e40-ebabc82a8a32-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-m68qh\" (UID: \"2573d04f-f7a6-43fb-8e40-ebabc82a8a32\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-m68qh"
Sep 09 21:34:42.867972 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:34:42.867933    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2573d04f-f7a6-43fb-8e40-ebabc82a8a32-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-m68qh\" (UID: \"2573d04f-f7a6-43fb-8e40-ebabc82a8a32\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-m68qh"
Sep 09 21:34:42.867972 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:34:42.867966    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2573d04f-f7a6-43fb-8e40-ebabc82a8a32-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-m68qh\" (UID: \"2573d04f-f7a6-43fb-8e40-ebabc82a8a32\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-m68qh"
Sep 09 21:34:42.870028 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:34:42.870005    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2573d04f-f7a6-43fb-8e40-ebabc82a8a32-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-m68qh\" (UID: \"2573d04f-f7a6-43fb-8e40-ebabc82a8a32\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-m68qh"
Sep 09 21:34:42.870195 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:34:42.870177    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2573d04f-f7a6-43fb-8e40-ebabc82a8a32-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-m68qh\" (UID: \"2573d04f-f7a6-43fb-8e40-ebabc82a8a32\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-m68qh"
Sep 09 21:34:42.886463 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:34:42.886439    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bk6c\" (UniqueName: \"kubernetes.io/projected/2573d04f-f7a6-43fb-8e40-ebabc82a8a32-kube-api-access-4bk6c\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-m68qh\" (UID: \"2573d04f-f7a6-43fb-8e40-ebabc82a8a32\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-m68qh"
Sep 09 21:34:42.958010 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:34:42.957918    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-m68qh"
Sep 09 21:34:43.047602 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:34:43.047568    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:34:43.242944 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:34:43.242855    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-m68qh"]
Sep 09 21:34:43.246134 ip-10-0-134-72 kubenswrapper[2576]: W0909 21:34:43.246109    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2573d04f_f7a6_43fb_8e40_ebabc82a8a32.slice/crio-a3ef743d4fca6c5f4b05ab1ac70e29099b75412251bc69bb80acfd4071df82fc WatchSource:0}: Error finding container a3ef743d4fca6c5f4b05ab1ac70e29099b75412251bc69bb80acfd4071df82fc: Status 404 returned error can't find the container with id a3ef743d4fca6c5f4b05ab1ac70e29099b75412251bc69bb80acfd4071df82fc
Sep 09 21:34:44.240645 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:34:44.240608    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-m68qh" event={"ID":"2573d04f-f7a6-43fb-8e40-ebabc82a8a32","Type":"ContainerStarted","Data":"7c0ec8d137deac9fdb14cf5dd24f702e1fceb7e6a3ea0f393fb9a7cdd62f089d"}
Sep 09 21:34:44.241008 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:34:44.240667    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-m68qh" event={"ID":"2573d04f-f7a6-43fb-8e40-ebabc82a8a32","Type":"ContainerStarted","Data":"a3ef743d4fca6c5f4b05ab1ac70e29099b75412251bc69bb80acfd4071df82fc"}
Sep 09 21:34:44.287810 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:34:44.287756    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-m68qh" podStartSLOduration=2.287740856 podStartE2EDuration="2.287740856s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:34:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:34:44.285904614 +0000 UTC m=+2527.399590473" watchObservedRunningTime="2026-09-09 21:34:44.287740856 +0000 UTC m=+2527.401426714"
Sep 09 21:34:44.958458 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:34:44.958420    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-m68qh"
Sep 09 21:34:48.269401 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:34:48.269345    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:34:48.391145 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:34:48.391114    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:34:52.958433 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:34:52.958399    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-m68qh"
Sep 09 21:34:52.971110 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:34:52.971084    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-m68qh"
Sep 09 21:34:53.303434 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:34:53.303328    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-m68qh"
Sep 09 21:35:14.146197 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:35:14.146161    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:35:14.226628 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:35:14.226585    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:35:21.608865 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:35:21.608826    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:35:26.284656 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:35:26.284610    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:35:26.389484 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:35:26.389450    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:35:27.826807 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:35:27.826770    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:35:27.918018 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:35:27.917984    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:35:38.409011 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:35:38.408969    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9955wg74"]
Sep 09 21:35:38.412404 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:35:38.412383    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9955wg74"
Sep 09 21:35:38.416552 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:35:38.416523    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvb11a9c9832b99b016bc8f8e0ea095712-kserve-self-signed-certs\""
Sep 09 21:35:38.430458 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:35:38.430435    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9955wg74"]
Sep 09 21:35:38.438583 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:35:38.438553    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:35:38.546236 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:35:38.546201    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73fc2c16-0c2e-4b73-a187-8afee74d0f35-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9955wg74\" (UID: \"73fc2c16-0c2e-4b73-a187-8afee74d0f35\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9955wg74"
Sep 09 21:35:38.546236 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:35:38.546235    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73fc2c16-0c2e-4b73-a187-8afee74d0f35-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9955wg74\" (UID: \"73fc2c16-0c2e-4b73-a187-8afee74d0f35\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9955wg74"
Sep 09 21:35:38.546492 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:35:38.546297    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73fc2c16-0c2e-4b73-a187-8afee74d0f35-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9955wg74\" (UID: \"73fc2c16-0c2e-4b73-a187-8afee74d0f35\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9955wg74"
Sep 09 21:35:38.546492 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:35:38.546357    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73fc2c16-0c2e-4b73-a187-8afee74d0f35-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9955wg74\" (UID: \"73fc2c16-0c2e-4b73-a187-8afee74d0f35\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9955wg74"
Sep 09 21:35:38.546492 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:35:38.546402    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73fc2c16-0c2e-4b73-a187-8afee74d0f35-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9955wg74\" (UID: \"73fc2c16-0c2e-4b73-a187-8afee74d0f35\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9955wg74"
Sep 09 21:35:38.546492 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:35:38.546454    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28zrx\" (UniqueName: \"kubernetes.io/projected/73fc2c16-0c2e-4b73-a187-8afee74d0f35-kube-api-access-28zrx\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9955wg74\" (UID: \"73fc2c16-0c2e-4b73-a187-8afee74d0f35\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9955wg74"
Sep 09 21:35:38.647625 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:35:38.647593    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73fc2c16-0c2e-4b73-a187-8afee74d0f35-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9955wg74\" (UID: \"73fc2c16-0c2e-4b73-a187-8afee74d0f35\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9955wg74"
Sep 09 21:35:38.647625 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:35:38.647624    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73fc2c16-0c2e-4b73-a187-8afee74d0f35-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9955wg74\" (UID: \"73fc2c16-0c2e-4b73-a187-8afee74d0f35\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9955wg74"
Sep 09 21:35:38.647863 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:35:38.647660    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28zrx\" (UniqueName: \"kubernetes.io/projected/73fc2c16-0c2e-4b73-a187-8afee74d0f35-kube-api-access-28zrx\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9955wg74\" (UID: \"73fc2c16-0c2e-4b73-a187-8afee74d0f35\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9955wg74"
Sep 09 21:35:38.647863 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:35:38.647686    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73fc2c16-0c2e-4b73-a187-8afee74d0f35-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9955wg74\" (UID: \"73fc2c16-0c2e-4b73-a187-8afee74d0f35\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9955wg74"
Sep 09 21:35:38.647863 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:35:38.647704    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73fc2c16-0c2e-4b73-a187-8afee74d0f35-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9955wg74\" (UID: \"73fc2c16-0c2e-4b73-a187-8afee74d0f35\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9955wg74"
Sep 09 21:35:38.647863 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:35:38.647737    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73fc2c16-0c2e-4b73-a187-8afee74d0f35-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9955wg74\" (UID: \"73fc2c16-0c2e-4b73-a187-8afee74d0f35\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9955wg74"
Sep 09 21:35:38.648104 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:35:38.648051    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73fc2c16-0c2e-4b73-a187-8afee74d0f35-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9955wg74\" (UID: \"73fc2c16-0c2e-4b73-a187-8afee74d0f35\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9955wg74"
Sep 09 21:35:38.648104 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:35:38.648070    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73fc2c16-0c2e-4b73-a187-8afee74d0f35-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9955wg74\" (UID: \"73fc2c16-0c2e-4b73-a187-8afee74d0f35\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9955wg74"
Sep 09 21:35:38.648221 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:35:38.648134    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73fc2c16-0c2e-4b73-a187-8afee74d0f35-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9955wg74\" (UID: \"73fc2c16-0c2e-4b73-a187-8afee74d0f35\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9955wg74"
Sep 09 21:35:38.650288 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:35:38.650267    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73fc2c16-0c2e-4b73-a187-8afee74d0f35-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9955wg74\" (UID: \"73fc2c16-0c2e-4b73-a187-8afee74d0f35\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9955wg74"
Sep 09 21:35:38.650578 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:35:38.650560    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73fc2c16-0c2e-4b73-a187-8afee74d0f35-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9955wg74\" (UID: \"73fc2c16-0c2e-4b73-a187-8afee74d0f35\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9955wg74"
Sep 09 21:35:38.658483 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:35:38.658459    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-28zrx\" (UniqueName: \"kubernetes.io/projected/73fc2c16-0c2e-4b73-a187-8afee74d0f35-kube-api-access-28zrx\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9955wg74\" (UID: \"73fc2c16-0c2e-4b73-a187-8afee74d0f35\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9955wg74"
Sep 09 21:35:38.680984 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:35:38.680912    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:35:38.722048 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:35:38.722021    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9955wg74"
Sep 09 21:35:39.000653 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:35:39.000627    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9955wg74"]
Sep 09 21:35:39.002466 ip-10-0-134-72 kubenswrapper[2576]: W0909 21:35:39.002440    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73fc2c16_0c2e_4b73_a187_8afee74d0f35.slice/crio-bc7a756f34c4a87a01cb6c8bf7d551382a7fc84d40cfd8421afec63394680dd6 WatchSource:0}: Error finding container bc7a756f34c4a87a01cb6c8bf7d551382a7fc84d40cfd8421afec63394680dd6: Status 404 returned error can't find the container with id bc7a756f34c4a87a01cb6c8bf7d551382a7fc84d40cfd8421afec63394680dd6
Sep 09 21:35:39.517738 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:35:39.517703    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9955wg74" event={"ID":"73fc2c16-0c2e-4b73-a187-8afee74d0f35","Type":"ContainerStarted","Data":"b40751a12eb470fda2c358fc41955a3060edf3dd5fb750982a3e1051af96349d"}
Sep 09 21:35:39.518132 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:35:39.517757    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9955wg74" event={"ID":"73fc2c16-0c2e-4b73-a187-8afee74d0f35","Type":"ContainerStarted","Data":"bc7a756f34c4a87a01cb6c8bf7d551382a7fc84d40cfd8421afec63394680dd6"}
Sep 09 21:35:39.544110 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:35:39.544062    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9955wg74" podStartSLOduration=1.5440468429999998 podStartE2EDuration="1.544046843s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:35:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:35:39.541506779 +0000 UTC m=+2582.655192638" watchObservedRunningTime="2026-09-09 21:35:39.544046843 +0000 UTC m=+2582.657732702"
Sep 09 21:35:39.722348 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:35:39.722311    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9955wg74"
Sep 09 21:35:47.247119 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:35:47.247083    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:35:48.722764 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:35:48.722726    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9955wg74"
Sep 09 21:35:48.735644 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:35:48.735616    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9955wg74"
Sep 09 21:35:49.580415 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:35:49.580388    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9955wg74"
Sep 09 21:36:03.948836 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:36:03.948754    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:36:10.055870 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:36:10.055834    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:36:29.670345 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:36:29.670308    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:36:29.691174 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:36:29.691142    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9955wg74"]
Sep 09 21:36:29.706030 ip-10-0-134-72 kubenswrapper[2576]: E0909 21:36:29.705987    2576 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 09 21:36:29.706030 ip-10-0-134-72 kubenswrapper[2576]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 21:36:29.706030 ip-10-0-134-72 kubenswrapper[2576]:         , stdout: , stderr: , exit code -1
Sep 09 21:36:29.706030 ip-10-0-134-72 kubenswrapper[2576]:  > containerID="b40751a12eb470fda2c358fc41955a3060edf3dd5fb750982a3e1051af96349d" cmd=["/bin/sleep","15"]
Sep 09 21:36:29.706348 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:36:29.706045    2576 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9955wg74" message="" err=<
Sep 09 21:36:29.706348 ip-10-0-134-72 kubenswrapper[2576]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 21:36:29.706348 ip-10-0-134-72 kubenswrapper[2576]:         , stdout: , stderr: , exit code -1
Sep 09 21:36:29.706348 ip-10-0-134-72 kubenswrapper[2576]:  >
Sep 09 21:36:29.706348 ip-10-0-134-72 kubenswrapper[2576]: E0909 21:36:29.706058    2576 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 09 21:36:29.706348 ip-10-0-134-72 kubenswrapper[2576]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 21:36:29.706348 ip-10-0-134-72 kubenswrapper[2576]:         , stdout: , stderr: , exit code -1
Sep 09 21:36:29.706348 ip-10-0-134-72 kubenswrapper[2576]:  > pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9955wg74" podUID="73fc2c16-0c2e-4b73-a187-8afee74d0f35" containerName="main" containerID="cri-o://b40751a12eb470fda2c358fc41955a3060edf3dd5fb750982a3e1051af96349d"
Sep 09 21:36:29.706348 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:36:29.706102    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9955wg74" podUID="73fc2c16-0c2e-4b73-a187-8afee74d0f35" containerName="main" containerID="cri-o://b40751a12eb470fda2c358fc41955a3060edf3dd5fb750982a3e1051af96349d" gracePeriod=60
Sep 09 21:36:29.764869 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:36:29.764841    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:36:30.064324 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:36:30.064299    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9955wg74"
Sep 09 21:36:30.187626 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:36:30.187541    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-28zrx\" (UniqueName: \"kubernetes.io/projected/73fc2c16-0c2e-4b73-a187-8afee74d0f35-kube-api-access-28zrx\") pod \"73fc2c16-0c2e-4b73-a187-8afee74d0f35\" (UID: \"73fc2c16-0c2e-4b73-a187-8afee74d0f35\") "
Sep 09 21:36:30.187799 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:36:30.187635    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73fc2c16-0c2e-4b73-a187-8afee74d0f35-dshm\") pod \"73fc2c16-0c2e-4b73-a187-8afee74d0f35\" (UID: \"73fc2c16-0c2e-4b73-a187-8afee74d0f35\") "
Sep 09 21:36:30.187799 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:36:30.187680    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73fc2c16-0c2e-4b73-a187-8afee74d0f35-tmp-dir\") pod \"73fc2c16-0c2e-4b73-a187-8afee74d0f35\" (UID: \"73fc2c16-0c2e-4b73-a187-8afee74d0f35\") "
Sep 09 21:36:30.187799 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:36:30.187714    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73fc2c16-0c2e-4b73-a187-8afee74d0f35-home\") pod \"73fc2c16-0c2e-4b73-a187-8afee74d0f35\" (UID: \"73fc2c16-0c2e-4b73-a187-8afee74d0f35\") "
Sep 09 21:36:30.187799 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:36:30.187754    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73fc2c16-0c2e-4b73-a187-8afee74d0f35-tls-certs\") pod \"73fc2c16-0c2e-4b73-a187-8afee74d0f35\" (UID: \"73fc2c16-0c2e-4b73-a187-8afee74d0f35\") "
Sep 09 21:36:30.188038 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:36:30.187802    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73fc2c16-0c2e-4b73-a187-8afee74d0f35-model-cache\") pod \"73fc2c16-0c2e-4b73-a187-8afee74d0f35\" (UID: \"73fc2c16-0c2e-4b73-a187-8afee74d0f35\") "
Sep 09 21:36:30.188038 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:36:30.187932    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73fc2c16-0c2e-4b73-a187-8afee74d0f35-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "73fc2c16-0c2e-4b73-a187-8afee74d0f35" (UID: "73fc2c16-0c2e-4b73-a187-8afee74d0f35"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:36:30.188038 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:36:30.187991    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73fc2c16-0c2e-4b73-a187-8afee74d0f35-home" (OuterVolumeSpecName: "home") pod "73fc2c16-0c2e-4b73-a187-8afee74d0f35" (UID: "73fc2c16-0c2e-4b73-a187-8afee74d0f35"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:36:30.188197 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:36:30.188114    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73fc2c16-0c2e-4b73-a187-8afee74d0f35-model-cache" (OuterVolumeSpecName: "model-cache") pod "73fc2c16-0c2e-4b73-a187-8afee74d0f35" (UID: "73fc2c16-0c2e-4b73-a187-8afee74d0f35"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:36:30.188197 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:36:30.188138    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73fc2c16-0c2e-4b73-a187-8afee74d0f35-tmp-dir\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:36:30.188197 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:36:30.188158    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73fc2c16-0c2e-4b73-a187-8afee74d0f35-home\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:36:30.190011 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:36:30.189986    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73fc2c16-0c2e-4b73-a187-8afee74d0f35-dshm" (OuterVolumeSpecName: "dshm") pod "73fc2c16-0c2e-4b73-a187-8afee74d0f35" (UID: "73fc2c16-0c2e-4b73-a187-8afee74d0f35"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:36:30.190158 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:36:30.190136    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73fc2c16-0c2e-4b73-a187-8afee74d0f35-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "73fc2c16-0c2e-4b73-a187-8afee74d0f35" (UID: "73fc2c16-0c2e-4b73-a187-8afee74d0f35"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:36:30.190232 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:36:30.190196    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73fc2c16-0c2e-4b73-a187-8afee74d0f35-kube-api-access-28zrx" (OuterVolumeSpecName: "kube-api-access-28zrx") pod "73fc2c16-0c2e-4b73-a187-8afee74d0f35" (UID: "73fc2c16-0c2e-4b73-a187-8afee74d0f35"). InnerVolumeSpecName "kube-api-access-28zrx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:36:30.289540 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:36:30.289506    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73fc2c16-0c2e-4b73-a187-8afee74d0f35-tls-certs\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:36:30.289540 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:36:30.289533    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73fc2c16-0c2e-4b73-a187-8afee74d0f35-model-cache\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:36:30.289540 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:36:30.289542    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28zrx\" (UniqueName: \"kubernetes.io/projected/73fc2c16-0c2e-4b73-a187-8afee74d0f35-kube-api-access-28zrx\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:36:30.289768 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:36:30.289551    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73fc2c16-0c2e-4b73-a187-8afee74d0f35-dshm\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:36:30.783709 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:36:30.783674    2576 generic.go:356] "Generic (PLEG): container finished" podID="73fc2c16-0c2e-4b73-a187-8afee74d0f35" containerID="b40751a12eb470fda2c358fc41955a3060edf3dd5fb750982a3e1051af96349d" exitCode=0
Sep 09 21:36:30.784174 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:36:30.783718    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9955wg74"
Sep 09 21:36:30.784174 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:36:30.783732    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9955wg74" event={"ID":"73fc2c16-0c2e-4b73-a187-8afee74d0f35","Type":"ContainerDied","Data":"b40751a12eb470fda2c358fc41955a3060edf3dd5fb750982a3e1051af96349d"}
Sep 09 21:36:30.784174 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:36:30.783785    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9955wg74" event={"ID":"73fc2c16-0c2e-4b73-a187-8afee74d0f35","Type":"ContainerDied","Data":"bc7a756f34c4a87a01cb6c8bf7d551382a7fc84d40cfd8421afec63394680dd6"}
Sep 09 21:36:30.784174 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:36:30.783819    2576 scope.go:117] "RemoveContainer" containerID="b40751a12eb470fda2c358fc41955a3060edf3dd5fb750982a3e1051af96349d"
Sep 09 21:36:30.793547 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:36:30.793529    2576 scope.go:117] "RemoveContainer" containerID="b40751a12eb470fda2c358fc41955a3060edf3dd5fb750982a3e1051af96349d"
Sep 09 21:36:30.793794 ip-10-0-134-72 kubenswrapper[2576]: E0909 21:36:30.793775    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b40751a12eb470fda2c358fc41955a3060edf3dd5fb750982a3e1051af96349d\": container with ID starting with b40751a12eb470fda2c358fc41955a3060edf3dd5fb750982a3e1051af96349d not found: ID does not exist" containerID="b40751a12eb470fda2c358fc41955a3060edf3dd5fb750982a3e1051af96349d"
Sep 09 21:36:30.793855 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:36:30.793806    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b40751a12eb470fda2c358fc41955a3060edf3dd5fb750982a3e1051af96349d"} err="failed to get container status \"b40751a12eb470fda2c358fc41955a3060edf3dd5fb750982a3e1051af96349d\": rpc error: code = NotFound desc = could not find container \"b40751a12eb470fda2c358fc41955a3060edf3dd5fb750982a3e1051af96349d\": container with ID starting with b40751a12eb470fda2c358fc41955a3060edf3dd5fb750982a3e1051af96349d not found: ID does not exist"
Sep 09 21:36:30.810013 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:36:30.809988    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9955wg74"]
Sep 09 21:36:30.816927 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:36:30.816907    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9955wg74"]
Sep 09 21:36:31.408850 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:36:31.408815    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="73fc2c16-0c2e-4b73-a187-8afee74d0f35" path="/var/lib/kubelet/pods/73fc2c16-0c2e-4b73-a187-8afee74d0f35/volumes"
Sep 09 21:36:42.264686 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:36:42.264638    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:37:03.915729 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:37:03.915693    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:37:04.011512 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:37:04.011473    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:37:04.300278 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:37:04.300247    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:37:04.433434 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:37:04.433391    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:37:07.903662 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:37:07.903627    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:37:09.060679 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:37:09.060646    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:37:17.105381 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:37:17.105317    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:37:18.703983 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:37:18.703949    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mf5gl"]
Sep 09 21:37:18.704375 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:37:18.704305    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73fc2c16-0c2e-4b73-a187-8afee74d0f35" containerName="main"
Sep 09 21:37:18.704375 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:37:18.704317    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="73fc2c16-0c2e-4b73-a187-8afee74d0f35" containerName="main"
Sep 09 21:37:18.704474 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:37:18.704399    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="73fc2c16-0c2e-4b73-a187-8afee74d0f35" containerName="main"
Sep 09 21:37:18.707306 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:37:18.707282    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mf5gl"
Sep 09 21:37:18.710936 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:37:18.710916    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"default-dockercfg-cpz5x\""
Sep 09 21:37:18.711449 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:37:18.711432    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"openshift-service-ca.crt\""
Sep 09 21:37:18.711449 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:37:18.711443    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"kube-root-ca.crt\""
Sep 09 21:37:18.711630 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:37:18.711430    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"llmisv122f03714c5bdf915a2917fdf1262b98-kserve-self-signed-certs\""
Sep 09 21:37:18.748460 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:37:18.748435    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mf5gl"]
Sep 09 21:37:18.758915 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:37:18.758889    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:37:18.777964 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:37:18.777942    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4f8f35e-811e-4581-bdf6-a5771ff2875a-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mf5gl\" (UID: \"d4f8f35e-811e-4581-bdf6-a5771ff2875a\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mf5gl"
Sep 09 21:37:18.778052 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:37:18.777995    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4f8f35e-811e-4581-bdf6-a5771ff2875a-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mf5gl\" (UID: \"d4f8f35e-811e-4581-bdf6-a5771ff2875a\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mf5gl"
Sep 09 21:37:18.778052 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:37:18.778020    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7m4bn\" (UniqueName: \"kubernetes.io/projected/d4f8f35e-811e-4581-bdf6-a5771ff2875a-kube-api-access-7m4bn\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mf5gl\" (UID: \"d4f8f35e-811e-4581-bdf6-a5771ff2875a\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mf5gl"
Sep 09 21:37:18.778137 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:37:18.778053    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4f8f35e-811e-4581-bdf6-a5771ff2875a-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mf5gl\" (UID: \"d4f8f35e-811e-4581-bdf6-a5771ff2875a\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mf5gl"
Sep 09 21:37:18.778137 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:37:18.778094    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4f8f35e-811e-4581-bdf6-a5771ff2875a-home\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mf5gl\" (UID: \"d4f8f35e-811e-4581-bdf6-a5771ff2875a\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mf5gl"
Sep 09 21:37:18.778137 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:37:18.778108    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4f8f35e-811e-4581-bdf6-a5771ff2875a-dshm\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mf5gl\" (UID: \"d4f8f35e-811e-4581-bdf6-a5771ff2875a\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mf5gl"
Sep 09 21:37:18.878696 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:37:18.878668    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4f8f35e-811e-4581-bdf6-a5771ff2875a-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mf5gl\" (UID: \"d4f8f35e-811e-4581-bdf6-a5771ff2875a\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mf5gl"
Sep 09 21:37:18.878696 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:37:18.878698    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7m4bn\" (UniqueName: \"kubernetes.io/projected/d4f8f35e-811e-4581-bdf6-a5771ff2875a-kube-api-access-7m4bn\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mf5gl\" (UID: \"d4f8f35e-811e-4581-bdf6-a5771ff2875a\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mf5gl"
Sep 09 21:37:18.878877 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:37:18.878721    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4f8f35e-811e-4581-bdf6-a5771ff2875a-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mf5gl\" (UID: \"d4f8f35e-811e-4581-bdf6-a5771ff2875a\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mf5gl"
Sep 09 21:37:18.878877 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:37:18.878774    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4f8f35e-811e-4581-bdf6-a5771ff2875a-home\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mf5gl\" (UID: \"d4f8f35e-811e-4581-bdf6-a5771ff2875a\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mf5gl"
Sep 09 21:37:18.878877 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:37:18.878798    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4f8f35e-811e-4581-bdf6-a5771ff2875a-dshm\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mf5gl\" (UID: \"d4f8f35e-811e-4581-bdf6-a5771ff2875a\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mf5gl"
Sep 09 21:37:18.878877 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:37:18.878850    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4f8f35e-811e-4581-bdf6-a5771ff2875a-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mf5gl\" (UID: \"d4f8f35e-811e-4581-bdf6-a5771ff2875a\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mf5gl"
Sep 09 21:37:18.879207 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:37:18.879188    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4f8f35e-811e-4581-bdf6-a5771ff2875a-home\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mf5gl\" (UID: \"d4f8f35e-811e-4581-bdf6-a5771ff2875a\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mf5gl"
Sep 09 21:37:18.879283 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:37:18.879212    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4f8f35e-811e-4581-bdf6-a5771ff2875a-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mf5gl\" (UID: \"d4f8f35e-811e-4581-bdf6-a5771ff2875a\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mf5gl"
Sep 09 21:37:18.879283 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:37:18.879262    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4f8f35e-811e-4581-bdf6-a5771ff2875a-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mf5gl\" (UID: \"d4f8f35e-811e-4581-bdf6-a5771ff2875a\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mf5gl"
Sep 09 21:37:18.881245 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:37:18.881225    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4f8f35e-811e-4581-bdf6-a5771ff2875a-dshm\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mf5gl\" (UID: \"d4f8f35e-811e-4581-bdf6-a5771ff2875a\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mf5gl"
Sep 09 21:37:18.881349 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:37:18.881332    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4f8f35e-811e-4581-bdf6-a5771ff2875a-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mf5gl\" (UID: \"d4f8f35e-811e-4581-bdf6-a5771ff2875a\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mf5gl"
Sep 09 21:37:18.890241 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:37:18.890217    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7m4bn\" (UniqueName: \"kubernetes.io/projected/d4f8f35e-811e-4581-bdf6-a5771ff2875a-kube-api-access-7m4bn\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mf5gl\" (UID: \"d4f8f35e-811e-4581-bdf6-a5771ff2875a\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mf5gl"
Sep 09 21:37:19.018127 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:37:19.018050    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mf5gl"
Sep 09 21:37:19.299598 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:37:19.299569    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mf5gl"]
Sep 09 21:37:19.300217 ip-10-0-134-72 kubenswrapper[2576]: W0909 21:37:19.300193    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4f8f35e_811e_4581_bdf6_a5771ff2875a.slice/crio-193ea732636a01f75ab5607862d2152e93b849876cc89e4f87b3b032eb2ed4f2 WatchSource:0}: Error finding container 193ea732636a01f75ab5607862d2152e93b849876cc89e4f87b3b032eb2ed4f2: Status 404 returned error can't find the container with id 193ea732636a01f75ab5607862d2152e93b849876cc89e4f87b3b032eb2ed4f2
Sep 09 21:37:20.038004 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:37:20.037973    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mf5gl" event={"ID":"d4f8f35e-811e-4581-bdf6-a5771ff2875a","Type":"ContainerStarted","Data":"260655afe7adf583fffa2bda35abeb324892b0441f1c5cebb15bada1e91108d4"}
Sep 09 21:37:20.038432 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:37:20.038024    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mf5gl" event={"ID":"d4f8f35e-811e-4581-bdf6-a5771ff2875a","Type":"ContainerStarted","Data":"193ea732636a01f75ab5607862d2152e93b849876cc89e4f87b3b032eb2ed4f2"}
Sep 09 21:37:20.067650 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:37:20.067585    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mf5gl" podStartSLOduration=2.06756438 podStartE2EDuration="2.06756438s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:37:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:37:20.0632919 +0000 UTC m=+2683.176977759" watchObservedRunningTime="2026-09-09 21:37:20.06756438 +0000 UTC m=+2683.181250239"
Sep 09 21:37:21.018283 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:37:21.018246    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mf5gl"
Sep 09 21:37:21.321130 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:37:21.321031    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-m68qh"]
Sep 09 21:37:21.322264 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:37:21.322211    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:37:21.330395 ip-10-0-134-72 kubenswrapper[2576]: E0909 21:37:21.330342    2576 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 09 21:37:21.330395 ip-10-0-134-72 kubenswrapper[2576]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 21:37:21.330395 ip-10-0-134-72 kubenswrapper[2576]:         , stdout: , stderr: , exit code -1
Sep 09 21:37:21.330395 ip-10-0-134-72 kubenswrapper[2576]:  > containerID="7c0ec8d137deac9fdb14cf5dd24f702e1fceb7e6a3ea0f393fb9a7cdd62f089d" cmd=["/bin/sleep","15"]
Sep 09 21:37:21.330661 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:37:21.330418    2576 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-m68qh" message="" err=<
Sep 09 21:37:21.330661 ip-10-0-134-72 kubenswrapper[2576]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 21:37:21.330661 ip-10-0-134-72 kubenswrapper[2576]:         , stdout: , stderr: , exit code -1
Sep 09 21:37:21.330661 ip-10-0-134-72 kubenswrapper[2576]:  >
Sep 09 21:37:21.330661 ip-10-0-134-72 kubenswrapper[2576]: E0909 21:37:21.330436    2576 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 09 21:37:21.330661 ip-10-0-134-72 kubenswrapper[2576]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 21:37:21.330661 ip-10-0-134-72 kubenswrapper[2576]:         , stdout: , stderr: , exit code -1
Sep 09 21:37:21.330661 ip-10-0-134-72 kubenswrapper[2576]:  > pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-m68qh" podUID="2573d04f-f7a6-43fb-8e40-ebabc82a8a32" containerName="main" containerID="cri-o://7c0ec8d137deac9fdb14cf5dd24f702e1fceb7e6a3ea0f393fb9a7cdd62f089d"
Sep 09 21:37:21.330661 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:37:21.330481    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-m68qh" podUID="2573d04f-f7a6-43fb-8e40-ebabc82a8a32" containerName="main" containerID="cri-o://7c0ec8d137deac9fdb14cf5dd24f702e1fceb7e6a3ea0f393fb9a7cdd62f089d" gracePeriod=60
Sep 09 21:37:21.421718 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:37:21.421683    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:37:22.049842 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:37:22.049806    2576 generic.go:356] "Generic (PLEG): container finished" podID="2573d04f-f7a6-43fb-8e40-ebabc82a8a32" containerID="7c0ec8d137deac9fdb14cf5dd24f702e1fceb7e6a3ea0f393fb9a7cdd62f089d" exitCode=0
Sep 09 21:37:22.050039 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:37:22.049885    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-m68qh" event={"ID":"2573d04f-f7a6-43fb-8e40-ebabc82a8a32","Type":"ContainerDied","Data":"7c0ec8d137deac9fdb14cf5dd24f702e1fceb7e6a3ea0f393fb9a7cdd62f089d"}
Sep 09 21:37:22.092123 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:37:22.092089    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-m68qh"
Sep 09 21:37:22.208924 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:37:22.208845    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2573d04f-f7a6-43fb-8e40-ebabc82a8a32-model-cache\") pod \"2573d04f-f7a6-43fb-8e40-ebabc82a8a32\" (UID: \"2573d04f-f7a6-43fb-8e40-ebabc82a8a32\") "
Sep 09 21:37:22.208924 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:37:22.208896    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2573d04f-f7a6-43fb-8e40-ebabc82a8a32-tmp-dir\") pod \"2573d04f-f7a6-43fb-8e40-ebabc82a8a32\" (UID: \"2573d04f-f7a6-43fb-8e40-ebabc82a8a32\") "
Sep 09 21:37:22.208924 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:37:22.208914    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2573d04f-f7a6-43fb-8e40-ebabc82a8a32-home\") pod \"2573d04f-f7a6-43fb-8e40-ebabc82a8a32\" (UID: \"2573d04f-f7a6-43fb-8e40-ebabc82a8a32\") "
Sep 09 21:37:22.209182 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:37:22.208940    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2573d04f-f7a6-43fb-8e40-ebabc82a8a32-tls-certs\") pod \"2573d04f-f7a6-43fb-8e40-ebabc82a8a32\" (UID: \"2573d04f-f7a6-43fb-8e40-ebabc82a8a32\") "
Sep 09 21:37:22.209182 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:37:22.208965    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4bk6c\" (UniqueName: \"kubernetes.io/projected/2573d04f-f7a6-43fb-8e40-ebabc82a8a32-kube-api-access-4bk6c\") pod \"2573d04f-f7a6-43fb-8e40-ebabc82a8a32\" (UID: \"2573d04f-f7a6-43fb-8e40-ebabc82a8a32\") "
Sep 09 21:37:22.209182 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:37:22.209003    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2573d04f-f7a6-43fb-8e40-ebabc82a8a32-dshm\") pod \"2573d04f-f7a6-43fb-8e40-ebabc82a8a32\" (UID: \"2573d04f-f7a6-43fb-8e40-ebabc82a8a32\") "
Sep 09 21:37:22.209182 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:37:22.209138    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2573d04f-f7a6-43fb-8e40-ebabc82a8a32-home" (OuterVolumeSpecName: "home") pod "2573d04f-f7a6-43fb-8e40-ebabc82a8a32" (UID: "2573d04f-f7a6-43fb-8e40-ebabc82a8a32"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:37:22.209182 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:37:22.209150    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2573d04f-f7a6-43fb-8e40-ebabc82a8a32-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2573d04f-f7a6-43fb-8e40-ebabc82a8a32" (UID: "2573d04f-f7a6-43fb-8e40-ebabc82a8a32"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:37:22.209182 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:37:22.209158    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2573d04f-f7a6-43fb-8e40-ebabc82a8a32-model-cache" (OuterVolumeSpecName: "model-cache") pod "2573d04f-f7a6-43fb-8e40-ebabc82a8a32" (UID: "2573d04f-f7a6-43fb-8e40-ebabc82a8a32"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:37:22.211356 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:37:22.211322    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2573d04f-f7a6-43fb-8e40-ebabc82a8a32-kube-api-access-4bk6c" (OuterVolumeSpecName: "kube-api-access-4bk6c") pod "2573d04f-f7a6-43fb-8e40-ebabc82a8a32" (UID: "2573d04f-f7a6-43fb-8e40-ebabc82a8a32"). InnerVolumeSpecName "kube-api-access-4bk6c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:37:22.211508 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:37:22.211424    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2573d04f-f7a6-43fb-8e40-ebabc82a8a32-dshm" (OuterVolumeSpecName: "dshm") pod "2573d04f-f7a6-43fb-8e40-ebabc82a8a32" (UID: "2573d04f-f7a6-43fb-8e40-ebabc82a8a32"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:37:22.211508 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:37:22.211445    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2573d04f-f7a6-43fb-8e40-ebabc82a8a32-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2573d04f-f7a6-43fb-8e40-ebabc82a8a32" (UID: "2573d04f-f7a6-43fb-8e40-ebabc82a8a32"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:37:22.310433 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:37:22.310401    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2573d04f-f7a6-43fb-8e40-ebabc82a8a32-dshm\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:37:22.310433 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:37:22.310427    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2573d04f-f7a6-43fb-8e40-ebabc82a8a32-model-cache\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:37:22.310433 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:37:22.310438    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2573d04f-f7a6-43fb-8e40-ebabc82a8a32-tmp-dir\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:37:22.310644 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:37:22.310445    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2573d04f-f7a6-43fb-8e40-ebabc82a8a32-home\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:37:22.310644 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:37:22.310453    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2573d04f-f7a6-43fb-8e40-ebabc82a8a32-tls-certs\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:37:22.310644 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:37:22.310461    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bk6c\" (UniqueName: \"kubernetes.io/projected/2573d04f-f7a6-43fb-8e40-ebabc82a8a32-kube-api-access-4bk6c\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:37:23.056677 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:37:23.056644    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-m68qh" event={"ID":"2573d04f-f7a6-43fb-8e40-ebabc82a8a32","Type":"ContainerDied","Data":"a3ef743d4fca6c5f4b05ab1ac70e29099b75412251bc69bb80acfd4071df82fc"}
Sep 09 21:37:23.056677 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:37:23.056680    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-m68qh"
Sep 09 21:37:23.057184 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:37:23.056707    2576 scope.go:117] "RemoveContainer" containerID="7c0ec8d137deac9fdb14cf5dd24f702e1fceb7e6a3ea0f393fb9a7cdd62f089d"
Sep 09 21:37:23.086756 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:37:23.086727    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-m68qh"]
Sep 09 21:37:23.091243 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:37:23.091221    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-m68qh"]
Sep 09 21:37:23.409746 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:37:23.409707    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2573d04f-f7a6-43fb-8e40-ebabc82a8a32" path="/var/lib/kubelet/pods/2573d04f-f7a6-43fb-8e40-ebabc82a8a32/volumes"
Sep 09 21:37:29.018521 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:37:29.018439    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mf5gl"
Sep 09 21:37:29.030898 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:37:29.030870    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mf5gl"
Sep 09 21:37:29.099346 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:37:29.099322    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mf5gl"
Sep 09 21:37:47.608025 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:37:47.607992    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:37:54.396526 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:37:54.396493    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:37:54.531101 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:37:54.531071    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:38:09.892663 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:38:09.892627    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:38:10.007177 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:38:10.007136    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:38:17.780607 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:38:17.780569    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:38:24.649332 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:38:24.649292    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:38:24.748839 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:38:24.748801    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:38:27.315130 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:38:27.315100    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969gl4vj"]
Sep 09 21:38:27.315623 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:38:27.315454    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2573d04f-f7a6-43fb-8e40-ebabc82a8a32" containerName="main"
Sep 09 21:38:27.315623 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:38:27.315466    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="2573d04f-f7a6-43fb-8e40-ebabc82a8a32" containerName="main"
Sep 09 21:38:27.315623 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:38:27.315541    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="2573d04f-f7a6-43fb-8e40-ebabc82a8a32" containerName="main"
Sep 09 21:38:27.318578 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:38:27.318562    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969gl4vj"
Sep 09 21:38:27.324219 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:38:27.324192    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"kube-root-ca.crt\""
Sep 09 21:38:27.324359 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:38:27.324192    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvca2d2d7d499abb359505529ebe02c136-kserve-self-signed-certs\""
Sep 09 21:38:27.324359 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:38:27.324230    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"default-dockercfg-spp4s\""
Sep 09 21:38:27.324533 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:38:27.324520    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"openshift-service-ca.crt\""
Sep 09 21:38:27.343787 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:38:27.343760    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969gl4vj"]
Sep 09 21:38:27.353934 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:38:27.353908    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:38:27.462280 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:38:27.462231    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fddf9260-84a7-4724-a9e7-1cec80caeb31-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969gl4vj\" (UID: \"fddf9260-84a7-4724-a9e7-1cec80caeb31\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969gl4vj"
Sep 09 21:38:27.462280 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:38:27.462281    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fddf9260-84a7-4724-a9e7-1cec80caeb31-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969gl4vj\" (UID: \"fddf9260-84a7-4724-a9e7-1cec80caeb31\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969gl4vj"
Sep 09 21:38:27.462546 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:38:27.462344    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fddf9260-84a7-4724-a9e7-1cec80caeb31-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969gl4vj\" (UID: \"fddf9260-84a7-4724-a9e7-1cec80caeb31\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969gl4vj"
Sep 09 21:38:27.462546 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:38:27.462433    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnp4x\" (UniqueName: \"kubernetes.io/projected/fddf9260-84a7-4724-a9e7-1cec80caeb31-kube-api-access-vnp4x\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969gl4vj\" (UID: \"fddf9260-84a7-4724-a9e7-1cec80caeb31\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969gl4vj"
Sep 09 21:38:27.462546 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:38:27.462491    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fddf9260-84a7-4724-a9e7-1cec80caeb31-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969gl4vj\" (UID: \"fddf9260-84a7-4724-a9e7-1cec80caeb31\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969gl4vj"
Sep 09 21:38:27.462546 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:38:27.462515    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fddf9260-84a7-4724-a9e7-1cec80caeb31-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969gl4vj\" (UID: \"fddf9260-84a7-4724-a9e7-1cec80caeb31\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969gl4vj"
Sep 09 21:38:27.563065 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:38:27.563031    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fddf9260-84a7-4724-a9e7-1cec80caeb31-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969gl4vj\" (UID: \"fddf9260-84a7-4724-a9e7-1cec80caeb31\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969gl4vj"
Sep 09 21:38:27.563065 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:38:27.563073    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fddf9260-84a7-4724-a9e7-1cec80caeb31-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969gl4vj\" (UID: \"fddf9260-84a7-4724-a9e7-1cec80caeb31\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969gl4vj"
Sep 09 21:38:27.563406 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:38:27.563113    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fddf9260-84a7-4724-a9e7-1cec80caeb31-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969gl4vj\" (UID: \"fddf9260-84a7-4724-a9e7-1cec80caeb31\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969gl4vj"
Sep 09 21:38:27.563406 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:38:27.563160    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnp4x\" (UniqueName: \"kubernetes.io/projected/fddf9260-84a7-4724-a9e7-1cec80caeb31-kube-api-access-vnp4x\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969gl4vj\" (UID: \"fddf9260-84a7-4724-a9e7-1cec80caeb31\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969gl4vj"
Sep 09 21:38:27.563406 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:38:27.563230    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fddf9260-84a7-4724-a9e7-1cec80caeb31-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969gl4vj\" (UID: \"fddf9260-84a7-4724-a9e7-1cec80caeb31\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969gl4vj"
Sep 09 21:38:27.563406 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:38:27.563264    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fddf9260-84a7-4724-a9e7-1cec80caeb31-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969gl4vj\" (UID: \"fddf9260-84a7-4724-a9e7-1cec80caeb31\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969gl4vj"
Sep 09 21:38:27.563663 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:38:27.563634    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fddf9260-84a7-4724-a9e7-1cec80caeb31-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969gl4vj\" (UID: \"fddf9260-84a7-4724-a9e7-1cec80caeb31\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969gl4vj"
Sep 09 21:38:27.563795 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:38:27.563725    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fddf9260-84a7-4724-a9e7-1cec80caeb31-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969gl4vj\" (UID: \"fddf9260-84a7-4724-a9e7-1cec80caeb31\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969gl4vj"
Sep 09 21:38:27.563861 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:38:27.563792    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fddf9260-84a7-4724-a9e7-1cec80caeb31-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969gl4vj\" (UID: \"fddf9260-84a7-4724-a9e7-1cec80caeb31\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969gl4vj"
Sep 09 21:38:27.566179 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:38:27.566126    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fddf9260-84a7-4724-a9e7-1cec80caeb31-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969gl4vj\" (UID: \"fddf9260-84a7-4724-a9e7-1cec80caeb31\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969gl4vj"
Sep 09 21:38:27.566663 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:38:27.566643    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fddf9260-84a7-4724-a9e7-1cec80caeb31-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969gl4vj\" (UID: \"fddf9260-84a7-4724-a9e7-1cec80caeb31\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969gl4vj"
Sep 09 21:38:27.576149 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:38:27.576124    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnp4x\" (UniqueName: \"kubernetes.io/projected/fddf9260-84a7-4724-a9e7-1cec80caeb31-kube-api-access-vnp4x\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969gl4vj\" (UID: \"fddf9260-84a7-4724-a9e7-1cec80caeb31\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969gl4vj"
Sep 09 21:38:27.621759 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:38:27.621728    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:38:27.628625 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:38:27.628602    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969gl4vj"
Sep 09 21:38:27.906817 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:38:27.906791    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969gl4vj"]
Sep 09 21:38:27.908719 ip-10-0-134-72 kubenswrapper[2576]: W0909 21:38:27.908691    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfddf9260_84a7_4724_a9e7_1cec80caeb31.slice/crio-2eee30add80b11ca1c91fc9a77b9e2bd88d722e273dd2c5e77359006d67071d9 WatchSource:0}: Error finding container 2eee30add80b11ca1c91fc9a77b9e2bd88d722e273dd2c5e77359006d67071d9: Status 404 returned error can't find the container with id 2eee30add80b11ca1c91fc9a77b9e2bd88d722e273dd2c5e77359006d67071d9
Sep 09 21:38:28.382233 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:38:28.382194    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969gl4vj" event={"ID":"fddf9260-84a7-4724-a9e7-1cec80caeb31","Type":"ContainerStarted","Data":"473bec5a9dfc001b870684cd65f0b89df3246cd15d9e4226a4a8fb8f627533e6"}
Sep 09 21:38:28.382633 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:38:28.382250    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969gl4vj" event={"ID":"fddf9260-84a7-4724-a9e7-1cec80caeb31","Type":"ContainerStarted","Data":"2eee30add80b11ca1c91fc9a77b9e2bd88d722e273dd2c5e77359006d67071d9"}
Sep 09 21:38:28.415818 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:38:28.415760    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969gl4vj" podStartSLOduration=1.415741402 podStartE2EDuration="1.415741402s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:38:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:38:28.413094981 +0000 UTC m=+2751.526780840" watchObservedRunningTime="2026-09-09 21:38:28.415741402 +0000 UTC m=+2751.529427264"
Sep 09 21:38:28.629290 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:38:28.629241    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969gl4vj"
Sep 09 21:38:37.629062 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:38:37.629023    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969gl4vj"
Sep 09 21:38:37.641608 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:38:37.641580    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969gl4vj"
Sep 09 21:38:38.448053 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:38:38.448023    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969gl4vj"
Sep 09 21:38:43.875851 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:38:43.875819    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:39:07.862243 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:39:07.862155    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:39:08.166472 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:39:08.166383    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:39:20.422685 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:39:20.422643    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:39:35.036739 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:39:35.036708    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mf5gl"]
Sep 09 21:39:35.045831 ip-10-0-134-72 kubenswrapper[2576]: E0909 21:39:35.045794    2576 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 09 21:39:35.045831 ip-10-0-134-72 kubenswrapper[2576]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 21:39:35.045831 ip-10-0-134-72 kubenswrapper[2576]:         , stdout: , stderr: , exit code -1
Sep 09 21:39:35.045831 ip-10-0-134-72 kubenswrapper[2576]:  > containerID="260655afe7adf583fffa2bda35abeb324892b0441f1c5cebb15bada1e91108d4" cmd=["/bin/sleep","15"]
Sep 09 21:39:35.046067 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:39:35.045844    2576 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mf5gl" message="" err=<
Sep 09 21:39:35.046067 ip-10-0-134-72 kubenswrapper[2576]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 21:39:35.046067 ip-10-0-134-72 kubenswrapper[2576]:         , stdout: , stderr: , exit code -1
Sep 09 21:39:35.046067 ip-10-0-134-72 kubenswrapper[2576]:  >
Sep 09 21:39:35.046067 ip-10-0-134-72 kubenswrapper[2576]: E0909 21:39:35.045858    2576 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 09 21:39:35.046067 ip-10-0-134-72 kubenswrapper[2576]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 21:39:35.046067 ip-10-0-134-72 kubenswrapper[2576]:         , stdout: , stderr: , exit code -1
Sep 09 21:39:35.046067 ip-10-0-134-72 kubenswrapper[2576]:  > pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mf5gl" podUID="d4f8f35e-811e-4581-bdf6-a5771ff2875a" containerName="main" containerID="cri-o://260655afe7adf583fffa2bda35abeb324892b0441f1c5cebb15bada1e91108d4"
Sep 09 21:39:35.046067 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:39:35.045893    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mf5gl" podUID="d4f8f35e-811e-4581-bdf6-a5771ff2875a" containerName="main" containerID="cri-o://260655afe7adf583fffa2bda35abeb324892b0441f1c5cebb15bada1e91108d4" gracePeriod=60
Sep 09 21:39:35.057538 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:39:35.057512    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:39:35.089630 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:39:35.089599    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mf5gl" podUID="d4f8f35e-811e-4581-bdf6-a5771ff2875a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": dial tcp 10.132.0.71:8000: connect: connection refused"
Sep 09 21:39:35.388385 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:39:35.388337    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mf5gl"
Sep 09 21:39:35.542294 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:39:35.542262    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4f8f35e-811e-4581-bdf6-a5771ff2875a-home\") pod \"d4f8f35e-811e-4581-bdf6-a5771ff2875a\" (UID: \"d4f8f35e-811e-4581-bdf6-a5771ff2875a\") "
Sep 09 21:39:35.542508 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:39:35.542306    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7m4bn\" (UniqueName: \"kubernetes.io/projected/d4f8f35e-811e-4581-bdf6-a5771ff2875a-kube-api-access-7m4bn\") pod \"d4f8f35e-811e-4581-bdf6-a5771ff2875a\" (UID: \"d4f8f35e-811e-4581-bdf6-a5771ff2875a\") "
Sep 09 21:39:35.542508 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:39:35.542325    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4f8f35e-811e-4581-bdf6-a5771ff2875a-model-cache\") pod \"d4f8f35e-811e-4581-bdf6-a5771ff2875a\" (UID: \"d4f8f35e-811e-4581-bdf6-a5771ff2875a\") "
Sep 09 21:39:35.542508 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:39:35.542424    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4f8f35e-811e-4581-bdf6-a5771ff2875a-dshm\") pod \"d4f8f35e-811e-4581-bdf6-a5771ff2875a\" (UID: \"d4f8f35e-811e-4581-bdf6-a5771ff2875a\") "
Sep 09 21:39:35.542508 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:39:35.542461    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4f8f35e-811e-4581-bdf6-a5771ff2875a-tmp-dir\") pod \"d4f8f35e-811e-4581-bdf6-a5771ff2875a\" (UID: \"d4f8f35e-811e-4581-bdf6-a5771ff2875a\") "
Sep 09 21:39:35.542736 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:39:35.542512    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4f8f35e-811e-4581-bdf6-a5771ff2875a-tls-certs\") pod \"d4f8f35e-811e-4581-bdf6-a5771ff2875a\" (UID: \"d4f8f35e-811e-4581-bdf6-a5771ff2875a\") "
Sep 09 21:39:35.542736 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:39:35.542617    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4f8f35e-811e-4581-bdf6-a5771ff2875a-model-cache" (OuterVolumeSpecName: "model-cache") pod "d4f8f35e-811e-4581-bdf6-a5771ff2875a" (UID: "d4f8f35e-811e-4581-bdf6-a5771ff2875a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:39:35.542736 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:39:35.542624    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4f8f35e-811e-4581-bdf6-a5771ff2875a-home" (OuterVolumeSpecName: "home") pod "d4f8f35e-811e-4581-bdf6-a5771ff2875a" (UID: "d4f8f35e-811e-4581-bdf6-a5771ff2875a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:39:35.542898 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:39:35.542775    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4f8f35e-811e-4581-bdf6-a5771ff2875a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d4f8f35e-811e-4581-bdf6-a5771ff2875a" (UID: "d4f8f35e-811e-4581-bdf6-a5771ff2875a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:39:35.542898 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:39:35.542884    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4f8f35e-811e-4581-bdf6-a5771ff2875a-home\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:39:35.542998 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:39:35.542909    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4f8f35e-811e-4581-bdf6-a5771ff2875a-model-cache\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:39:35.542998 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:39:35.542930    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4f8f35e-811e-4581-bdf6-a5771ff2875a-tmp-dir\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:39:35.544792 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:39:35.544769    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4f8f35e-811e-4581-bdf6-a5771ff2875a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d4f8f35e-811e-4581-bdf6-a5771ff2875a" (UID: "d4f8f35e-811e-4581-bdf6-a5771ff2875a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:39:35.544895 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:39:35.544841    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4f8f35e-811e-4581-bdf6-a5771ff2875a-kube-api-access-7m4bn" (OuterVolumeSpecName: "kube-api-access-7m4bn") pod "d4f8f35e-811e-4581-bdf6-a5771ff2875a" (UID: "d4f8f35e-811e-4581-bdf6-a5771ff2875a"). InnerVolumeSpecName "kube-api-access-7m4bn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:39:35.544999 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:39:35.544978    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4f8f35e-811e-4581-bdf6-a5771ff2875a-dshm" (OuterVolumeSpecName: "dshm") pod "d4f8f35e-811e-4581-bdf6-a5771ff2875a" (UID: "d4f8f35e-811e-4581-bdf6-a5771ff2875a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:39:35.644273 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:39:35.644243    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4f8f35e-811e-4581-bdf6-a5771ff2875a-dshm\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:39:35.644273 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:39:35.644269    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4f8f35e-811e-4581-bdf6-a5771ff2875a-tls-certs\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:39:35.644273 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:39:35.644280    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7m4bn\" (UniqueName: \"kubernetes.io/projected/d4f8f35e-811e-4581-bdf6-a5771ff2875a-kube-api-access-7m4bn\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:39:35.732285 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:39:35.732245    2576 generic.go:356] "Generic (PLEG): container finished" podID="d4f8f35e-811e-4581-bdf6-a5771ff2875a" containerID="260655afe7adf583fffa2bda35abeb324892b0441f1c5cebb15bada1e91108d4" exitCode=0
Sep 09 21:39:35.732486 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:39:35.732303    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mf5gl"
Sep 09 21:39:35.732486 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:39:35.732346    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mf5gl" event={"ID":"d4f8f35e-811e-4581-bdf6-a5771ff2875a","Type":"ContainerDied","Data":"260655afe7adf583fffa2bda35abeb324892b0441f1c5cebb15bada1e91108d4"}
Sep 09 21:39:35.732486 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:39:35.732456    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mf5gl" event={"ID":"d4f8f35e-811e-4581-bdf6-a5771ff2875a","Type":"ContainerDied","Data":"193ea732636a01f75ab5607862d2152e93b849876cc89e4f87b3b032eb2ed4f2"}
Sep 09 21:39:35.732656 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:39:35.732489    2576 scope.go:117] "RemoveContainer" containerID="260655afe7adf583fffa2bda35abeb324892b0441f1c5cebb15bada1e91108d4"
Sep 09 21:39:35.742646 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:39:35.742623    2576 scope.go:117] "RemoveContainer" containerID="260655afe7adf583fffa2bda35abeb324892b0441f1c5cebb15bada1e91108d4"
Sep 09 21:39:35.742991 ip-10-0-134-72 kubenswrapper[2576]: E0909 21:39:35.742966    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"260655afe7adf583fffa2bda35abeb324892b0441f1c5cebb15bada1e91108d4\": container with ID starting with 260655afe7adf583fffa2bda35abeb324892b0441f1c5cebb15bada1e91108d4 not found: ID does not exist" containerID="260655afe7adf583fffa2bda35abeb324892b0441f1c5cebb15bada1e91108d4"
Sep 09 21:39:35.743079 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:39:35.743002    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"260655afe7adf583fffa2bda35abeb324892b0441f1c5cebb15bada1e91108d4"} err="failed to get container status \"260655afe7adf583fffa2bda35abeb324892b0441f1c5cebb15bada1e91108d4\": rpc error: code = NotFound desc = could not find container \"260655afe7adf583fffa2bda35abeb324892b0441f1c5cebb15bada1e91108d4\": container with ID starting with 260655afe7adf583fffa2bda35abeb324892b0441f1c5cebb15bada1e91108d4 not found: ID does not exist"
Sep 09 21:39:35.760861 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:39:35.760834    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mf5gl"]
Sep 09 21:39:35.774858 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:39:35.774829    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mf5gl"]
Sep 09 21:39:37.408983 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:39:37.408950    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d4f8f35e-811e-4581-bdf6-a5771ff2875a" path="/var/lib/kubelet/pods/d4f8f35e-811e-4581-bdf6-a5771ff2875a/volumes"
Sep 09 21:39:39.888631 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:39:39.888602    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:39:40.007353 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:39:40.007299    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:39:45.064398 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:39:45.064347    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:39:56.749736 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:39:56.749703    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:39:56.970736 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:39:56.970694    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schefkgdl"]
Sep 09 21:39:56.971048 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:39:56.971033    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4f8f35e-811e-4581-bdf6-a5771ff2875a" containerName="main"
Sep 09 21:39:56.971048 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:39:56.971048    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4f8f35e-811e-4581-bdf6-a5771ff2875a" containerName="main"
Sep 09 21:39:56.971203 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:39:56.971106    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4f8f35e-811e-4581-bdf6-a5771ff2875a" containerName="main"
Sep 09 21:39:56.973734 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:39:56.973715    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schefkgdl"
Sep 09 21:39:56.977579 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:39:56.977555    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"kube-root-ca.crt\""
Sep 09 21:39:56.977677 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:39:56.977557    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"llmisva690bbc929faec8bc98c767f16c003c1-kserve-self-signed-certs\""
Sep 09 21:39:56.977677 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:39:56.977654    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"llmisvc-model-fb-opt-125m-route-87882a8e-epp-sa-dockercfg-sdn5m\""
Sep 09 21:39:56.977754 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:39:56.977563    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"openshift-service-ca.crt\""
Sep 09 21:39:56.977978 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:39:56.977947    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"default-dockercfg-6s7bj\""
Sep 09 21:39:56.988881 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:39:56.988858    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schefkgdl"]
Sep 09 21:39:57.017545 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:39:57.017481    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sn2rp\" (UniqueName: \"kubernetes.io/projected/44ae4d85-d50f-4b0d-99a5-8a9184721611-kube-api-access-sn2rp\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schefkgdl\" (UID: \"44ae4d85-d50f-4b0d-99a5-8a9184721611\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schefkgdl"
Sep 09 21:39:57.017667 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:39:57.017589    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44ae4d85-d50f-4b0d-99a5-8a9184721611-tls-certs\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schefkgdl\" (UID: \"44ae4d85-d50f-4b0d-99a5-8a9184721611\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schefkgdl"
Sep 09 21:39:57.020884 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:39:57.020862    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:39:57.118707 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:39:57.118680    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44ae4d85-d50f-4b0d-99a5-8a9184721611-tls-certs\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schefkgdl\" (UID: \"44ae4d85-d50f-4b0d-99a5-8a9184721611\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schefkgdl"
Sep 09 21:39:57.118862 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:39:57.118717    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sn2rp\" (UniqueName: \"kubernetes.io/projected/44ae4d85-d50f-4b0d-99a5-8a9184721611-kube-api-access-sn2rp\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schefkgdl\" (UID: \"44ae4d85-d50f-4b0d-99a5-8a9184721611\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schefkgdl"
Sep 09 21:39:57.121261 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:39:57.121232    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44ae4d85-d50f-4b0d-99a5-8a9184721611-tls-certs\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schefkgdl\" (UID: \"44ae4d85-d50f-4b0d-99a5-8a9184721611\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schefkgdl"
Sep 09 21:39:57.128355 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:39:57.128320    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sn2rp\" (UniqueName: \"kubernetes.io/projected/44ae4d85-d50f-4b0d-99a5-8a9184721611-kube-api-access-sn2rp\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schefkgdl\" (UID: \"44ae4d85-d50f-4b0d-99a5-8a9184721611\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schefkgdl"
Sep 09 21:39:57.283519 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:39:57.283432    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schefkgdl"
Sep 09 21:39:57.567181 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:39:57.567157    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schefkgdl"]
Sep 09 21:39:57.568891 ip-10-0-134-72 kubenswrapper[2576]: W0909 21:39:57.568859    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44ae4d85_d50f_4b0d_99a5_8a9184721611.slice/crio-da8a3dc63b630cfe601c2e29a848b3a96af2c7cb08a909715a68fb793fc8a2f8 WatchSource:0}: Error finding container da8a3dc63b630cfe601c2e29a848b3a96af2c7cb08a909715a68fb793fc8a2f8: Status 404 returned error can't find the container with id da8a3dc63b630cfe601c2e29a848b3a96af2c7cb08a909715a68fb793fc8a2f8
Sep 09 21:39:57.846176 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:39:57.846147    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schefkgdl" event={"ID":"44ae4d85-d50f-4b0d-99a5-8a9184721611","Type":"ContainerStarted","Data":"97ea42af4318fa1bc821020ae5ad910225968723f79d56129e3d2e2d7f8c75b1"}
Sep 09 21:39:57.846607 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:39:57.846201    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schefkgdl" event={"ID":"44ae4d85-d50f-4b0d-99a5-8a9184721611","Type":"ContainerStarted","Data":"da8a3dc63b630cfe601c2e29a848b3a96af2c7cb08a909715a68fb793fc8a2f8"}
Sep 09 21:39:57.846607 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:39:57.846226    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schefkgdl"
Sep 09 21:39:57.877648 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:39:57.877603    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schefkgdl" podStartSLOduration=1.877587763 podStartE2EDuration="1.877587763s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:39:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:39:57.875742251 +0000 UTC m=+2840.989428133" watchObservedRunningTime="2026-09-09 21:39:57.877587763 +0000 UTC m=+2840.991273619"
Sep 09 21:40:06.044430 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:40:06.044395    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:40:06.794217 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:40:06.794179    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:40:08.836703 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:40:08.836668    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:40:14.721700 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:40:14.721663    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:40:15.193953 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:40:15.193914    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:40:16.014775 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:40:16.014734    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:40:16.111563 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:40:16.111533    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:40:16.208763 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:40:16.208725    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:40:27.974904 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:40:27.974865    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:40:28.853180 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:40:28.853110    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schefkgdl"
Sep 09 21:40:31.998128 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:40:31.998083    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:40:32.097917 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:40:32.097875    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:41:03.866409 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:41:03.866348    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969gl4vj"]
Sep 09 21:41:03.871049 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:41:03.871013    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:41:03.876889 ip-10-0-134-72 kubenswrapper[2576]: E0909 21:41:03.876857    2576 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 09 21:41:03.876889 ip-10-0-134-72 kubenswrapper[2576]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 21:41:03.876889 ip-10-0-134-72 kubenswrapper[2576]:         , stdout: , stderr: , exit code -1
Sep 09 21:41:03.876889 ip-10-0-134-72 kubenswrapper[2576]:  > containerID="473bec5a9dfc001b870684cd65f0b89df3246cd15d9e4226a4a8fb8f627533e6" cmd=["/bin/sleep","15"]
Sep 09 21:41:03.877096 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:41:03.876905    2576 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969gl4vj" message="" err=<
Sep 09 21:41:03.877096 ip-10-0-134-72 kubenswrapper[2576]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 21:41:03.877096 ip-10-0-134-72 kubenswrapper[2576]:         , stdout: , stderr: , exit code -1
Sep 09 21:41:03.877096 ip-10-0-134-72 kubenswrapper[2576]:  >
Sep 09 21:41:03.877096 ip-10-0-134-72 kubenswrapper[2576]: E0909 21:41:03.876918    2576 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 09 21:41:03.877096 ip-10-0-134-72 kubenswrapper[2576]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 21:41:03.877096 ip-10-0-134-72 kubenswrapper[2576]:         , stdout: , stderr: , exit code -1
Sep 09 21:41:03.877096 ip-10-0-134-72 kubenswrapper[2576]:  > pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969gl4vj" podUID="fddf9260-84a7-4724-a9e7-1cec80caeb31" containerName="main" containerID="cri-o://473bec5a9dfc001b870684cd65f0b89df3246cd15d9e4226a4a8fb8f627533e6"
Sep 09 21:41:03.877096 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:41:03.876960    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969gl4vj" podUID="fddf9260-84a7-4724-a9e7-1cec80caeb31" containerName="main" containerID="cri-o://473bec5a9dfc001b870684cd65f0b89df3246cd15d9e4226a4a8fb8f627533e6" gracePeriod=60
Sep 09 21:41:04.186804 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:41:04.186773    2576 generic.go:356] "Generic (PLEG): container finished" podID="fddf9260-84a7-4724-a9e7-1cec80caeb31" containerID="473bec5a9dfc001b870684cd65f0b89df3246cd15d9e4226a4a8fb8f627533e6" exitCode=0
Sep 09 21:41:04.186968 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:41:04.186818    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969gl4vj" event={"ID":"fddf9260-84a7-4724-a9e7-1cec80caeb31","Type":"ContainerDied","Data":"473bec5a9dfc001b870684cd65f0b89df3246cd15d9e4226a4a8fb8f627533e6"}
Sep 09 21:41:04.228548 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:41:04.228518    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969gl4vj"
Sep 09 21:41:04.301235 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:41:04.301198    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vnp4x\" (UniqueName: \"kubernetes.io/projected/fddf9260-84a7-4724-a9e7-1cec80caeb31-kube-api-access-vnp4x\") pod \"fddf9260-84a7-4724-a9e7-1cec80caeb31\" (UID: \"fddf9260-84a7-4724-a9e7-1cec80caeb31\") "
Sep 09 21:41:04.301235 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:41:04.301242    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fddf9260-84a7-4724-a9e7-1cec80caeb31-dshm\") pod \"fddf9260-84a7-4724-a9e7-1cec80caeb31\" (UID: \"fddf9260-84a7-4724-a9e7-1cec80caeb31\") "
Sep 09 21:41:04.301508 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:41:04.301278    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fddf9260-84a7-4724-a9e7-1cec80caeb31-model-cache\") pod \"fddf9260-84a7-4724-a9e7-1cec80caeb31\" (UID: \"fddf9260-84a7-4724-a9e7-1cec80caeb31\") "
Sep 09 21:41:04.301508 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:41:04.301293    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fddf9260-84a7-4724-a9e7-1cec80caeb31-tmp-dir\") pod \"fddf9260-84a7-4724-a9e7-1cec80caeb31\" (UID: \"fddf9260-84a7-4724-a9e7-1cec80caeb31\") "
Sep 09 21:41:04.301508 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:41:04.301345    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fddf9260-84a7-4724-a9e7-1cec80caeb31-home\") pod \"fddf9260-84a7-4724-a9e7-1cec80caeb31\" (UID: \"fddf9260-84a7-4724-a9e7-1cec80caeb31\") "
Sep 09 21:41:04.301508 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:41:04.301397    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fddf9260-84a7-4724-a9e7-1cec80caeb31-tls-certs\") pod \"fddf9260-84a7-4724-a9e7-1cec80caeb31\" (UID: \"fddf9260-84a7-4724-a9e7-1cec80caeb31\") "
Sep 09 21:41:04.301722 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:41:04.301551    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fddf9260-84a7-4724-a9e7-1cec80caeb31-model-cache" (OuterVolumeSpecName: "model-cache") pod "fddf9260-84a7-4724-a9e7-1cec80caeb31" (UID: "fddf9260-84a7-4724-a9e7-1cec80caeb31"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:41:04.301722 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:41:04.301588    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fddf9260-84a7-4724-a9e7-1cec80caeb31-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fddf9260-84a7-4724-a9e7-1cec80caeb31" (UID: "fddf9260-84a7-4724-a9e7-1cec80caeb31"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:41:04.301722 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:41:04.301660    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fddf9260-84a7-4724-a9e7-1cec80caeb31-home" (OuterVolumeSpecName: "home") pod "fddf9260-84a7-4724-a9e7-1cec80caeb31" (UID: "fddf9260-84a7-4724-a9e7-1cec80caeb31"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:41:04.301722 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:41:04.301702    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fddf9260-84a7-4724-a9e7-1cec80caeb31-model-cache\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:41:04.301722 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:41:04.301718    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fddf9260-84a7-4724-a9e7-1cec80caeb31-tmp-dir\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:41:04.303821 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:41:04.303789    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fddf9260-84a7-4724-a9e7-1cec80caeb31-kube-api-access-vnp4x" (OuterVolumeSpecName: "kube-api-access-vnp4x") pod "fddf9260-84a7-4724-a9e7-1cec80caeb31" (UID: "fddf9260-84a7-4724-a9e7-1cec80caeb31"). InnerVolumeSpecName "kube-api-access-vnp4x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:41:04.303943 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:41:04.303866    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fddf9260-84a7-4724-a9e7-1cec80caeb31-dshm" (OuterVolumeSpecName: "dshm") pod "fddf9260-84a7-4724-a9e7-1cec80caeb31" (UID: "fddf9260-84a7-4724-a9e7-1cec80caeb31"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:41:04.303943 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:41:04.303872    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fddf9260-84a7-4724-a9e7-1cec80caeb31-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fddf9260-84a7-4724-a9e7-1cec80caeb31" (UID: "fddf9260-84a7-4724-a9e7-1cec80caeb31"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:41:04.402223 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:41:04.402145    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vnp4x\" (UniqueName: \"kubernetes.io/projected/fddf9260-84a7-4724-a9e7-1cec80caeb31-kube-api-access-vnp4x\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:41:04.402223 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:41:04.402171    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fddf9260-84a7-4724-a9e7-1cec80caeb31-dshm\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:41:04.402223 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:41:04.402181    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fddf9260-84a7-4724-a9e7-1cec80caeb31-home\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:41:04.402223 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:41:04.402189    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fddf9260-84a7-4724-a9e7-1cec80caeb31-tls-certs\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:41:05.192929 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:41:05.192894    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969gl4vj" event={"ID":"fddf9260-84a7-4724-a9e7-1cec80caeb31","Type":"ContainerDied","Data":"2eee30add80b11ca1c91fc9a77b9e2bd88d722e273dd2c5e77359006d67071d9"}
Sep 09 21:41:05.193350 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:41:05.192977    2576 scope.go:117] "RemoveContainer" containerID="473bec5a9dfc001b870684cd65f0b89df3246cd15d9e4226a4a8fb8f627533e6"
Sep 09 21:41:05.193350 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:41:05.192910    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969gl4vj"
Sep 09 21:41:05.214545 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:41:05.214516    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969gl4vj"]
Sep 09 21:41:05.218460 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:41:05.218438    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969gl4vj"]
Sep 09 21:41:05.408980 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:41:05.408946    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fddf9260-84a7-4724-a9e7-1cec80caeb31" path="/var/lib/kubelet/pods/fddf9260-84a7-4724-a9e7-1cec80caeb31/volumes"
Sep 09 21:41:05.664191 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:41:05.664159    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:41:47.616268 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:41:47.616227    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:41:53.941930 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:41:53.941890    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:41:54.053394 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:41:54.053338    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:41:59.298621 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:41:59.298533    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-bd4b9f94-6dggq"]
Sep 09 21:41:59.299079 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:41:59.298890    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fddf9260-84a7-4724-a9e7-1cec80caeb31" containerName="main"
Sep 09 21:41:59.299079 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:41:59.298901    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="fddf9260-84a7-4724-a9e7-1cec80caeb31" containerName="main"
Sep 09 21:41:59.299079 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:41:59.298962    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="fddf9260-84a7-4724-a9e7-1cec80caeb31" containerName="main"
Sep 09 21:41:59.302071 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:41:59.302051    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-bd4b9f94-6dggq"
Sep 09 21:41:59.304544 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:41:59.304515    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"default-dockercfg-mr546\""
Sep 09 21:41:59.304766 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:41:59.304641    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"llmisv3e414c2ba058a022dfd694dbcbac5b51-kserve-self-signed-certs\""
Sep 09 21:41:59.304766 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:41:59.304680    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"kube-root-ca.crt\""
Sep 09 21:41:59.304927 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:41:59.304805    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"seaweedfs-s3-creds\""
Sep 09 21:41:59.304927 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:41:59.304891    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"openshift-service-ca.crt\""
Sep 09 21:41:59.305038 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:41:59.304927    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"odh-kserve-custom-ca-bundle\""
Sep 09 21:41:59.313538 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:41:59.313518    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-bd4b9f94-6dggq"]
Sep 09 21:41:59.327376 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:41:59.327346    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:41:59.372180 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:41:59.372152    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2c44b2a-8a84-4fde-97eb-ea02793e160b-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-bd4b9f94-6dggq\" (UID: \"a2c44b2a-8a84-4fde-97eb-ea02793e160b\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-bd4b9f94-6dggq"
Sep 09 21:41:59.372311 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:41:59.372195    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2c44b2a-8a84-4fde-97eb-ea02793e160b-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-bd4b9f94-6dggq\" (UID: \"a2c44b2a-8a84-4fde-97eb-ea02793e160b\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-bd4b9f94-6dggq"
Sep 09 21:41:59.372311 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:41:59.372220    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a2c44b2a-8a84-4fde-97eb-ea02793e160b-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-bd4b9f94-6dggq\" (UID: \"a2c44b2a-8a84-4fde-97eb-ea02793e160b\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-bd4b9f94-6dggq"
Sep 09 21:41:59.372311 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:41:59.372259    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2c44b2a-8a84-4fde-97eb-ea02793e160b-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-bd4b9f94-6dggq\" (UID: \"a2c44b2a-8a84-4fde-97eb-ea02793e160b\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-bd4b9f94-6dggq"
Sep 09 21:41:59.372311 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:41:59.372287    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2c44b2a-8a84-4fde-97eb-ea02793e160b-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-bd4b9f94-6dggq\" (UID: \"a2c44b2a-8a84-4fde-97eb-ea02793e160b\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-bd4b9f94-6dggq"
Sep 09 21:41:59.372311 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:41:59.372306    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a2c44b2a-8a84-4fde-97eb-ea02793e160b-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-bd4b9f94-6dggq\" (UID: \"a2c44b2a-8a84-4fde-97eb-ea02793e160b\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-bd4b9f94-6dggq"
Sep 09 21:41:59.372691 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:41:59.372348    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kt4j6\" (UniqueName: \"kubernetes.io/projected/a2c44b2a-8a84-4fde-97eb-ea02793e160b-kube-api-access-kt4j6\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-bd4b9f94-6dggq\" (UID: \"a2c44b2a-8a84-4fde-97eb-ea02793e160b\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-bd4b9f94-6dggq"
Sep 09 21:41:59.372691 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:41:59.372438    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2c44b2a-8a84-4fde-97eb-ea02793e160b-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-bd4b9f94-6dggq\" (UID: \"a2c44b2a-8a84-4fde-97eb-ea02793e160b\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-bd4b9f94-6dggq"
Sep 09 21:41:59.472894 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:41:59.472854    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2c44b2a-8a84-4fde-97eb-ea02793e160b-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-bd4b9f94-6dggq\" (UID: \"a2c44b2a-8a84-4fde-97eb-ea02793e160b\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-bd4b9f94-6dggq"
Sep 09 21:41:59.472894 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:41:59.472895    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a2c44b2a-8a84-4fde-97eb-ea02793e160b-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-bd4b9f94-6dggq\" (UID: \"a2c44b2a-8a84-4fde-97eb-ea02793e160b\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-bd4b9f94-6dggq"
Sep 09 21:41:59.473143 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:41:59.472912    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kt4j6\" (UniqueName: \"kubernetes.io/projected/a2c44b2a-8a84-4fde-97eb-ea02793e160b-kube-api-access-kt4j6\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-bd4b9f94-6dggq\" (UID: \"a2c44b2a-8a84-4fde-97eb-ea02793e160b\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-bd4b9f94-6dggq"
Sep 09 21:41:59.473143 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:41:59.473035    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2c44b2a-8a84-4fde-97eb-ea02793e160b-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-bd4b9f94-6dggq\" (UID: \"a2c44b2a-8a84-4fde-97eb-ea02793e160b\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-bd4b9f94-6dggq"
Sep 09 21:41:59.473143 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:41:59.473091    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2c44b2a-8a84-4fde-97eb-ea02793e160b-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-bd4b9f94-6dggq\" (UID: \"a2c44b2a-8a84-4fde-97eb-ea02793e160b\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-bd4b9f94-6dggq"
Sep 09 21:41:59.473325 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:41:59.473187    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2c44b2a-8a84-4fde-97eb-ea02793e160b-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-bd4b9f94-6dggq\" (UID: \"a2c44b2a-8a84-4fde-97eb-ea02793e160b\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-bd4b9f94-6dggq"
Sep 09 21:41:59.473325 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:41:59.473219    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a2c44b2a-8a84-4fde-97eb-ea02793e160b-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-bd4b9f94-6dggq\" (UID: \"a2c44b2a-8a84-4fde-97eb-ea02793e160b\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-bd4b9f94-6dggq"
Sep 09 21:41:59.473325 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:41:59.473252    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2c44b2a-8a84-4fde-97eb-ea02793e160b-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-bd4b9f94-6dggq\" (UID: \"a2c44b2a-8a84-4fde-97eb-ea02793e160b\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-bd4b9f94-6dggq"
Sep 09 21:41:59.473582 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:41:59.473560    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2c44b2a-8a84-4fde-97eb-ea02793e160b-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-bd4b9f94-6dggq\" (UID: \"a2c44b2a-8a84-4fde-97eb-ea02793e160b\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-bd4b9f94-6dggq"
Sep 09 21:41:59.473647 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:41:59.473611    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2c44b2a-8a84-4fde-97eb-ea02793e160b-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-bd4b9f94-6dggq\" (UID: \"a2c44b2a-8a84-4fde-97eb-ea02793e160b\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-bd4b9f94-6dggq"
Sep 09 21:41:59.473647 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:41:59.473634    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2c44b2a-8a84-4fde-97eb-ea02793e160b-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-bd4b9f94-6dggq\" (UID: \"a2c44b2a-8a84-4fde-97eb-ea02793e160b\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-bd4b9f94-6dggq"
Sep 09 21:41:59.473724 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:41:59.473644    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a2c44b2a-8a84-4fde-97eb-ea02793e160b-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-bd4b9f94-6dggq\" (UID: \"a2c44b2a-8a84-4fde-97eb-ea02793e160b\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-bd4b9f94-6dggq"
Sep 09 21:41:59.473724 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:41:59.473681    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a2c44b2a-8a84-4fde-97eb-ea02793e160b-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-bd4b9f94-6dggq\" (UID: \"a2c44b2a-8a84-4fde-97eb-ea02793e160b\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-bd4b9f94-6dggq"
Sep 09 21:41:59.475951 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:41:59.475931    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2c44b2a-8a84-4fde-97eb-ea02793e160b-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-bd4b9f94-6dggq\" (UID: \"a2c44b2a-8a84-4fde-97eb-ea02793e160b\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-bd4b9f94-6dggq"
Sep 09 21:41:59.476323 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:41:59.476302    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2c44b2a-8a84-4fde-97eb-ea02793e160b-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-bd4b9f94-6dggq\" (UID: \"a2c44b2a-8a84-4fde-97eb-ea02793e160b\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-bd4b9f94-6dggq"
Sep 09 21:41:59.481042 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:41:59.481021    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kt4j6\" (UniqueName: \"kubernetes.io/projected/a2c44b2a-8a84-4fde-97eb-ea02793e160b-kube-api-access-kt4j6\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-bd4b9f94-6dggq\" (UID: \"a2c44b2a-8a84-4fde-97eb-ea02793e160b\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-bd4b9f94-6dggq"
Sep 09 21:41:59.613049 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:41:59.613013    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-bd4b9f94-6dggq"
Sep 09 21:41:59.621785 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:41:59.621750    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:41:59.889493 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:41:59.889464    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-bd4b9f94-6dggq"]
Sep 09 21:41:59.891084 ip-10-0-134-72 kubenswrapper[2576]: W0909 21:41:59.891059    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2c44b2a_8a84_4fde_97eb_ea02793e160b.slice/crio-1f605781fadb678bb81c784bc326bc92f6f757a488c1a1644a58a755e24621a5 WatchSource:0}: Error finding container 1f605781fadb678bb81c784bc326bc92f6f757a488c1a1644a58a755e24621a5: Status 404 returned error can't find the container with id 1f605781fadb678bb81c784bc326bc92f6f757a488c1a1644a58a755e24621a5
Sep 09 21:42:00.466192 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:42:00.466156    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-bd4b9f94-6dggq" event={"ID":"a2c44b2a-8a84-4fde-97eb-ea02793e160b","Type":"ContainerStarted","Data":"8399fca39a67153cf5f427c9844fd37679846f8356d96c9d89731d96a0c664b4"}
Sep 09 21:42:00.466575 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:42:00.466222    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-bd4b9f94-6dggq" event={"ID":"a2c44b2a-8a84-4fde-97eb-ea02793e160b","Type":"ContainerStarted","Data":"1f605781fadb678bb81c784bc326bc92f6f757a488c1a1644a58a755e24621a5"}
Sep 09 21:42:01.557600 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:42:01.557562    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:42:03.485786 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:42:03.485753    2576 generic.go:356] "Generic (PLEG): container finished" podID="a2c44b2a-8a84-4fde-97eb-ea02793e160b" containerID="8399fca39a67153cf5f427c9844fd37679846f8356d96c9d89731d96a0c664b4" exitCode=0
Sep 09 21:42:03.486164 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:42:03.485814    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-bd4b9f94-6dggq" event={"ID":"a2c44b2a-8a84-4fde-97eb-ea02793e160b","Type":"ContainerDied","Data":"8399fca39a67153cf5f427c9844fd37679846f8356d96c9d89731d96a0c664b4"}
Sep 09 21:42:04.493108 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:42:04.493071    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-bd4b9f94-6dggq" event={"ID":"a2c44b2a-8a84-4fde-97eb-ea02793e160b","Type":"ContainerStarted","Data":"64d657695d1dad6c306a5957d68fe12fc48e8919f051572956c5573362401491"}
Sep 09 21:42:04.517455 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:42:04.517404    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-bd4b9f94-6dggq" podStartSLOduration=2.660301258 podStartE2EDuration="5.51738979s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.857088532s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:41:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:42:04.514731417 +0000 UTC m=+2967.628417288" watchObservedRunningTime="2026-09-09 21:42:04.51738979 +0000 UTC m=+2967.631075641"
Sep 09 21:42:04.614132 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:42:04.614094    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-bd4b9f94-6dggq"
Sep 09 21:42:07.137032 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:42:07.136988    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:42:07.199499 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:42:07.199466    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:42:09.613939 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:42:09.613902    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-bd4b9f94-6dggq"
Sep 09 21:42:09.615730 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:42:09.615681    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-bd4b9f94-6dggq" podUID="a2c44b2a-8a84-4fde-97eb-ea02793e160b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": dial tcp 10.132.0.74:8000: connect: connection refused"
Sep 09 21:42:13.590164 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:42:13.590126    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:42:15.391021 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:42:15.390982    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schefkgdl"]
Sep 09 21:42:15.400930 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:42:15.400896    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:42:15.463380 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:42:15.463316    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:42:18.853343 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:42:18.853303    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schefkgdl" podUID="44ae4d85-d50f-4b0d-99a5-8a9184721611" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:42:19.614217 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:42:19.614159    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-bd4b9f94-6dggq" podUID="a2c44b2a-8a84-4fde-97eb-ea02793e160b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": dial tcp 10.132.0.74:8000: connect: connection refused"
Sep 09 21:42:28.853068 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:42:28.853029    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schefkgdl" podUID="44ae4d85-d50f-4b0d-99a5-8a9184721611" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:42:29.614196 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:42:29.614157    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-bd4b9f94-6dggq" podUID="a2c44b2a-8a84-4fde-97eb-ea02793e160b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": dial tcp 10.132.0.74:8000: connect: connection refused"
Sep 09 21:42:30.415685 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:42:30.415636    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schefkgdl" podUID="44ae4d85-d50f-4b0d-99a5-8a9184721611" containerName="main" containerID="cri-o://97ea42af4318fa1bc821020ae5ad910225968723f79d56129e3d2e2d7f8c75b1" gracePeriod=45
Sep 09 21:42:38.853746 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:42:38.853707    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schefkgdl" podUID="44ae4d85-d50f-4b0d-99a5-8a9184721611" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:42:38.854110 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:42:38.853815    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schefkgdl"
Sep 09 21:42:39.614037 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:42:39.613990    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-bd4b9f94-6dggq" podUID="a2c44b2a-8a84-4fde-97eb-ea02793e160b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": dial tcp 10.132.0.74:8000: connect: connection refused"
Sep 09 21:42:48.853320 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:42:48.853269    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schefkgdl" podUID="44ae4d85-d50f-4b0d-99a5-8a9184721611" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:42:49.614412 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:42:49.614346    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-bd4b9f94-6dggq" podUID="a2c44b2a-8a84-4fde-97eb-ea02793e160b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": dial tcp 10.132.0.74:8000: connect: connection refused"
Sep 09 21:42:58.853570 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:42:58.853530    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schefkgdl" podUID="44ae4d85-d50f-4b0d-99a5-8a9184721611" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:42:59.614116 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:42:59.614074    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-bd4b9f94-6dggq" podUID="a2c44b2a-8a84-4fde-97eb-ea02793e160b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": dial tcp 10.132.0.74:8000: connect: connection refused"
Sep 09 21:43:00.674653 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:43:00.674627    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schefkgdl"
Sep 09 21:43:00.797261 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:43:00.797180    2576 generic.go:356] "Generic (PLEG): container finished" podID="44ae4d85-d50f-4b0d-99a5-8a9184721611" containerID="97ea42af4318fa1bc821020ae5ad910225968723f79d56129e3d2e2d7f8c75b1" exitCode=0
Sep 09 21:43:00.797261 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:43:00.797229    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schefkgdl"
Sep 09 21:43:00.797478 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:43:00.797236    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schefkgdl" event={"ID":"44ae4d85-d50f-4b0d-99a5-8a9184721611","Type":"ContainerDied","Data":"97ea42af4318fa1bc821020ae5ad910225968723f79d56129e3d2e2d7f8c75b1"}
Sep 09 21:43:00.797478 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:43:00.797387    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schefkgdl" event={"ID":"44ae4d85-d50f-4b0d-99a5-8a9184721611","Type":"ContainerDied","Data":"da8a3dc63b630cfe601c2e29a848b3a96af2c7cb08a909715a68fb793fc8a2f8"}
Sep 09 21:43:00.797478 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:43:00.797423    2576 scope.go:117] "RemoveContainer" containerID="97ea42af4318fa1bc821020ae5ad910225968723f79d56129e3d2e2d7f8c75b1"
Sep 09 21:43:00.809180 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:43:00.809155    2576 scope.go:117] "RemoveContainer" containerID="97ea42af4318fa1bc821020ae5ad910225968723f79d56129e3d2e2d7f8c75b1"
Sep 09 21:43:00.809496 ip-10-0-134-72 kubenswrapper[2576]: E0909 21:43:00.809467    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97ea42af4318fa1bc821020ae5ad910225968723f79d56129e3d2e2d7f8c75b1\": container with ID starting with 97ea42af4318fa1bc821020ae5ad910225968723f79d56129e3d2e2d7f8c75b1 not found: ID does not exist" containerID="97ea42af4318fa1bc821020ae5ad910225968723f79d56129e3d2e2d7f8c75b1"
Sep 09 21:43:00.809603 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:43:00.809497    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97ea42af4318fa1bc821020ae5ad910225968723f79d56129e3d2e2d7f8c75b1"} err="failed to get container status \"97ea42af4318fa1bc821020ae5ad910225968723f79d56129e3d2e2d7f8c75b1\": rpc error: code = NotFound desc = could not find container \"97ea42af4318fa1bc821020ae5ad910225968723f79d56129e3d2e2d7f8c75b1\": container with ID starting with 97ea42af4318fa1bc821020ae5ad910225968723f79d56129e3d2e2d7f8c75b1 not found: ID does not exist"
Sep 09 21:43:00.823735 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:43:00.823715    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44ae4d85-d50f-4b0d-99a5-8a9184721611-tls-certs\") pod \"44ae4d85-d50f-4b0d-99a5-8a9184721611\" (UID: \"44ae4d85-d50f-4b0d-99a5-8a9184721611\") "
Sep 09 21:43:00.823858 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:43:00.823816    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sn2rp\" (UniqueName: \"kubernetes.io/projected/44ae4d85-d50f-4b0d-99a5-8a9184721611-kube-api-access-sn2rp\") pod \"44ae4d85-d50f-4b0d-99a5-8a9184721611\" (UID: \"44ae4d85-d50f-4b0d-99a5-8a9184721611\") "
Sep 09 21:43:00.826056 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:43:00.826027    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44ae4d85-d50f-4b0d-99a5-8a9184721611-kube-api-access-sn2rp" (OuterVolumeSpecName: "kube-api-access-sn2rp") pod "44ae4d85-d50f-4b0d-99a5-8a9184721611" (UID: "44ae4d85-d50f-4b0d-99a5-8a9184721611"). InnerVolumeSpecName "kube-api-access-sn2rp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:43:00.826153 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:43:00.826080    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44ae4d85-d50f-4b0d-99a5-8a9184721611-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "44ae4d85-d50f-4b0d-99a5-8a9184721611" (UID: "44ae4d85-d50f-4b0d-99a5-8a9184721611"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:43:00.925480 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:43:00.925441    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sn2rp\" (UniqueName: \"kubernetes.io/projected/44ae4d85-d50f-4b0d-99a5-8a9184721611-kube-api-access-sn2rp\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:43:00.925480 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:43:00.925479    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44ae4d85-d50f-4b0d-99a5-8a9184721611-tls-certs\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:43:01.119561 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:43:01.119530    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schefkgdl"]
Sep 09 21:43:01.121516 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:43:01.121497    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schefkgdl"]
Sep 09 21:43:01.415802 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:43:01.415721    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="44ae4d85-d50f-4b0d-99a5-8a9184721611" path="/var/lib/kubelet/pods/44ae4d85-d50f-4b0d-99a5-8a9184721611/volumes"
Sep 09 21:43:03.714049 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:43:03.714010    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:43:03.808788 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:43:03.808752    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:43:09.614027 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:43:09.613983    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-bd4b9f94-6dggq" podUID="a2c44b2a-8a84-4fde-97eb-ea02793e160b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": dial tcp 10.132.0.74:8000: connect: connection refused"
Sep 09 21:43:10.083387 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:43:10.083265    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:43:15.972268 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:43:15.972211    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:43:16.075193 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:43:16.075155    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:43:16.130078 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:43:16.130050    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:43:19.623634 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:43:19.623603    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-bd4b9f94-6dggq"
Sep 09 21:43:19.631248 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:43:19.631220    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-bd4b9f94-6dggq"
Sep 09 21:43:20.878087 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:43:20.878050    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:43:21.137283 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:43:21.137198    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:43:31.499824 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:43:31.499744    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:43:48.255880 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:43:48.255826    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:43:48.256726 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:43:48.256698    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-bd4b9f94-6dggq"]
Sep 09 21:43:48.314910 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:43:48.314864    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:43:50.970282 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:43:50.970243    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:44:03.285156 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:44:03.285110    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-bd4b9f94-6dggq" podUID="a2c44b2a-8a84-4fde-97eb-ea02793e160b" containerName="main" containerID="cri-o://64d657695d1dad6c306a5957d68fe12fc48e8919f051572956c5573362401491" gracePeriod=45
Sep 09 21:44:03.624891 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:44:03.624838    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-bd4b9f94-6dggq" podUID="a2c44b2a-8a84-4fde-97eb-ea02793e160b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": dial tcp 10.132.0.74:8000: connect: connection refused"
Sep 09 21:44:03.933760 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:44:03.933734    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-bd4b9f94-6dggq"
Sep 09 21:44:03.975275 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:44:03.975250    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2c44b2a-8a84-4fde-97eb-ea02793e160b-model-cache\") pod \"a2c44b2a-8a84-4fde-97eb-ea02793e160b\" (UID: \"a2c44b2a-8a84-4fde-97eb-ea02793e160b\") "
Sep 09 21:44:03.975453 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:44:03.975289    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a2c44b2a-8a84-4fde-97eb-ea02793e160b-kserve-provision-location\") pod \"a2c44b2a-8a84-4fde-97eb-ea02793e160b\" (UID: \"a2c44b2a-8a84-4fde-97eb-ea02793e160b\") "
Sep 09 21:44:03.975453 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:44:03.975316    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kt4j6\" (UniqueName: \"kubernetes.io/projected/a2c44b2a-8a84-4fde-97eb-ea02793e160b-kube-api-access-kt4j6\") pod \"a2c44b2a-8a84-4fde-97eb-ea02793e160b\" (UID: \"a2c44b2a-8a84-4fde-97eb-ea02793e160b\") "
Sep 09 21:44:03.975453 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:44:03.975338    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2c44b2a-8a84-4fde-97eb-ea02793e160b-home\") pod \"a2c44b2a-8a84-4fde-97eb-ea02793e160b\" (UID: \"a2c44b2a-8a84-4fde-97eb-ea02793e160b\") "
Sep 09 21:44:03.975453 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:44:03.975401    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2c44b2a-8a84-4fde-97eb-ea02793e160b-dshm\") pod \"a2c44b2a-8a84-4fde-97eb-ea02793e160b\" (UID: \"a2c44b2a-8a84-4fde-97eb-ea02793e160b\") "
Sep 09 21:44:03.975453 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:44:03.975431    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2c44b2a-8a84-4fde-97eb-ea02793e160b-tls-certs\") pod \"a2c44b2a-8a84-4fde-97eb-ea02793e160b\" (UID: \"a2c44b2a-8a84-4fde-97eb-ea02793e160b\") "
Sep 09 21:44:03.975745 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:44:03.975471    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a2c44b2a-8a84-4fde-97eb-ea02793e160b-cabundle-cert\") pod \"a2c44b2a-8a84-4fde-97eb-ea02793e160b\" (UID: \"a2c44b2a-8a84-4fde-97eb-ea02793e160b\") "
Sep 09 21:44:03.975745 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:44:03.975528    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2c44b2a-8a84-4fde-97eb-ea02793e160b-tmp-dir\") pod \"a2c44b2a-8a84-4fde-97eb-ea02793e160b\" (UID: \"a2c44b2a-8a84-4fde-97eb-ea02793e160b\") "
Sep 09 21:44:03.975745 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:44:03.975588    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2c44b2a-8a84-4fde-97eb-ea02793e160b-model-cache" (OuterVolumeSpecName: "model-cache") pod "a2c44b2a-8a84-4fde-97eb-ea02793e160b" (UID: "a2c44b2a-8a84-4fde-97eb-ea02793e160b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:44:03.975906 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:44:03.975803    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2c44b2a-8a84-4fde-97eb-ea02793e160b-model-cache\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:44:03.975964 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:44:03.975933    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2c44b2a-8a84-4fde-97eb-ea02793e160b-home" (OuterVolumeSpecName: "home") pod "a2c44b2a-8a84-4fde-97eb-ea02793e160b" (UID: "a2c44b2a-8a84-4fde-97eb-ea02793e160b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:44:03.975964 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:44:03.975938    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2c44b2a-8a84-4fde-97eb-ea02793e160b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a2c44b2a-8a84-4fde-97eb-ea02793e160b" (UID: "a2c44b2a-8a84-4fde-97eb-ea02793e160b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:44:03.977987 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:44:03.977961    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2c44b2a-8a84-4fde-97eb-ea02793e160b-kube-api-access-kt4j6" (OuterVolumeSpecName: "kube-api-access-kt4j6") pod "a2c44b2a-8a84-4fde-97eb-ea02793e160b" (UID: "a2c44b2a-8a84-4fde-97eb-ea02793e160b"). InnerVolumeSpecName "kube-api-access-kt4j6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:44:03.978185 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:44:03.978156    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2c44b2a-8a84-4fde-97eb-ea02793e160b-dshm" (OuterVolumeSpecName: "dshm") pod "a2c44b2a-8a84-4fde-97eb-ea02793e160b" (UID: "a2c44b2a-8a84-4fde-97eb-ea02793e160b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:44:03.978320 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:44:03.978300    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2c44b2a-8a84-4fde-97eb-ea02793e160b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a2c44b2a-8a84-4fde-97eb-ea02793e160b" (UID: "a2c44b2a-8a84-4fde-97eb-ea02793e160b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:44:03.986065 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:44:03.986021    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2c44b2a-8a84-4fde-97eb-ea02793e160b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a2c44b2a-8a84-4fde-97eb-ea02793e160b" (UID: "a2c44b2a-8a84-4fde-97eb-ea02793e160b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:44:03.988654 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:44:03.988634    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2c44b2a-8a84-4fde-97eb-ea02793e160b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a2c44b2a-8a84-4fde-97eb-ea02793e160b" (UID: "a2c44b2a-8a84-4fde-97eb-ea02793e160b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:44:04.077010 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:44:04.076980    2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a2c44b2a-8a84-4fde-97eb-ea02793e160b-cabundle-cert\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:44:04.077137 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:44:04.077023    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2c44b2a-8a84-4fde-97eb-ea02793e160b-tmp-dir\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:44:04.077137 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:44:04.077032    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a2c44b2a-8a84-4fde-97eb-ea02793e160b-kserve-provision-location\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:44:04.077137 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:44:04.077041    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kt4j6\" (UniqueName: \"kubernetes.io/projected/a2c44b2a-8a84-4fde-97eb-ea02793e160b-kube-api-access-kt4j6\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:44:04.077137 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:44:04.077049    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2c44b2a-8a84-4fde-97eb-ea02793e160b-home\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:44:04.077137 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:44:04.077057    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2c44b2a-8a84-4fde-97eb-ea02793e160b-dshm\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:44:04.077137 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:44:04.077065    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2c44b2a-8a84-4fde-97eb-ea02793e160b-tls-certs\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:44:04.157591 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:44:04.157560    2576 generic.go:356] "Generic (PLEG): container finished" podID="a2c44b2a-8a84-4fde-97eb-ea02793e160b" containerID="64d657695d1dad6c306a5957d68fe12fc48e8919f051572956c5573362401491" exitCode=0
Sep 09 21:44:04.157723 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:44:04.157609    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-bd4b9f94-6dggq"
Sep 09 21:44:04.157723 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:44:04.157654    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-bd4b9f94-6dggq" event={"ID":"a2c44b2a-8a84-4fde-97eb-ea02793e160b","Type":"ContainerDied","Data":"64d657695d1dad6c306a5957d68fe12fc48e8919f051572956c5573362401491"}
Sep 09 21:44:04.157723 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:44:04.157716    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-bd4b9f94-6dggq" event={"ID":"a2c44b2a-8a84-4fde-97eb-ea02793e160b","Type":"ContainerDied","Data":"1f605781fadb678bb81c784bc326bc92f6f757a488c1a1644a58a755e24621a5"}
Sep 09 21:44:04.157903 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:44:04.157753    2576 scope.go:117] "RemoveContainer" containerID="64d657695d1dad6c306a5957d68fe12fc48e8919f051572956c5573362401491"
Sep 09 21:44:04.166856 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:44:04.166834    2576 scope.go:117] "RemoveContainer" containerID="8399fca39a67153cf5f427c9844fd37679846f8356d96c9d89731d96a0c664b4"
Sep 09 21:44:04.175028 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:44:04.175013    2576 scope.go:117] "RemoveContainer" containerID="64d657695d1dad6c306a5957d68fe12fc48e8919f051572956c5573362401491"
Sep 09 21:44:04.175290 ip-10-0-134-72 kubenswrapper[2576]: E0909 21:44:04.175269    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64d657695d1dad6c306a5957d68fe12fc48e8919f051572956c5573362401491\": container with ID starting with 64d657695d1dad6c306a5957d68fe12fc48e8919f051572956c5573362401491 not found: ID does not exist" containerID="64d657695d1dad6c306a5957d68fe12fc48e8919f051572956c5573362401491"
Sep 09 21:44:04.175354 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:44:04.175297    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64d657695d1dad6c306a5957d68fe12fc48e8919f051572956c5573362401491"} err="failed to get container status \"64d657695d1dad6c306a5957d68fe12fc48e8919f051572956c5573362401491\": rpc error: code = NotFound desc = could not find container \"64d657695d1dad6c306a5957d68fe12fc48e8919f051572956c5573362401491\": container with ID starting with 64d657695d1dad6c306a5957d68fe12fc48e8919f051572956c5573362401491 not found: ID does not exist"
Sep 09 21:44:04.175354 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:44:04.175322    2576 scope.go:117] "RemoveContainer" containerID="8399fca39a67153cf5f427c9844fd37679846f8356d96c9d89731d96a0c664b4"
Sep 09 21:44:04.175608 ip-10-0-134-72 kubenswrapper[2576]: E0909 21:44:04.175587    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8399fca39a67153cf5f427c9844fd37679846f8356d96c9d89731d96a0c664b4\": container with ID starting with 8399fca39a67153cf5f427c9844fd37679846f8356d96c9d89731d96a0c664b4 not found: ID does not exist" containerID="8399fca39a67153cf5f427c9844fd37679846f8356d96c9d89731d96a0c664b4"
Sep 09 21:44:04.175658 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:44:04.175614    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8399fca39a67153cf5f427c9844fd37679846f8356d96c9d89731d96a0c664b4"} err="failed to get container status \"8399fca39a67153cf5f427c9844fd37679846f8356d96c9d89731d96a0c664b4\": rpc error: code = NotFound desc = could not find container \"8399fca39a67153cf5f427c9844fd37679846f8356d96c9d89731d96a0c664b4\": container with ID starting with 8399fca39a67153cf5f427c9844fd37679846f8356d96c9d89731d96a0c664b4 not found: ID does not exist"
Sep 09 21:44:04.179293 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:44:04.179273    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-bd4b9f94-6dggq"]
Sep 09 21:44:04.182820 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:44:04.182800    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-bd4b9f94-6dggq"]
Sep 09 21:44:05.409054 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:44:05.409020    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a2c44b2a-8a84-4fde-97eb-ea02793e160b" path="/var/lib/kubelet/pods/a2c44b2a-8a84-4fde-97eb-ea02793e160b/volumes"
Sep 09 21:44:38.388870 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:44:38.388822    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:44:38.482644 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:44:38.482603    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:44:44.699277 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:44:44.699241    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:44:50.620426 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:44:50.620393    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:44:50.676152 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:44:50.676118    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:44:58.694060 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:44:58.693990    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-79cdd87d9c-znm6k"]
Sep 09 21:44:58.694445 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:44:58.694305    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44ae4d85-d50f-4b0d-99a5-8a9184721611" containerName="main"
Sep 09 21:44:58.694445 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:44:58.694315    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="44ae4d85-d50f-4b0d-99a5-8a9184721611" containerName="main"
Sep 09 21:44:58.694445 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:44:58.694332    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2c44b2a-8a84-4fde-97eb-ea02793e160b" containerName="storage-initializer"
Sep 09 21:44:58.694445 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:44:58.694338    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2c44b2a-8a84-4fde-97eb-ea02793e160b" containerName="storage-initializer"
Sep 09 21:44:58.694445 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:44:58.694344    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2c44b2a-8a84-4fde-97eb-ea02793e160b" containerName="main"
Sep 09 21:44:58.694445 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:44:58.694350    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2c44b2a-8a84-4fde-97eb-ea02793e160b" containerName="main"
Sep 09 21:44:58.694445 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:44:58.694430    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="44ae4d85-d50f-4b0d-99a5-8a9184721611" containerName="main"
Sep 09 21:44:58.694445 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:44:58.694438    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2c44b2a-8a84-4fde-97eb-ea02793e160b" containerName="main"
Sep 09 21:44:58.697477 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:44:58.697458    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-79cdd87d9c-znm6k"
Sep 09 21:44:58.699726 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:44:58.699703    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"seaweedfs-s3-creds\""
Sep 09 21:44:58.699832 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:44:58.699807    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs\""
Sep 09 21:44:58.699876 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:44:58.699807    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"kube-root-ca.crt\""
Sep 09 21:44:58.700117 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:44:58.700097    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"openshift-service-ca.crt\""
Sep 09 21:44:58.700215 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:44:58.700103    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"odh-kserve-custom-ca-bundle\""
Sep 09 21:44:58.700905 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:44:58.700870    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"default-dockercfg-c4qw2\""
Sep 09 21:44:58.708577 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:44:58.708555    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-79cdd87d9c-znm6k"]
Sep 09 21:44:58.715311 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:44:58.715271    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:44:58.827695 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:44:58.827657    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzbb7\" (UniqueName: \"kubernetes.io/projected/82e89840-e9d1-46b9-b13d-c19c802a0b7b-kube-api-access-dzbb7\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-79cdd87d9c-znm6k\" (UID: \"82e89840-e9d1-46b9-b13d-c19c802a0b7b\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-79cdd87d9c-znm6k"
Sep 09 21:44:58.827868 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:44:58.827705    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82e89840-e9d1-46b9-b13d-c19c802a0b7b-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-79cdd87d9c-znm6k\" (UID: \"82e89840-e9d1-46b9-b13d-c19c802a0b7b\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-79cdd87d9c-znm6k"
Sep 09 21:44:58.827868 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:44:58.827723    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82e89840-e9d1-46b9-b13d-c19c802a0b7b-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-79cdd87d9c-znm6k\" (UID: \"82e89840-e9d1-46b9-b13d-c19c802a0b7b\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-79cdd87d9c-znm6k"
Sep 09 21:44:58.827868 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:44:58.827744    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82e89840-e9d1-46b9-b13d-c19c802a0b7b-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-79cdd87d9c-znm6k\" (UID: \"82e89840-e9d1-46b9-b13d-c19c802a0b7b\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-79cdd87d9c-znm6k"
Sep 09 21:44:58.827868 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:44:58.827826    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82e89840-e9d1-46b9-b13d-c19c802a0b7b-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-79cdd87d9c-znm6k\" (UID: \"82e89840-e9d1-46b9-b13d-c19c802a0b7b\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-79cdd87d9c-znm6k"
Sep 09 21:44:58.828067 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:44:58.827875    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/82e89840-e9d1-46b9-b13d-c19c802a0b7b-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-79cdd87d9c-znm6k\" (UID: \"82e89840-e9d1-46b9-b13d-c19c802a0b7b\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-79cdd87d9c-znm6k"
Sep 09 21:44:58.828067 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:44:58.827893    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82e89840-e9d1-46b9-b13d-c19c802a0b7b-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-79cdd87d9c-znm6k\" (UID: \"82e89840-e9d1-46b9-b13d-c19c802a0b7b\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-79cdd87d9c-znm6k"
Sep 09 21:44:58.828067 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:44:58.827955    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/82e89840-e9d1-46b9-b13d-c19c802a0b7b-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-79cdd87d9c-znm6k\" (UID: \"82e89840-e9d1-46b9-b13d-c19c802a0b7b\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-79cdd87d9c-znm6k"
Sep 09 21:44:58.928803 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:44:58.928770    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/82e89840-e9d1-46b9-b13d-c19c802a0b7b-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-79cdd87d9c-znm6k\" (UID: \"82e89840-e9d1-46b9-b13d-c19c802a0b7b\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-79cdd87d9c-znm6k"
Sep 09 21:44:58.928803 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:44:58.928804    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzbb7\" (UniqueName: \"kubernetes.io/projected/82e89840-e9d1-46b9-b13d-c19c802a0b7b-kube-api-access-dzbb7\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-79cdd87d9c-znm6k\" (UID: \"82e89840-e9d1-46b9-b13d-c19c802a0b7b\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-79cdd87d9c-znm6k"
Sep 09 21:44:58.929045 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:44:58.928833    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82e89840-e9d1-46b9-b13d-c19c802a0b7b-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-79cdd87d9c-znm6k\" (UID: \"82e89840-e9d1-46b9-b13d-c19c802a0b7b\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-79cdd87d9c-znm6k"
Sep 09 21:44:58.929045 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:44:58.928941    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82e89840-e9d1-46b9-b13d-c19c802a0b7b-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-79cdd87d9c-znm6k\" (UID: \"82e89840-e9d1-46b9-b13d-c19c802a0b7b\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-79cdd87d9c-znm6k"
Sep 09 21:44:58.929045 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:44:58.928989    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82e89840-e9d1-46b9-b13d-c19c802a0b7b-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-79cdd87d9c-znm6k\" (UID: \"82e89840-e9d1-46b9-b13d-c19c802a0b7b\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-79cdd87d9c-znm6k"
Sep 09 21:44:58.929045 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:44:58.929041    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82e89840-e9d1-46b9-b13d-c19c802a0b7b-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-79cdd87d9c-znm6k\" (UID: \"82e89840-e9d1-46b9-b13d-c19c802a0b7b\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-79cdd87d9c-znm6k"
Sep 09 21:44:58.929271 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:44:58.929095    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/82e89840-e9d1-46b9-b13d-c19c802a0b7b-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-79cdd87d9c-znm6k\" (UID: \"82e89840-e9d1-46b9-b13d-c19c802a0b7b\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-79cdd87d9c-znm6k"
Sep 09 21:44:58.929271 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:44:58.929123    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82e89840-e9d1-46b9-b13d-c19c802a0b7b-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-79cdd87d9c-znm6k\" (UID: \"82e89840-e9d1-46b9-b13d-c19c802a0b7b\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-79cdd87d9c-znm6k"
Sep 09 21:44:58.929410 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:44:58.929251    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/82e89840-e9d1-46b9-b13d-c19c802a0b7b-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-79cdd87d9c-znm6k\" (UID: \"82e89840-e9d1-46b9-b13d-c19c802a0b7b\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-79cdd87d9c-znm6k"
Sep 09 21:44:58.929410 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:44:58.929351    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82e89840-e9d1-46b9-b13d-c19c802a0b7b-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-79cdd87d9c-znm6k\" (UID: \"82e89840-e9d1-46b9-b13d-c19c802a0b7b\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-79cdd87d9c-znm6k"
Sep 09 21:44:58.929520 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:44:58.929432    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82e89840-e9d1-46b9-b13d-c19c802a0b7b-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-79cdd87d9c-znm6k\" (UID: \"82e89840-e9d1-46b9-b13d-c19c802a0b7b\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-79cdd87d9c-znm6k"
Sep 09 21:44:58.929520 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:44:58.929474    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82e89840-e9d1-46b9-b13d-c19c802a0b7b-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-79cdd87d9c-znm6k\" (UID: \"82e89840-e9d1-46b9-b13d-c19c802a0b7b\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-79cdd87d9c-znm6k"
Sep 09 21:44:58.929807 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:44:58.929740    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/82e89840-e9d1-46b9-b13d-c19c802a0b7b-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-79cdd87d9c-znm6k\" (UID: \"82e89840-e9d1-46b9-b13d-c19c802a0b7b\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-79cdd87d9c-znm6k"
Sep 09 21:44:58.931465 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:44:58.931442    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82e89840-e9d1-46b9-b13d-c19c802a0b7b-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-79cdd87d9c-znm6k\" (UID: \"82e89840-e9d1-46b9-b13d-c19c802a0b7b\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-79cdd87d9c-znm6k"
Sep 09 21:44:58.931854 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:44:58.931831    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82e89840-e9d1-46b9-b13d-c19c802a0b7b-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-79cdd87d9c-znm6k\" (UID: \"82e89840-e9d1-46b9-b13d-c19c802a0b7b\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-79cdd87d9c-znm6k"
Sep 09 21:44:58.936304 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:44:58.936279    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzbb7\" (UniqueName: \"kubernetes.io/projected/82e89840-e9d1-46b9-b13d-c19c802a0b7b-kube-api-access-dzbb7\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-79cdd87d9c-znm6k\" (UID: \"82e89840-e9d1-46b9-b13d-c19c802a0b7b\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-79cdd87d9c-znm6k"
Sep 09 21:44:59.008830 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:44:59.008752    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-79cdd87d9c-znm6k"
Sep 09 21:44:59.017972 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:44:59.017934    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:44:59.291029 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:44:59.291003    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-79cdd87d9c-znm6k"]
Sep 09 21:44:59.291981 ip-10-0-134-72 kubenswrapper[2576]: W0909 21:44:59.291957    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82e89840_e9d1_46b9_b13d_c19c802a0b7b.slice/crio-07111c7f9bf3a126d2ffe6b7b1e2e8e84a4321568fbcdb30db857e8b8454b294 WatchSource:0}: Error finding container 07111c7f9bf3a126d2ffe6b7b1e2e8e84a4321568fbcdb30db857e8b8454b294: Status 404 returned error can't find the container with id 07111c7f9bf3a126d2ffe6b7b1e2e8e84a4321568fbcdb30db857e8b8454b294
Sep 09 21:44:59.426293 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:44:59.426261    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-79cdd87d9c-znm6k" event={"ID":"82e89840-e9d1-46b9-b13d-c19c802a0b7b","Type":"ContainerStarted","Data":"feb7d1c5541051834981177a84b0bad4980f7c2051789c4863ee195c6f7a62c9"}
Sep 09 21:44:59.426494 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:44:59.426307    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-79cdd87d9c-znm6k" event={"ID":"82e89840-e9d1-46b9-b13d-c19c802a0b7b","Type":"ContainerStarted","Data":"07111c7f9bf3a126d2ffe6b7b1e2e8e84a4321568fbcdb30db857e8b8454b294"}
Sep 09 21:45:02.442784 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:45:02.442750    2576 generic.go:356] "Generic (PLEG): container finished" podID="82e89840-e9d1-46b9-b13d-c19c802a0b7b" containerID="feb7d1c5541051834981177a84b0bad4980f7c2051789c4863ee195c6f7a62c9" exitCode=0
Sep 09 21:45:02.443185 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:45:02.442823    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-79cdd87d9c-znm6k" event={"ID":"82e89840-e9d1-46b9-b13d-c19c802a0b7b","Type":"ContainerDied","Data":"feb7d1c5541051834981177a84b0bad4980f7c2051789c4863ee195c6f7a62c9"}
Sep 09 21:45:03.450280 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:45:03.450241    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-79cdd87d9c-znm6k" event={"ID":"82e89840-e9d1-46b9-b13d-c19c802a0b7b","Type":"ContainerStarted","Data":"6ee5e267a0218d2dc139c5f6a49b2f496e24902f829874a6e08a8d139c9c23ea"}
Sep 09 21:45:03.475147 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:45:03.475078    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-79cdd87d9c-znm6k" podStartSLOduration=2.6186171099999997 podStartE2EDuration="5.475057996s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.856440886s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:44:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:45:03.472276651 +0000 UTC m=+3146.585962519" watchObservedRunningTime="2026-09-09 21:45:03.475057996 +0000 UTC m=+3146.588743856"
Sep 09 21:45:04.009457 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:45:04.009422    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-79cdd87d9c-znm6k"
Sep 09 21:45:09.009147 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:45:09.009113    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-79cdd87d9c-znm6k"
Sep 09 21:45:09.010679 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:45:09.010647    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-79cdd87d9c-znm6k" podUID="82e89840-e9d1-46b9-b13d-c19c802a0b7b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/health\": dial tcp 10.132.0.75:8000: connect: connection refused"
Sep 09 21:45:09.691144 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:45:09.691110    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:45:19.009871 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:45:19.009827    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-79cdd87d9c-znm6k" podUID="82e89840-e9d1-46b9-b13d-c19c802a0b7b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/health\": dial tcp 10.132.0.75:8000: connect: connection refused"
Sep 09 21:45:29.010012 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:45:29.009961    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-79cdd87d9c-znm6k" podUID="82e89840-e9d1-46b9-b13d-c19c802a0b7b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/health\": dial tcp 10.132.0.75:8000: connect: connection refused"
Sep 09 21:45:35.714011 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:45:35.713970    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:45:35.769222 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:45:35.769141    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:45:38.929328 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:45:38.929288    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:45:39.009650 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:45:39.009610    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-79cdd87d9c-znm6k" podUID="82e89840-e9d1-46b9-b13d-c19c802a0b7b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/health\": dial tcp 10.132.0.75:8000: connect: connection refused"
Sep 09 21:45:49.009831 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:45:49.009774    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-79cdd87d9c-znm6k" podUID="82e89840-e9d1-46b9-b13d-c19c802a0b7b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/health\": dial tcp 10.132.0.75:8000: connect: connection refused"
Sep 09 21:45:59.010080 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:45:59.010020    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-79cdd87d9c-znm6k" podUID="82e89840-e9d1-46b9-b13d-c19c802a0b7b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/health\": dial tcp 10.132.0.75:8000: connect: connection refused"
Sep 09 21:46:09.009468 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:46:09.009424    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-79cdd87d9c-znm6k" podUID="82e89840-e9d1-46b9-b13d-c19c802a0b7b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/health\": dial tcp 10.132.0.75:8000: connect: connection refused"
Sep 09 21:46:19.019253 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:46:19.019221    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-79cdd87d9c-znm6k"
Sep 09 21:46:19.026792 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:46:19.026767    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-79cdd87d9c-znm6k"
Sep 09 21:46:20.610967 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:46:20.610923    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:46:20.624806 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:46:20.624778    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-79cdd87d9c-znm6k"]
Sep 09 21:46:21.715142 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:46:21.715101    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:46:25.516197 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:46:25.516158    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:46:25.825984 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:46:25.825889    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:46:25.930942 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:46:25.930910    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:46:31.989612 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:46:31.989538    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-557b9dcccf-csrxs"]
Sep 09 21:46:31.991946 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:46:31.991927    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-557b9dcccf-csrxs"
Sep 09 21:46:31.994728 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:46:31.994575    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"odh-kserve-custom-ca-bundle\""
Sep 09 21:46:31.994728 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:46:31.994587    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"custom-route-timeout-test-kserve-self-signed-certs\""
Sep 09 21:46:31.994728 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:46:31.994575    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"seaweedfs-s3-creds\""
Sep 09 21:46:31.994728 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:46:31.994624    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"openshift-service-ca.crt\""
Sep 09 21:46:31.994728 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:46:31.994578    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"default-dockercfg-n2vvp\""
Sep 09 21:46:31.995071 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:46:31.994826    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"kube-root-ca.crt\""
Sep 09 21:46:32.003697 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:46:32.003669    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-557b9dcccf-csrxs"]
Sep 09 21:46:32.011640 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:46:32.011612    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:46:32.019835 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:46:32.019806    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3014647-a76e-4ffc-88f6-27206f6d5088-home\") pod \"custom-route-timeout-test-kserve-557b9dcccf-csrxs\" (UID: \"a3014647-a76e-4ffc-88f6-27206f6d5088\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-557b9dcccf-csrxs"
Sep 09 21:46:32.019932 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:46:32.019859    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2h2k\" (UniqueName: \"kubernetes.io/projected/a3014647-a76e-4ffc-88f6-27206f6d5088-kube-api-access-m2h2k\") pod \"custom-route-timeout-test-kserve-557b9dcccf-csrxs\" (UID: \"a3014647-a76e-4ffc-88f6-27206f6d5088\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-557b9dcccf-csrxs"
Sep 09 21:46:32.019976 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:46:32.019948    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3014647-a76e-4ffc-88f6-27206f6d5088-tmp-dir\") pod \"custom-route-timeout-test-kserve-557b9dcccf-csrxs\" (UID: \"a3014647-a76e-4ffc-88f6-27206f6d5088\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-557b9dcccf-csrxs"
Sep 09 21:46:32.020013 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:46:32.019979    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a3014647-a76e-4ffc-88f6-27206f6d5088-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-557b9dcccf-csrxs\" (UID: \"a3014647-a76e-4ffc-88f6-27206f6d5088\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-557b9dcccf-csrxs"
Sep 09 21:46:32.020053 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:46:32.020012    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3014647-a76e-4ffc-88f6-27206f6d5088-tls-certs\") pod \"custom-route-timeout-test-kserve-557b9dcccf-csrxs\" (UID: \"a3014647-a76e-4ffc-88f6-27206f6d5088\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-557b9dcccf-csrxs"
Sep 09 21:46:32.020087 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:46:32.020055    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3014647-a76e-4ffc-88f6-27206f6d5088-model-cache\") pod \"custom-route-timeout-test-kserve-557b9dcccf-csrxs\" (UID: \"a3014647-a76e-4ffc-88f6-27206f6d5088\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-557b9dcccf-csrxs"
Sep 09 21:46:32.020087 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:46:32.020081    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a3014647-a76e-4ffc-88f6-27206f6d5088-cabundle-cert\") pod \"custom-route-timeout-test-kserve-557b9dcccf-csrxs\" (UID: \"a3014647-a76e-4ffc-88f6-27206f6d5088\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-557b9dcccf-csrxs"
Sep 09 21:46:32.020152 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:46:32.020138    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3014647-a76e-4ffc-88f6-27206f6d5088-dshm\") pod \"custom-route-timeout-test-kserve-557b9dcccf-csrxs\" (UID: \"a3014647-a76e-4ffc-88f6-27206f6d5088\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-557b9dcccf-csrxs"
Sep 09 21:46:32.120937 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:46:32.120909    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3014647-a76e-4ffc-88f6-27206f6d5088-dshm\") pod \"custom-route-timeout-test-kserve-557b9dcccf-csrxs\" (UID: \"a3014647-a76e-4ffc-88f6-27206f6d5088\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-557b9dcccf-csrxs"
Sep 09 21:46:32.121125 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:46:32.120957    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3014647-a76e-4ffc-88f6-27206f6d5088-home\") pod \"custom-route-timeout-test-kserve-557b9dcccf-csrxs\" (UID: \"a3014647-a76e-4ffc-88f6-27206f6d5088\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-557b9dcccf-csrxs"
Sep 09 21:46:32.121125 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:46:32.120994    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2h2k\" (UniqueName: \"kubernetes.io/projected/a3014647-a76e-4ffc-88f6-27206f6d5088-kube-api-access-m2h2k\") pod \"custom-route-timeout-test-kserve-557b9dcccf-csrxs\" (UID: \"a3014647-a76e-4ffc-88f6-27206f6d5088\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-557b9dcccf-csrxs"
Sep 09 21:46:32.121125 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:46:32.121030    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3014647-a76e-4ffc-88f6-27206f6d5088-tmp-dir\") pod \"custom-route-timeout-test-kserve-557b9dcccf-csrxs\" (UID: \"a3014647-a76e-4ffc-88f6-27206f6d5088\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-557b9dcccf-csrxs"
Sep 09 21:46:32.121125 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:46:32.121053    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a3014647-a76e-4ffc-88f6-27206f6d5088-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-557b9dcccf-csrxs\" (UID: \"a3014647-a76e-4ffc-88f6-27206f6d5088\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-557b9dcccf-csrxs"
Sep 09 21:46:32.121125 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:46:32.121082    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3014647-a76e-4ffc-88f6-27206f6d5088-tls-certs\") pod \"custom-route-timeout-test-kserve-557b9dcccf-csrxs\" (UID: \"a3014647-a76e-4ffc-88f6-27206f6d5088\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-557b9dcccf-csrxs"
Sep 09 21:46:32.121125 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:46:32.121111    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3014647-a76e-4ffc-88f6-27206f6d5088-model-cache\") pod \"custom-route-timeout-test-kserve-557b9dcccf-csrxs\" (UID: \"a3014647-a76e-4ffc-88f6-27206f6d5088\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-557b9dcccf-csrxs"
Sep 09 21:46:32.121475 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:46:32.121137    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a3014647-a76e-4ffc-88f6-27206f6d5088-cabundle-cert\") pod \"custom-route-timeout-test-kserve-557b9dcccf-csrxs\" (UID: \"a3014647-a76e-4ffc-88f6-27206f6d5088\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-557b9dcccf-csrxs"
Sep 09 21:46:32.121535 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:46:32.121491    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3014647-a76e-4ffc-88f6-27206f6d5088-home\") pod \"custom-route-timeout-test-kserve-557b9dcccf-csrxs\" (UID: \"a3014647-a76e-4ffc-88f6-27206f6d5088\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-557b9dcccf-csrxs"
Sep 09 21:46:32.121598 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:46:32.121549    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3014647-a76e-4ffc-88f6-27206f6d5088-tmp-dir\") pod \"custom-route-timeout-test-kserve-557b9dcccf-csrxs\" (UID: \"a3014647-a76e-4ffc-88f6-27206f6d5088\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-557b9dcccf-csrxs"
Sep 09 21:46:32.121641 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:46:32.121592    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3014647-a76e-4ffc-88f6-27206f6d5088-model-cache\") pod \"custom-route-timeout-test-kserve-557b9dcccf-csrxs\" (UID: \"a3014647-a76e-4ffc-88f6-27206f6d5088\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-557b9dcccf-csrxs"
Sep 09 21:46:32.121691 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:46:32.121641    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a3014647-a76e-4ffc-88f6-27206f6d5088-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-557b9dcccf-csrxs\" (UID: \"a3014647-a76e-4ffc-88f6-27206f6d5088\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-557b9dcccf-csrxs"
Sep 09 21:46:32.122135 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:46:32.122114    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a3014647-a76e-4ffc-88f6-27206f6d5088-cabundle-cert\") pod \"custom-route-timeout-test-kserve-557b9dcccf-csrxs\" (UID: \"a3014647-a76e-4ffc-88f6-27206f6d5088\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-557b9dcccf-csrxs"
Sep 09 21:46:32.123600 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:46:32.123579    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3014647-a76e-4ffc-88f6-27206f6d5088-dshm\") pod \"custom-route-timeout-test-kserve-557b9dcccf-csrxs\" (UID: \"a3014647-a76e-4ffc-88f6-27206f6d5088\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-557b9dcccf-csrxs"
Sep 09 21:46:32.123877 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:46:32.123856    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3014647-a76e-4ffc-88f6-27206f6d5088-tls-certs\") pod \"custom-route-timeout-test-kserve-557b9dcccf-csrxs\" (UID: \"a3014647-a76e-4ffc-88f6-27206f6d5088\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-557b9dcccf-csrxs"
Sep 09 21:46:32.128717 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:46:32.128692    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2h2k\" (UniqueName: \"kubernetes.io/projected/a3014647-a76e-4ffc-88f6-27206f6d5088-kube-api-access-m2h2k\") pod \"custom-route-timeout-test-kserve-557b9dcccf-csrxs\" (UID: \"a3014647-a76e-4ffc-88f6-27206f6d5088\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-557b9dcccf-csrxs"
Sep 09 21:46:32.208754 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:46:32.208706    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:46:32.302805 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:46:32.302719    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-557b9dcccf-csrxs"
Sep 09 21:46:32.443229 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:46:32.443190    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:46:32.578917 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:46:32.578889    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-557b9dcccf-csrxs"]
Sep 09 21:46:32.580908 ip-10-0-134-72 kubenswrapper[2576]: W0909 21:46:32.580878    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3014647_a76e_4ffc_88f6_27206f6d5088.slice/crio-77178d015d38763a02db829d60f4ea887a21032a67f7881de5391f5b0c04fa75 WatchSource:0}: Error finding container 77178d015d38763a02db829d60f4ea887a21032a67f7881de5391f5b0c04fa75: Status 404 returned error can't find the container with id 77178d015d38763a02db829d60f4ea887a21032a67f7881de5391f5b0c04fa75
Sep 09 21:46:32.919330 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:46:32.919290    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-557b9dcccf-csrxs" event={"ID":"a3014647-a76e-4ffc-88f6-27206f6d5088","Type":"ContainerStarted","Data":"89353801495d897348aa12f447399c0d6cc335baa15c96263067f0cee2bdb450"}
Sep 09 21:46:32.919531 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:46:32.919384    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-557b9dcccf-csrxs" event={"ID":"a3014647-a76e-4ffc-88f6-27206f6d5088","Type":"ContainerStarted","Data":"77178d015d38763a02db829d60f4ea887a21032a67f7881de5391f5b0c04fa75"}
Sep 09 21:46:34.930791 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:46:34.930753    2576 generic.go:356] "Generic (PLEG): container finished" podID="a3014647-a76e-4ffc-88f6-27206f6d5088" containerID="89353801495d897348aa12f447399c0d6cc335baa15c96263067f0cee2bdb450" exitCode=0
Sep 09 21:46:34.931168 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:46:34.930823    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-557b9dcccf-csrxs" event={"ID":"a3014647-a76e-4ffc-88f6-27206f6d5088","Type":"ContainerDied","Data":"89353801495d897348aa12f447399c0d6cc335baa15c96263067f0cee2bdb450"}
Sep 09 21:46:35.880769 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:46:35.880724    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-79cdd87d9c-znm6k" podUID="82e89840-e9d1-46b9-b13d-c19c802a0b7b" containerName="main" containerID="cri-o://6ee5e267a0218d2dc139c5f6a49b2f496e24902f829874a6e08a8d139c9c23ea" gracePeriod=45
Sep 09 21:46:35.939113 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:46:35.939077    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-557b9dcccf-csrxs" event={"ID":"a3014647-a76e-4ffc-88f6-27206f6d5088","Type":"ContainerStarted","Data":"501d770895d5807b95183744f8d5ecc83d89d2bc6450e3c10d96cb925e44f8dd"}
Sep 09 21:46:35.964756 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:46:35.964689    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-557b9dcccf-csrxs" podStartSLOduration=2.8544509 podStartE2EDuration="4.964666673s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.110215773s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:46:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:46:35.95983908 +0000 UTC m=+3239.073524938" watchObservedRunningTime="2026-09-09 21:46:35.964666673 +0000 UTC m=+3239.078352539"
Sep 09 21:46:36.019828 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:46:36.019779    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-79cdd87d9c-znm6k" podUID="82e89840-e9d1-46b9-b13d-c19c802a0b7b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/health\": dial tcp 10.132.0.75:8000: connect: connection refused"
Sep 09 21:46:36.303736 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:46:36.303701    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-557b9dcccf-csrxs"
Sep 09 21:46:36.469554 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:46:36.469527    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-79cdd87d9c-znm6k"
Sep 09 21:46:36.553266 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:46:36.553190    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82e89840-e9d1-46b9-b13d-c19c802a0b7b-dshm\") pod \"82e89840-e9d1-46b9-b13d-c19c802a0b7b\" (UID: \"82e89840-e9d1-46b9-b13d-c19c802a0b7b\") "
Sep 09 21:46:36.553266 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:46:36.553227    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/82e89840-e9d1-46b9-b13d-c19c802a0b7b-kserve-provision-location\") pod \"82e89840-e9d1-46b9-b13d-c19c802a0b7b\" (UID: \"82e89840-e9d1-46b9-b13d-c19c802a0b7b\") "
Sep 09 21:46:36.553266 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:46:36.553247    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dzbb7\" (UniqueName: \"kubernetes.io/projected/82e89840-e9d1-46b9-b13d-c19c802a0b7b-kube-api-access-dzbb7\") pod \"82e89840-e9d1-46b9-b13d-c19c802a0b7b\" (UID: \"82e89840-e9d1-46b9-b13d-c19c802a0b7b\") "
Sep 09 21:46:36.553553 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:46:36.553276    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82e89840-e9d1-46b9-b13d-c19c802a0b7b-model-cache\") pod \"82e89840-e9d1-46b9-b13d-c19c802a0b7b\" (UID: \"82e89840-e9d1-46b9-b13d-c19c802a0b7b\") "
Sep 09 21:46:36.553553 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:46:36.553298    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82e89840-e9d1-46b9-b13d-c19c802a0b7b-tmp-dir\") pod \"82e89840-e9d1-46b9-b13d-c19c802a0b7b\" (UID: \"82e89840-e9d1-46b9-b13d-c19c802a0b7b\") "
Sep 09 21:46:36.553553 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:46:36.553346    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82e89840-e9d1-46b9-b13d-c19c802a0b7b-tls-certs\") pod \"82e89840-e9d1-46b9-b13d-c19c802a0b7b\" (UID: \"82e89840-e9d1-46b9-b13d-c19c802a0b7b\") "
Sep 09 21:46:36.553553 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:46:36.553407    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/82e89840-e9d1-46b9-b13d-c19c802a0b7b-cabundle-cert\") pod \"82e89840-e9d1-46b9-b13d-c19c802a0b7b\" (UID: \"82e89840-e9d1-46b9-b13d-c19c802a0b7b\") "
Sep 09 21:46:36.553553 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:46:36.553493    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82e89840-e9d1-46b9-b13d-c19c802a0b7b-home\") pod \"82e89840-e9d1-46b9-b13d-c19c802a0b7b\" (UID: \"82e89840-e9d1-46b9-b13d-c19c802a0b7b\") "
Sep 09 21:46:36.553858 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:46:36.553587    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82e89840-e9d1-46b9-b13d-c19c802a0b7b-model-cache" (OuterVolumeSpecName: "model-cache") pod "82e89840-e9d1-46b9-b13d-c19c802a0b7b" (UID: "82e89840-e9d1-46b9-b13d-c19c802a0b7b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:46:36.553858 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:46:36.553753    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82e89840-e9d1-46b9-b13d-c19c802a0b7b-model-cache\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:46:36.553962 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:46:36.553934    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82e89840-e9d1-46b9-b13d-c19c802a0b7b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "82e89840-e9d1-46b9-b13d-c19c802a0b7b" (UID: "82e89840-e9d1-46b9-b13d-c19c802a0b7b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:46:36.553962 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:46:36.553935    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82e89840-e9d1-46b9-b13d-c19c802a0b7b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "82e89840-e9d1-46b9-b13d-c19c802a0b7b" (UID: "82e89840-e9d1-46b9-b13d-c19c802a0b7b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:46:36.554043 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:46:36.554004    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82e89840-e9d1-46b9-b13d-c19c802a0b7b-home" (OuterVolumeSpecName: "home") pod "82e89840-e9d1-46b9-b13d-c19c802a0b7b" (UID: "82e89840-e9d1-46b9-b13d-c19c802a0b7b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:46:36.556061 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:46:36.556030    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82e89840-e9d1-46b9-b13d-c19c802a0b7b-kube-api-access-dzbb7" (OuterVolumeSpecName: "kube-api-access-dzbb7") pod "82e89840-e9d1-46b9-b13d-c19c802a0b7b" (UID: "82e89840-e9d1-46b9-b13d-c19c802a0b7b"). InnerVolumeSpecName "kube-api-access-dzbb7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:46:36.556061 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:46:36.556039    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82e89840-e9d1-46b9-b13d-c19c802a0b7b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "82e89840-e9d1-46b9-b13d-c19c802a0b7b" (UID: "82e89840-e9d1-46b9-b13d-c19c802a0b7b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:46:36.556212 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:46:36.556155    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82e89840-e9d1-46b9-b13d-c19c802a0b7b-dshm" (OuterVolumeSpecName: "dshm") pod "82e89840-e9d1-46b9-b13d-c19c802a0b7b" (UID: "82e89840-e9d1-46b9-b13d-c19c802a0b7b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:46:36.582911 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:46:36.582884    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82e89840-e9d1-46b9-b13d-c19c802a0b7b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "82e89840-e9d1-46b9-b13d-c19c802a0b7b" (UID: "82e89840-e9d1-46b9-b13d-c19c802a0b7b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:46:36.655110 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:46:36.655079    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82e89840-e9d1-46b9-b13d-c19c802a0b7b-home\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:46:36.655110 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:46:36.655106    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82e89840-e9d1-46b9-b13d-c19c802a0b7b-dshm\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:46:36.655110 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:46:36.655116    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/82e89840-e9d1-46b9-b13d-c19c802a0b7b-kserve-provision-location\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:46:36.655328 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:46:36.655125    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dzbb7\" (UniqueName: \"kubernetes.io/projected/82e89840-e9d1-46b9-b13d-c19c802a0b7b-kube-api-access-dzbb7\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:46:36.655328 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:46:36.655135    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82e89840-e9d1-46b9-b13d-c19c802a0b7b-tmp-dir\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:46:36.655328 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:46:36.655143    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82e89840-e9d1-46b9-b13d-c19c802a0b7b-tls-certs\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:46:36.655328 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:46:36.655151    2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/82e89840-e9d1-46b9-b13d-c19c802a0b7b-cabundle-cert\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:46:36.949785 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:46:36.949740    2576 generic.go:356] "Generic (PLEG): container finished" podID="82e89840-e9d1-46b9-b13d-c19c802a0b7b" containerID="6ee5e267a0218d2dc139c5f6a49b2f496e24902f829874a6e08a8d139c9c23ea" exitCode=0
Sep 09 21:46:36.950258 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:46:36.949814    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-79cdd87d9c-znm6k"
Sep 09 21:46:36.950258 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:46:36.949837    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-79cdd87d9c-znm6k" event={"ID":"82e89840-e9d1-46b9-b13d-c19c802a0b7b","Type":"ContainerDied","Data":"6ee5e267a0218d2dc139c5f6a49b2f496e24902f829874a6e08a8d139c9c23ea"}
Sep 09 21:46:36.950258 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:46:36.949915    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-79cdd87d9c-znm6k" event={"ID":"82e89840-e9d1-46b9-b13d-c19c802a0b7b","Type":"ContainerDied","Data":"07111c7f9bf3a126d2ffe6b7b1e2e8e84a4321568fbcdb30db857e8b8454b294"}
Sep 09 21:46:36.950258 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:46:36.949945    2576 scope.go:117] "RemoveContainer" containerID="6ee5e267a0218d2dc139c5f6a49b2f496e24902f829874a6e08a8d139c9c23ea"
Sep 09 21:46:36.960087 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:46:36.960067    2576 scope.go:117] "RemoveContainer" containerID="feb7d1c5541051834981177a84b0bad4980f7c2051789c4863ee195c6f7a62c9"
Sep 09 21:46:36.968488 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:46:36.968468    2576 scope.go:117] "RemoveContainer" containerID="6ee5e267a0218d2dc139c5f6a49b2f496e24902f829874a6e08a8d139c9c23ea"
Sep 09 21:46:36.968758 ip-10-0-134-72 kubenswrapper[2576]: E0909 21:46:36.968735    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ee5e267a0218d2dc139c5f6a49b2f496e24902f829874a6e08a8d139c9c23ea\": container with ID starting with 6ee5e267a0218d2dc139c5f6a49b2f496e24902f829874a6e08a8d139c9c23ea not found: ID does not exist" containerID="6ee5e267a0218d2dc139c5f6a49b2f496e24902f829874a6e08a8d139c9c23ea"
Sep 09 21:46:36.968832 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:46:36.968764    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ee5e267a0218d2dc139c5f6a49b2f496e24902f829874a6e08a8d139c9c23ea"} err="failed to get container status \"6ee5e267a0218d2dc139c5f6a49b2f496e24902f829874a6e08a8d139c9c23ea\": rpc error: code = NotFound desc = could not find container \"6ee5e267a0218d2dc139c5f6a49b2f496e24902f829874a6e08a8d139c9c23ea\": container with ID starting with 6ee5e267a0218d2dc139c5f6a49b2f496e24902f829874a6e08a8d139c9c23ea not found: ID does not exist"
Sep 09 21:46:36.968832 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:46:36.968779    2576 scope.go:117] "RemoveContainer" containerID="feb7d1c5541051834981177a84b0bad4980f7c2051789c4863ee195c6f7a62c9"
Sep 09 21:46:36.969003 ip-10-0-134-72 kubenswrapper[2576]: E0909 21:46:36.968984    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"feb7d1c5541051834981177a84b0bad4980f7c2051789c4863ee195c6f7a62c9\": container with ID starting with feb7d1c5541051834981177a84b0bad4980f7c2051789c4863ee195c6f7a62c9 not found: ID does not exist" containerID="feb7d1c5541051834981177a84b0bad4980f7c2051789c4863ee195c6f7a62c9"
Sep 09 21:46:36.969056 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:46:36.969007    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"feb7d1c5541051834981177a84b0bad4980f7c2051789c4863ee195c6f7a62c9"} err="failed to get container status \"feb7d1c5541051834981177a84b0bad4980f7c2051789c4863ee195c6f7a62c9\": rpc error: code = NotFound desc = could not find container \"feb7d1c5541051834981177a84b0bad4980f7c2051789c4863ee195c6f7a62c9\": container with ID starting with feb7d1c5541051834981177a84b0bad4980f7c2051789c4863ee195c6f7a62c9 not found: ID does not exist"
Sep 09 21:46:36.973508 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:46:36.973484    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-79cdd87d9c-znm6k"]
Sep 09 21:46:36.977419 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:46:36.977392    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-79cdd87d9c-znm6k"]
Sep 09 21:46:37.410453 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:46:37.410425    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="82e89840-e9d1-46b9-b13d-c19c802a0b7b" path="/var/lib/kubelet/pods/82e89840-e9d1-46b9-b13d-c19c802a0b7b/volumes"
Sep 09 21:46:38.259106 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:46:38.259066    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:46:38.323585 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:46:38.323540    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:46:42.303097 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:46:42.303066    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-557b9dcccf-csrxs"
Sep 09 21:46:42.304659 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:46:42.304629    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-557b9dcccf-csrxs" podUID="a3014647-a76e-4ffc-88f6-27206f6d5088" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": dial tcp 10.132.0.76:8000: connect: connection refused"
Sep 09 21:46:42.756521 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:46:42.756483    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:46:52.303135 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:46:52.303088    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-557b9dcccf-csrxs" podUID="a3014647-a76e-4ffc-88f6-27206f6d5088" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": dial tcp 10.132.0.76:8000: connect: connection refused"
Sep 09 21:47:02.303224 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:02.303177    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-557b9dcccf-csrxs" podUID="a3014647-a76e-4ffc-88f6-27206f6d5088" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": dial tcp 10.132.0.76:8000: connect: connection refused"
Sep 09 21:47:10.096854 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:10.096817    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:47:10.196105 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:10.196069    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:47:10.295654 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:10.295615    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:47:12.303862 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:12.303824    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-557b9dcccf-csrxs" podUID="a3014647-a76e-4ffc-88f6-27206f6d5088" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": dial tcp 10.132.0.76:8000: connect: connection refused"
Sep 09 21:47:12.329525 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:12.329485    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:47:12.432741 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:12.432704    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:47:12.531236 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:12.531200    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:47:16.609413 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:16.609326    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:47:18.410803 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:18.410762    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:47:18.510409 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:18.510357    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:47:22.240638 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:22.240587    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:47:22.303063 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:22.303019    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-557b9dcccf-csrxs" podUID="a3014647-a76e-4ffc-88f6-27206f6d5088" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": dial tcp 10.132.0.76:8000: connect: connection refused"
Sep 09 21:47:22.303639 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:22.303615    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:47:26.387302 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:26.387267    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-599f4c7c7d-txftf"]
Sep 09 21:47:26.387882 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:26.387803    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82e89840-e9d1-46b9-b13d-c19c802a0b7b" containerName="storage-initializer"
Sep 09 21:47:26.387882 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:26.387823    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="82e89840-e9d1-46b9-b13d-c19c802a0b7b" containerName="storage-initializer"
Sep 09 21:47:26.387882 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:26.387844    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82e89840-e9d1-46b9-b13d-c19c802a0b7b" containerName="main"
Sep 09 21:47:26.387882 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:26.387853    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="82e89840-e9d1-46b9-b13d-c19c802a0b7b" containerName="main"
Sep 09 21:47:26.388105 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:26.387952    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="82e89840-e9d1-46b9-b13d-c19c802a0b7b" containerName="main"
Sep 09 21:47:26.391038 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:26.391018    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-599f4c7c7d-txftf"
Sep 09 21:47:26.393541 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:26.393329    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"kube-root-ca.crt\""
Sep 09 21:47:26.393541 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:26.393402    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"seaweedfs-s3-creds\""
Sep 09 21:47:26.393541 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:26.393405    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"openshift-service-ca.crt\""
Sep 09 21:47:26.393541 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:26.393468    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"default-dockercfg-qqw2c\""
Sep 09 21:47:26.393812 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:26.393573    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"e2e-del-blocked-svc-kserve-self-signed-certs\""
Sep 09 21:47:26.393812 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:26.393735    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5dbddcde-f882-4627-ae57-82d06f8e31c1-home\") pod \"e2e-del-blocked-svc-kserve-599f4c7c7d-txftf\" (UID: \"5dbddcde-f882-4627-ae57-82d06f8e31c1\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-599f4c7c7d-txftf"
Sep 09 21:47:26.393812 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:26.393761    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5dbddcde-f882-4627-ae57-82d06f8e31c1-model-cache\") pod \"e2e-del-blocked-svc-kserve-599f4c7c7d-txftf\" (UID: \"5dbddcde-f882-4627-ae57-82d06f8e31c1\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-599f4c7c7d-txftf"
Sep 09 21:47:26.393957 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:26.393827    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlh5q\" (UniqueName: \"kubernetes.io/projected/5dbddcde-f882-4627-ae57-82d06f8e31c1-kube-api-access-qlh5q\") pod \"e2e-del-blocked-svc-kserve-599f4c7c7d-txftf\" (UID: \"5dbddcde-f882-4627-ae57-82d06f8e31c1\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-599f4c7c7d-txftf"
Sep 09 21:47:26.393957 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:26.393875    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5dbddcde-f882-4627-ae57-82d06f8e31c1-tls-certs\") pod \"e2e-del-blocked-svc-kserve-599f4c7c7d-txftf\" (UID: \"5dbddcde-f882-4627-ae57-82d06f8e31c1\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-599f4c7c7d-txftf"
Sep 09 21:47:26.393957 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:26.393904    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5dbddcde-f882-4627-ae57-82d06f8e31c1-kserve-provision-location\") pod \"e2e-del-blocked-svc-kserve-599f4c7c7d-txftf\" (UID: \"5dbddcde-f882-4627-ae57-82d06f8e31c1\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-599f4c7c7d-txftf"
Sep 09 21:47:26.393957 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:26.393942    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5dbddcde-f882-4627-ae57-82d06f8e31c1-tmp-dir\") pod \"e2e-del-blocked-svc-kserve-599f4c7c7d-txftf\" (UID: \"5dbddcde-f882-4627-ae57-82d06f8e31c1\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-599f4c7c7d-txftf"
Sep 09 21:47:26.394147 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:26.394013    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5dbddcde-f882-4627-ae57-82d06f8e31c1-dshm\") pod \"e2e-del-blocked-svc-kserve-599f4c7c7d-txftf\" (UID: \"5dbddcde-f882-4627-ae57-82d06f8e31c1\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-599f4c7c7d-txftf"
Sep 09 21:47:26.401831 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:26.401810    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-599f4c7c7d-txftf"]
Sep 09 21:47:26.414932 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:26.414897    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:47:26.494476 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:26.494434    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5dbddcde-f882-4627-ae57-82d06f8e31c1-home\") pod \"e2e-del-blocked-svc-kserve-599f4c7c7d-txftf\" (UID: \"5dbddcde-f882-4627-ae57-82d06f8e31c1\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-599f4c7c7d-txftf"
Sep 09 21:47:26.494476 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:26.494473    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5dbddcde-f882-4627-ae57-82d06f8e31c1-model-cache\") pod \"e2e-del-blocked-svc-kserve-599f4c7c7d-txftf\" (UID: \"5dbddcde-f882-4627-ae57-82d06f8e31c1\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-599f4c7c7d-txftf"
Sep 09 21:47:26.494745 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:26.494496    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlh5q\" (UniqueName: \"kubernetes.io/projected/5dbddcde-f882-4627-ae57-82d06f8e31c1-kube-api-access-qlh5q\") pod \"e2e-del-blocked-svc-kserve-599f4c7c7d-txftf\" (UID: \"5dbddcde-f882-4627-ae57-82d06f8e31c1\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-599f4c7c7d-txftf"
Sep 09 21:47:26.494745 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:26.494525    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5dbddcde-f882-4627-ae57-82d06f8e31c1-tls-certs\") pod \"e2e-del-blocked-svc-kserve-599f4c7c7d-txftf\" (UID: \"5dbddcde-f882-4627-ae57-82d06f8e31c1\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-599f4c7c7d-txftf"
Sep 09 21:47:26.494745 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:26.494550    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5dbddcde-f882-4627-ae57-82d06f8e31c1-kserve-provision-location\") pod \"e2e-del-blocked-svc-kserve-599f4c7c7d-txftf\" (UID: \"5dbddcde-f882-4627-ae57-82d06f8e31c1\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-599f4c7c7d-txftf"
Sep 09 21:47:26.494745 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:26.494720    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5dbddcde-f882-4627-ae57-82d06f8e31c1-tmp-dir\") pod \"e2e-del-blocked-svc-kserve-599f4c7c7d-txftf\" (UID: \"5dbddcde-f882-4627-ae57-82d06f8e31c1\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-599f4c7c7d-txftf"
Sep 09 21:47:26.494969 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:26.494820    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5dbddcde-f882-4627-ae57-82d06f8e31c1-dshm\") pod \"e2e-del-blocked-svc-kserve-599f4c7c7d-txftf\" (UID: \"5dbddcde-f882-4627-ae57-82d06f8e31c1\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-599f4c7c7d-txftf"
Sep 09 21:47:26.494969 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:26.494919    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5dbddcde-f882-4627-ae57-82d06f8e31c1-home\") pod \"e2e-del-blocked-svc-kserve-599f4c7c7d-txftf\" (UID: \"5dbddcde-f882-4627-ae57-82d06f8e31c1\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-599f4c7c7d-txftf"
Sep 09 21:47:26.495071 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:26.495006    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5dbddcde-f882-4627-ae57-82d06f8e31c1-kserve-provision-location\") pod \"e2e-del-blocked-svc-kserve-599f4c7c7d-txftf\" (UID: \"5dbddcde-f882-4627-ae57-82d06f8e31c1\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-599f4c7c7d-txftf"
Sep 09 21:47:26.495071 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:26.495035    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5dbddcde-f882-4627-ae57-82d06f8e31c1-model-cache\") pod \"e2e-del-blocked-svc-kserve-599f4c7c7d-txftf\" (UID: \"5dbddcde-f882-4627-ae57-82d06f8e31c1\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-599f4c7c7d-txftf"
Sep 09 21:47:26.495185 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:26.495161    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5dbddcde-f882-4627-ae57-82d06f8e31c1-tmp-dir\") pod \"e2e-del-blocked-svc-kserve-599f4c7c7d-txftf\" (UID: \"5dbddcde-f882-4627-ae57-82d06f8e31c1\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-599f4c7c7d-txftf"
Sep 09 21:47:26.497099 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:26.497076    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5dbddcde-f882-4627-ae57-82d06f8e31c1-dshm\") pod \"e2e-del-blocked-svc-kserve-599f4c7c7d-txftf\" (UID: \"5dbddcde-f882-4627-ae57-82d06f8e31c1\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-599f4c7c7d-txftf"
Sep 09 21:47:26.497293 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:26.497274    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5dbddcde-f882-4627-ae57-82d06f8e31c1-tls-certs\") pod \"e2e-del-blocked-svc-kserve-599f4c7c7d-txftf\" (UID: \"5dbddcde-f882-4627-ae57-82d06f8e31c1\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-599f4c7c7d-txftf"
Sep 09 21:47:26.502270 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:26.502249    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlh5q\" (UniqueName: \"kubernetes.io/projected/5dbddcde-f882-4627-ae57-82d06f8e31c1-kube-api-access-qlh5q\") pod \"e2e-del-blocked-svc-kserve-599f4c7c7d-txftf\" (UID: \"5dbddcde-f882-4627-ae57-82d06f8e31c1\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-599f4c7c7d-txftf"
Sep 09 21:47:26.606208 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:26.606155    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:47:26.703865 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:26.703771    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-599f4c7c7d-txftf"
Sep 09 21:47:26.980497 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:26.980471    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-599f4c7c7d-txftf"]
Sep 09 21:47:26.982444 ip-10-0-134-72 kubenswrapper[2576]: W0909 21:47:26.982415    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5dbddcde_f882_4627_ae57_82d06f8e31c1.slice/crio-0efc6c99786bc35bdc2a160d42e04fd078349dc8a7271418b987e4705a9e2033 WatchSource:0}: Error finding container 0efc6c99786bc35bdc2a160d42e04fd078349dc8a7271418b987e4705a9e2033: Status 404 returned error can't find the container with id 0efc6c99786bc35bdc2a160d42e04fd078349dc8a7271418b987e4705a9e2033
Sep 09 21:47:27.220016 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:27.219970    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-599f4c7c7d-txftf" event={"ID":"5dbddcde-f882-4627-ae57-82d06f8e31c1","Type":"ContainerStarted","Data":"35d7907f27351c8d60e8108d9b427409218561c04785e190382bfc389cf07fcf"}
Sep 09 21:47:27.220216 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:27.220037    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-599f4c7c7d-txftf" event={"ID":"5dbddcde-f882-4627-ae57-82d06f8e31c1","Type":"ContainerStarted","Data":"0efc6c99786bc35bdc2a160d42e04fd078349dc8a7271418b987e4705a9e2033"}
Sep 09 21:47:30.394859 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:30.394825    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-599f4c7c7d-txftf"]
Sep 09 21:47:30.395339 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:30.395062    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-599f4c7c7d-txftf" podUID="5dbddcde-f882-4627-ae57-82d06f8e31c1" containerName="storage-initializer" containerID="cri-o://35d7907f27351c8d60e8108d9b427409218561c04785e190382bfc389cf07fcf" gracePeriod=60
Sep 09 21:47:30.410152 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:30.410091    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:47:30.511472 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:30.511435    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:47:30.995872 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:30.995820    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:47:31.095380 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:31.095318    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:47:31.196387 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:31.196329    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:47:32.249503 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:32.249474    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-599f4c7c7d-txftf"
Sep 09 21:47:32.251121 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:32.251095    2576 generic.go:356] "Generic (PLEG): container finished" podID="5dbddcde-f882-4627-ae57-82d06f8e31c1" containerID="35d7907f27351c8d60e8108d9b427409218561c04785e190382bfc389cf07fcf" exitCode=0
Sep 09 21:47:32.251303 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:32.251133    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-599f4c7c7d-txftf"
Sep 09 21:47:32.251303 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:32.251184    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-599f4c7c7d-txftf" event={"ID":"5dbddcde-f882-4627-ae57-82d06f8e31c1","Type":"ContainerDied","Data":"35d7907f27351c8d60e8108d9b427409218561c04785e190382bfc389cf07fcf"}
Sep 09 21:47:32.251303 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:32.251233    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-599f4c7c7d-txftf" event={"ID":"5dbddcde-f882-4627-ae57-82d06f8e31c1","Type":"ContainerDied","Data":"0efc6c99786bc35bdc2a160d42e04fd078349dc8a7271418b987e4705a9e2033"}
Sep 09 21:47:32.252183 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:32.251260    2576 scope.go:117] "RemoveContainer" containerID="35d7907f27351c8d60e8108d9b427409218561c04785e190382bfc389cf07fcf"
Sep 09 21:47:32.263166 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:32.263143    2576 scope.go:117] "RemoveContainer" containerID="35d7907f27351c8d60e8108d9b427409218561c04785e190382bfc389cf07fcf"
Sep 09 21:47:32.263460 ip-10-0-134-72 kubenswrapper[2576]: E0909 21:47:32.263439    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35d7907f27351c8d60e8108d9b427409218561c04785e190382bfc389cf07fcf\": container with ID starting with 35d7907f27351c8d60e8108d9b427409218561c04785e190382bfc389cf07fcf not found: ID does not exist" containerID="35d7907f27351c8d60e8108d9b427409218561c04785e190382bfc389cf07fcf"
Sep 09 21:47:32.263556 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:32.263466    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35d7907f27351c8d60e8108d9b427409218561c04785e190382bfc389cf07fcf"} err="failed to get container status \"35d7907f27351c8d60e8108d9b427409218561c04785e190382bfc389cf07fcf\": rpc error: code = NotFound desc = could not find container \"35d7907f27351c8d60e8108d9b427409218561c04785e190382bfc389cf07fcf\": container with ID starting with 35d7907f27351c8d60e8108d9b427409218561c04785e190382bfc389cf07fcf not found: ID does not exist"
Sep 09 21:47:32.303608 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:32.303564    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-557b9dcccf-csrxs" podUID="a3014647-a76e-4ffc-88f6-27206f6d5088" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": dial tcp 10.132.0.76:8000: connect: connection refused"
Sep 09 21:47:32.347771 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:32.347742    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5dbddcde-f882-4627-ae57-82d06f8e31c1-model-cache\") pod \"5dbddcde-f882-4627-ae57-82d06f8e31c1\" (UID: \"5dbddcde-f882-4627-ae57-82d06f8e31c1\") "
Sep 09 21:47:32.347971 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:32.347784    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5dbddcde-f882-4627-ae57-82d06f8e31c1-tmp-dir\") pod \"5dbddcde-f882-4627-ae57-82d06f8e31c1\" (UID: \"5dbddcde-f882-4627-ae57-82d06f8e31c1\") "
Sep 09 21:47:32.347971 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:32.347815    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qlh5q\" (UniqueName: \"kubernetes.io/projected/5dbddcde-f882-4627-ae57-82d06f8e31c1-kube-api-access-qlh5q\") pod \"5dbddcde-f882-4627-ae57-82d06f8e31c1\" (UID: \"5dbddcde-f882-4627-ae57-82d06f8e31c1\") "
Sep 09 21:47:32.347971 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:32.347840    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5dbddcde-f882-4627-ae57-82d06f8e31c1-dshm\") pod \"5dbddcde-f882-4627-ae57-82d06f8e31c1\" (UID: \"5dbddcde-f882-4627-ae57-82d06f8e31c1\") "
Sep 09 21:47:32.347971 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:32.347879    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5dbddcde-f882-4627-ae57-82d06f8e31c1-kserve-provision-location\") pod \"5dbddcde-f882-4627-ae57-82d06f8e31c1\" (UID: \"5dbddcde-f882-4627-ae57-82d06f8e31c1\") "
Sep 09 21:47:32.347971 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:32.347899    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5dbddcde-f882-4627-ae57-82d06f8e31c1-home\") pod \"5dbddcde-f882-4627-ae57-82d06f8e31c1\" (UID: \"5dbddcde-f882-4627-ae57-82d06f8e31c1\") "
Sep 09 21:47:32.348409 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:32.348033    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dbddcde-f882-4627-ae57-82d06f8e31c1-model-cache" (OuterVolumeSpecName: "model-cache") pod "5dbddcde-f882-4627-ae57-82d06f8e31c1" (UID: "5dbddcde-f882-4627-ae57-82d06f8e31c1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:47:32.348409 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:32.348151    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dbddcde-f882-4627-ae57-82d06f8e31c1-home" (OuterVolumeSpecName: "home") pod "5dbddcde-f882-4627-ae57-82d06f8e31c1" (UID: "5dbddcde-f882-4627-ae57-82d06f8e31c1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:47:32.348409 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:32.348158    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5dbddcde-f882-4627-ae57-82d06f8e31c1-tls-certs\") pod \"5dbddcde-f882-4627-ae57-82d06f8e31c1\" (UID: \"5dbddcde-f882-4627-ae57-82d06f8e31c1\") "
Sep 09 21:47:32.348409 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:32.348172    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dbddcde-f882-4627-ae57-82d06f8e31c1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5dbddcde-f882-4627-ae57-82d06f8e31c1" (UID: "5dbddcde-f882-4627-ae57-82d06f8e31c1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:47:32.348857 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:32.348476    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5dbddcde-f882-4627-ae57-82d06f8e31c1-model-cache\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:47:32.348857 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:32.348494    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5dbddcde-f882-4627-ae57-82d06f8e31c1-tmp-dir\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:47:32.348857 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:32.348508    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5dbddcde-f882-4627-ae57-82d06f8e31c1-home\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:47:32.350668 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:32.350642    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dbddcde-f882-4627-ae57-82d06f8e31c1-kube-api-access-qlh5q" (OuterVolumeSpecName: "kube-api-access-qlh5q") pod "5dbddcde-f882-4627-ae57-82d06f8e31c1" (UID: "5dbddcde-f882-4627-ae57-82d06f8e31c1"). InnerVolumeSpecName "kube-api-access-qlh5q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:47:32.350811 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:32.350788    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5dbddcde-f882-4627-ae57-82d06f8e31c1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5dbddcde-f882-4627-ae57-82d06f8e31c1" (UID: "5dbddcde-f882-4627-ae57-82d06f8e31c1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:47:32.350887 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:32.350829    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dbddcde-f882-4627-ae57-82d06f8e31c1-dshm" (OuterVolumeSpecName: "dshm") pod "5dbddcde-f882-4627-ae57-82d06f8e31c1" (UID: "5dbddcde-f882-4627-ae57-82d06f8e31c1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:47:32.427488 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:32.427428    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dbddcde-f882-4627-ae57-82d06f8e31c1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5dbddcde-f882-4627-ae57-82d06f8e31c1" (UID: "5dbddcde-f882-4627-ae57-82d06f8e31c1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:47:32.449495 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:32.449455    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qlh5q\" (UniqueName: \"kubernetes.io/projected/5dbddcde-f882-4627-ae57-82d06f8e31c1-kube-api-access-qlh5q\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:47:32.449495 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:32.449481    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5dbddcde-f882-4627-ae57-82d06f8e31c1-dshm\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:47:32.449495 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:32.449492    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5dbddcde-f882-4627-ae57-82d06f8e31c1-kserve-provision-location\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:47:32.449495 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:32.449500    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5dbddcde-f882-4627-ae57-82d06f8e31c1-tls-certs\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:47:32.589312 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:32.589269    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-599f4c7c7d-txftf"]
Sep 09 21:47:32.594212 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:32.594179    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-599f4c7c7d-txftf"]
Sep 09 21:47:33.410482 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:33.410441    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5dbddcde-f882-4627-ae57-82d06f8e31c1" path="/var/lib/kubelet/pods/5dbddcde-f882-4627-ae57-82d06f8e31c1/volumes"
Sep 09 21:47:35.340842 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:35.340792    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:47:35.441318 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:35.441279    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:47:35.542242 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:35.542199    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:47:36.073139 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:36.073104    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:47:41.179702 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:41.179664    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:47:41.279598 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:41.279552    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:47:42.303427 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:42.303388    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-557b9dcccf-csrxs" podUID="a3014647-a76e-4ffc-88f6-27206f6d5088" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": dial tcp 10.132.0.76:8000: connect: connection refused"
Sep 09 21:47:46.488670 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:46.488637    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-config-deletion-unblocked-after-service-delet-de3351af/e2e-del-unblock-svc-kserve-5486858f86-qzlsc"]
Sep 09 21:47:46.489037 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:46.488956    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5dbddcde-f882-4627-ae57-82d06f8e31c1" containerName="storage-initializer"
Sep 09 21:47:46.489037 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:46.488967    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="5dbddcde-f882-4627-ae57-82d06f8e31c1" containerName="storage-initializer"
Sep 09 21:47:46.489110 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:46.489046    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="5dbddcde-f882-4627-ae57-82d06f8e31c1" containerName="storage-initializer"
Sep 09 21:47:46.493939 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:46.493916    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-unblocked-after-service-delet-de3351af/e2e-del-unblock-svc-kserve-5486858f86-qzlsc"
Sep 09 21:47:46.496460 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:46.496431    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-config-deletion-unblocked-after-service-delet-de3351af\"/\"kube-root-ca.crt\""
Sep 09 21:47:46.496584 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:46.496468    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-config-deletion-unblocked-after-service-delet-de3351af\"/\"e2e-del-unblock-svc-kserve-self-signed-certs\""
Sep 09 21:47:46.496584 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:46.496510    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-config-deletion-unblocked-after-service-delet-de3351af\"/\"default-dockercfg-k4pxq\""
Sep 09 21:47:46.496584 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:46.496468    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-config-deletion-unblocked-after-service-delet-de3351af\"/\"seaweedfs-s3-creds\""
Sep 09 21:47:46.496866 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:46.496847    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-config-deletion-unblocked-after-service-delet-de3351af\"/\"openshift-service-ca.crt\""
Sep 09 21:47:46.506682 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:46.506658    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-config-deletion-unblocked-after-service-delet-de3351af/e2e-del-unblock-svc-kserve-5486858f86-qzlsc"]
Sep 09 21:47:46.532438 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:46.532406    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:47:46.581748 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:46.581716    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a3d2d01-5bcb-4b53-9a3f-9b21e16cdac5-tmp-dir\") pod \"e2e-del-unblock-svc-kserve-5486858f86-qzlsc\" (UID: \"2a3d2d01-5bcb-4b53-9a3f-9b21e16cdac5\") " pod="e2e-test-config-deletion-unblocked-after-service-delet-de3351af/e2e-del-unblock-svc-kserve-5486858f86-qzlsc"
Sep 09 21:47:46.581955 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:46.581777    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a3d2d01-5bcb-4b53-9a3f-9b21e16cdac5-home\") pod \"e2e-del-unblock-svc-kserve-5486858f86-qzlsc\" (UID: \"2a3d2d01-5bcb-4b53-9a3f-9b21e16cdac5\") " pod="e2e-test-config-deletion-unblocked-after-service-delet-de3351af/e2e-del-unblock-svc-kserve-5486858f86-qzlsc"
Sep 09 21:47:46.581955 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:46.581811    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2a3d2d01-5bcb-4b53-9a3f-9b21e16cdac5-kserve-provision-location\") pod \"e2e-del-unblock-svc-kserve-5486858f86-qzlsc\" (UID: \"2a3d2d01-5bcb-4b53-9a3f-9b21e16cdac5\") " pod="e2e-test-config-deletion-unblocked-after-service-delet-de3351af/e2e-del-unblock-svc-kserve-5486858f86-qzlsc"
Sep 09 21:47:46.581955 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:46.581842    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a3d2d01-5bcb-4b53-9a3f-9b21e16cdac5-model-cache\") pod \"e2e-del-unblock-svc-kserve-5486858f86-qzlsc\" (UID: \"2a3d2d01-5bcb-4b53-9a3f-9b21e16cdac5\") " pod="e2e-test-config-deletion-unblocked-after-service-delet-de3351af/e2e-del-unblock-svc-kserve-5486858f86-qzlsc"
Sep 09 21:47:46.581955 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:46.581914    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a3d2d01-5bcb-4b53-9a3f-9b21e16cdac5-dshm\") pod \"e2e-del-unblock-svc-kserve-5486858f86-qzlsc\" (UID: \"2a3d2d01-5bcb-4b53-9a3f-9b21e16cdac5\") " pod="e2e-test-config-deletion-unblocked-after-service-delet-de3351af/e2e-del-unblock-svc-kserve-5486858f86-qzlsc"
Sep 09 21:47:46.582159 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:46.581962    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrrdg\" (UniqueName: \"kubernetes.io/projected/2a3d2d01-5bcb-4b53-9a3f-9b21e16cdac5-kube-api-access-lrrdg\") pod \"e2e-del-unblock-svc-kserve-5486858f86-qzlsc\" (UID: \"2a3d2d01-5bcb-4b53-9a3f-9b21e16cdac5\") " pod="e2e-test-config-deletion-unblocked-after-service-delet-de3351af/e2e-del-unblock-svc-kserve-5486858f86-qzlsc"
Sep 09 21:47:46.582159 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:46.582003    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a3d2d01-5bcb-4b53-9a3f-9b21e16cdac5-tls-certs\") pod \"e2e-del-unblock-svc-kserve-5486858f86-qzlsc\" (UID: \"2a3d2d01-5bcb-4b53-9a3f-9b21e16cdac5\") " pod="e2e-test-config-deletion-unblocked-after-service-delet-de3351af/e2e-del-unblock-svc-kserve-5486858f86-qzlsc"
Sep 09 21:47:46.683001 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:46.682962    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a3d2d01-5bcb-4b53-9a3f-9b21e16cdac5-home\") pod \"e2e-del-unblock-svc-kserve-5486858f86-qzlsc\" (UID: \"2a3d2d01-5bcb-4b53-9a3f-9b21e16cdac5\") " pod="e2e-test-config-deletion-unblocked-after-service-delet-de3351af/e2e-del-unblock-svc-kserve-5486858f86-qzlsc"
Sep 09 21:47:46.683179 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:46.683006    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2a3d2d01-5bcb-4b53-9a3f-9b21e16cdac5-kserve-provision-location\") pod \"e2e-del-unblock-svc-kserve-5486858f86-qzlsc\" (UID: \"2a3d2d01-5bcb-4b53-9a3f-9b21e16cdac5\") " pod="e2e-test-config-deletion-unblocked-after-service-delet-de3351af/e2e-del-unblock-svc-kserve-5486858f86-qzlsc"
Sep 09 21:47:46.683179 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:46.683050    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a3d2d01-5bcb-4b53-9a3f-9b21e16cdac5-model-cache\") pod \"e2e-del-unblock-svc-kserve-5486858f86-qzlsc\" (UID: \"2a3d2d01-5bcb-4b53-9a3f-9b21e16cdac5\") " pod="e2e-test-config-deletion-unblocked-after-service-delet-de3351af/e2e-del-unblock-svc-kserve-5486858f86-qzlsc"
Sep 09 21:47:46.683179 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:46.683078    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a3d2d01-5bcb-4b53-9a3f-9b21e16cdac5-dshm\") pod \"e2e-del-unblock-svc-kserve-5486858f86-qzlsc\" (UID: \"2a3d2d01-5bcb-4b53-9a3f-9b21e16cdac5\") " pod="e2e-test-config-deletion-unblocked-after-service-delet-de3351af/e2e-del-unblock-svc-kserve-5486858f86-qzlsc"
Sep 09 21:47:46.683179 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:46.683107    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrrdg\" (UniqueName: \"kubernetes.io/projected/2a3d2d01-5bcb-4b53-9a3f-9b21e16cdac5-kube-api-access-lrrdg\") pod \"e2e-del-unblock-svc-kserve-5486858f86-qzlsc\" (UID: \"2a3d2d01-5bcb-4b53-9a3f-9b21e16cdac5\") " pod="e2e-test-config-deletion-unblocked-after-service-delet-de3351af/e2e-del-unblock-svc-kserve-5486858f86-qzlsc"
Sep 09 21:47:46.683179 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:46.683157    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a3d2d01-5bcb-4b53-9a3f-9b21e16cdac5-tls-certs\") pod \"e2e-del-unblock-svc-kserve-5486858f86-qzlsc\" (UID: \"2a3d2d01-5bcb-4b53-9a3f-9b21e16cdac5\") " pod="e2e-test-config-deletion-unblocked-after-service-delet-de3351af/e2e-del-unblock-svc-kserve-5486858f86-qzlsc"
Sep 09 21:47:46.683553 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:46.683524    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2a3d2d01-5bcb-4b53-9a3f-9b21e16cdac5-kserve-provision-location\") pod \"e2e-del-unblock-svc-kserve-5486858f86-qzlsc\" (UID: \"2a3d2d01-5bcb-4b53-9a3f-9b21e16cdac5\") " pod="e2e-test-config-deletion-unblocked-after-service-delet-de3351af/e2e-del-unblock-svc-kserve-5486858f86-qzlsc"
Sep 09 21:47:46.683644 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:46.683577    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a3d2d01-5bcb-4b53-9a3f-9b21e16cdac5-tmp-dir\") pod \"e2e-del-unblock-svc-kserve-5486858f86-qzlsc\" (UID: \"2a3d2d01-5bcb-4b53-9a3f-9b21e16cdac5\") " pod="e2e-test-config-deletion-unblocked-after-service-delet-de3351af/e2e-del-unblock-svc-kserve-5486858f86-qzlsc"
Sep 09 21:47:46.683644 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:46.683587    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a3d2d01-5bcb-4b53-9a3f-9b21e16cdac5-model-cache\") pod \"e2e-del-unblock-svc-kserve-5486858f86-qzlsc\" (UID: \"2a3d2d01-5bcb-4b53-9a3f-9b21e16cdac5\") " pod="e2e-test-config-deletion-unblocked-after-service-delet-de3351af/e2e-del-unblock-svc-kserve-5486858f86-qzlsc"
Sep 09 21:47:46.683815 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:46.683796    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a3d2d01-5bcb-4b53-9a3f-9b21e16cdac5-home\") pod \"e2e-del-unblock-svc-kserve-5486858f86-qzlsc\" (UID: \"2a3d2d01-5bcb-4b53-9a3f-9b21e16cdac5\") " pod="e2e-test-config-deletion-unblocked-after-service-delet-de3351af/e2e-del-unblock-svc-kserve-5486858f86-qzlsc"
Sep 09 21:47:46.683922 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:46.683902    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a3d2d01-5bcb-4b53-9a3f-9b21e16cdac5-tmp-dir\") pod \"e2e-del-unblock-svc-kserve-5486858f86-qzlsc\" (UID: \"2a3d2d01-5bcb-4b53-9a3f-9b21e16cdac5\") " pod="e2e-test-config-deletion-unblocked-after-service-delet-de3351af/e2e-del-unblock-svc-kserve-5486858f86-qzlsc"
Sep 09 21:47:46.686280 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:46.686256    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a3d2d01-5bcb-4b53-9a3f-9b21e16cdac5-dshm\") pod \"e2e-del-unblock-svc-kserve-5486858f86-qzlsc\" (UID: \"2a3d2d01-5bcb-4b53-9a3f-9b21e16cdac5\") " pod="e2e-test-config-deletion-unblocked-after-service-delet-de3351af/e2e-del-unblock-svc-kserve-5486858f86-qzlsc"
Sep 09 21:47:46.686574 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:46.686551    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a3d2d01-5bcb-4b53-9a3f-9b21e16cdac5-tls-certs\") pod \"e2e-del-unblock-svc-kserve-5486858f86-qzlsc\" (UID: \"2a3d2d01-5bcb-4b53-9a3f-9b21e16cdac5\") " pod="e2e-test-config-deletion-unblocked-after-service-delet-de3351af/e2e-del-unblock-svc-kserve-5486858f86-qzlsc"
Sep 09 21:47:46.696003 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:46.695982    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrrdg\" (UniqueName: \"kubernetes.io/projected/2a3d2d01-5bcb-4b53-9a3f-9b21e16cdac5-kube-api-access-lrrdg\") pod \"e2e-del-unblock-svc-kserve-5486858f86-qzlsc\" (UID: \"2a3d2d01-5bcb-4b53-9a3f-9b21e16cdac5\") " pod="e2e-test-config-deletion-unblocked-after-service-delet-de3351af/e2e-del-unblock-svc-kserve-5486858f86-qzlsc"
Sep 09 21:47:46.733905 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:46.733867    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:47:46.808123 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:46.808033    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-unblocked-after-service-delet-de3351af/e2e-del-unblock-svc-kserve-5486858f86-qzlsc"
Sep 09 21:47:47.094791 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:47.094761    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-config-deletion-unblocked-after-service-delet-de3351af/e2e-del-unblock-svc-kserve-5486858f86-qzlsc"]
Sep 09 21:47:47.096332 ip-10-0-134-72 kubenswrapper[2576]: W0909 21:47:47.096306    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a3d2d01_5bcb_4b53_9a3f_9b21e16cdac5.slice/crio-1b4425eccdc7826a2c26b25dcaa8489e52d0c238bebf21ef1aad694817c9ec34 WatchSource:0}: Error finding container 1b4425eccdc7826a2c26b25dcaa8489e52d0c238bebf21ef1aad694817c9ec34: Status 404 returned error can't find the container with id 1b4425eccdc7826a2c26b25dcaa8489e52d0c238bebf21ef1aad694817c9ec34
Sep 09 21:47:47.330781 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:47.330743    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-unblocked-after-service-delet-de3351af/e2e-del-unblock-svc-kserve-5486858f86-qzlsc" event={"ID":"2a3d2d01-5bcb-4b53-9a3f-9b21e16cdac5","Type":"ContainerStarted","Data":"11a58e2283072140cedf022783a756a7f6fd620fd21755031cb7ecf3b2fae16f"}
Sep 09 21:47:47.330959 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:47.330817    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-unblocked-after-service-delet-de3351af/e2e-del-unblock-svc-kserve-5486858f86-qzlsc" event={"ID":"2a3d2d01-5bcb-4b53-9a3f-9b21e16cdac5","Type":"ContainerStarted","Data":"1b4425eccdc7826a2c26b25dcaa8489e52d0c238bebf21ef1aad694817c9ec34"}
Sep 09 21:47:49.938533 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:49.938491    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-config-deletion-unblocked-after-service-delet-de3351af/e2e-del-unblock-svc-kserve-5486858f86-qzlsc"]
Sep 09 21:47:49.938983 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:49.938730    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-config-deletion-unblocked-after-service-delet-de3351af/e2e-del-unblock-svc-kserve-5486858f86-qzlsc" podUID="2a3d2d01-5bcb-4b53-9a3f-9b21e16cdac5" containerName="storage-initializer" containerID="cri-o://11a58e2283072140cedf022783a756a7f6fd620fd21755031cb7ecf3b2fae16f" gracePeriod=60
Sep 09 21:47:50.232188 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:50.232085    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:47:50.353741 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:50.353708    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:47:51.833438 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:51.833400    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:47:52.046117 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:52.046074    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:47:52.143125 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:52.143086    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:47:52.303874 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:52.303831    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-557b9dcccf-csrxs" podUID="a3014647-a76e-4ffc-88f6-27206f6d5088" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": dial tcp 10.132.0.76:8000: connect: connection refused"
Sep 09 21:47:52.362085 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:52.362042    2576 generic.go:356] "Generic (PLEG): container finished" podID="2a3d2d01-5bcb-4b53-9a3f-9b21e16cdac5" containerID="11a58e2283072140cedf022783a756a7f6fd620fd21755031cb7ecf3b2fae16f" exitCode=0
Sep 09 21:47:52.362258 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:52.362100    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-unblocked-after-service-delet-de3351af/e2e-del-unblock-svc-kserve-5486858f86-qzlsc" event={"ID":"2a3d2d01-5bcb-4b53-9a3f-9b21e16cdac5","Type":"ContainerDied","Data":"11a58e2283072140cedf022783a756a7f6fd620fd21755031cb7ecf3b2fae16f"}
Sep 09 21:47:52.388545 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:52.388510    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-unblocked-after-service-delet-de3351af/e2e-del-unblock-svc-kserve-5486858f86-qzlsc"
Sep 09 21:47:52.536044 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:52.535966    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a3d2d01-5bcb-4b53-9a3f-9b21e16cdac5-home\") pod \"2a3d2d01-5bcb-4b53-9a3f-9b21e16cdac5\" (UID: \"2a3d2d01-5bcb-4b53-9a3f-9b21e16cdac5\") "
Sep 09 21:47:52.536044 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:52.536015    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a3d2d01-5bcb-4b53-9a3f-9b21e16cdac5-tmp-dir\") pod \"2a3d2d01-5bcb-4b53-9a3f-9b21e16cdac5\" (UID: \"2a3d2d01-5bcb-4b53-9a3f-9b21e16cdac5\") "
Sep 09 21:47:52.536258 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:52.536107    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a3d2d01-5bcb-4b53-9a3f-9b21e16cdac5-model-cache\") pod \"2a3d2d01-5bcb-4b53-9a3f-9b21e16cdac5\" (UID: \"2a3d2d01-5bcb-4b53-9a3f-9b21e16cdac5\") "
Sep 09 21:47:52.536258 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:52.536136    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a3d2d01-5bcb-4b53-9a3f-9b21e16cdac5-dshm\") pod \"2a3d2d01-5bcb-4b53-9a3f-9b21e16cdac5\" (UID: \"2a3d2d01-5bcb-4b53-9a3f-9b21e16cdac5\") "
Sep 09 21:47:52.536258 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:52.536183    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lrrdg\" (UniqueName: \"kubernetes.io/projected/2a3d2d01-5bcb-4b53-9a3f-9b21e16cdac5-kube-api-access-lrrdg\") pod \"2a3d2d01-5bcb-4b53-9a3f-9b21e16cdac5\" (UID: \"2a3d2d01-5bcb-4b53-9a3f-9b21e16cdac5\") "
Sep 09 21:47:52.536258 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:52.536212    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a3d2d01-5bcb-4b53-9a3f-9b21e16cdac5-home" (OuterVolumeSpecName: "home") pod "2a3d2d01-5bcb-4b53-9a3f-9b21e16cdac5" (UID: "2a3d2d01-5bcb-4b53-9a3f-9b21e16cdac5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:47:52.536258 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:52.536224    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2a3d2d01-5bcb-4b53-9a3f-9b21e16cdac5-kserve-provision-location\") pod \"2a3d2d01-5bcb-4b53-9a3f-9b21e16cdac5\" (UID: \"2a3d2d01-5bcb-4b53-9a3f-9b21e16cdac5\") "
Sep 09 21:47:52.536258 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:52.536249    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a3d2d01-5bcb-4b53-9a3f-9b21e16cdac5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2a3d2d01-5bcb-4b53-9a3f-9b21e16cdac5" (UID: "2a3d2d01-5bcb-4b53-9a3f-9b21e16cdac5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:47:52.536591 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:52.536297    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a3d2d01-5bcb-4b53-9a3f-9b21e16cdac5-tls-certs\") pod \"2a3d2d01-5bcb-4b53-9a3f-9b21e16cdac5\" (UID: \"2a3d2d01-5bcb-4b53-9a3f-9b21e16cdac5\") "
Sep 09 21:47:52.536591 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:52.536340    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a3d2d01-5bcb-4b53-9a3f-9b21e16cdac5-model-cache" (OuterVolumeSpecName: "model-cache") pod "2a3d2d01-5bcb-4b53-9a3f-9b21e16cdac5" (UID: "2a3d2d01-5bcb-4b53-9a3f-9b21e16cdac5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:47:52.536683 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:52.536588    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a3d2d01-5bcb-4b53-9a3f-9b21e16cdac5-model-cache\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:47:52.536683 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:52.536608    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a3d2d01-5bcb-4b53-9a3f-9b21e16cdac5-home\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:47:52.536683 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:52.536621    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a3d2d01-5bcb-4b53-9a3f-9b21e16cdac5-tmp-dir\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:47:52.538811 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:52.538783    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a3d2d01-5bcb-4b53-9a3f-9b21e16cdac5-dshm" (OuterVolumeSpecName: "dshm") pod "2a3d2d01-5bcb-4b53-9a3f-9b21e16cdac5" (UID: "2a3d2d01-5bcb-4b53-9a3f-9b21e16cdac5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:47:52.539141 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:52.539122    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a3d2d01-5bcb-4b53-9a3f-9b21e16cdac5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2a3d2d01-5bcb-4b53-9a3f-9b21e16cdac5" (UID: "2a3d2d01-5bcb-4b53-9a3f-9b21e16cdac5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:47:52.539241 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:52.539217    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a3d2d01-5bcb-4b53-9a3f-9b21e16cdac5-kube-api-access-lrrdg" (OuterVolumeSpecName: "kube-api-access-lrrdg") pod "2a3d2d01-5bcb-4b53-9a3f-9b21e16cdac5" (UID: "2a3d2d01-5bcb-4b53-9a3f-9b21e16cdac5"). InnerVolumeSpecName "kube-api-access-lrrdg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:47:52.605433 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:52.605390    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a3d2d01-5bcb-4b53-9a3f-9b21e16cdac5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2a3d2d01-5bcb-4b53-9a3f-9b21e16cdac5" (UID: "2a3d2d01-5bcb-4b53-9a3f-9b21e16cdac5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:47:52.637409 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:52.637375    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a3d2d01-5bcb-4b53-9a3f-9b21e16cdac5-dshm\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:47:52.637409 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:52.637402    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrrdg\" (UniqueName: \"kubernetes.io/projected/2a3d2d01-5bcb-4b53-9a3f-9b21e16cdac5-kube-api-access-lrrdg\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:47:52.637409 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:52.637414    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2a3d2d01-5bcb-4b53-9a3f-9b21e16cdac5-kserve-provision-location\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:47:52.637636 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:52.637424    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a3d2d01-5bcb-4b53-9a3f-9b21e16cdac5-tls-certs\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:47:53.369337 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:53.369308    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-unblocked-after-service-delet-de3351af/e2e-del-unblock-svc-kserve-5486858f86-qzlsc"
Sep 09 21:47:53.369794 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:53.369335    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-unblocked-after-service-delet-de3351af/e2e-del-unblock-svc-kserve-5486858f86-qzlsc" event={"ID":"2a3d2d01-5bcb-4b53-9a3f-9b21e16cdac5","Type":"ContainerDied","Data":"1b4425eccdc7826a2c26b25dcaa8489e52d0c238bebf21ef1aad694817c9ec34"}
Sep 09 21:47:53.369794 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:53.369424    2576 scope.go:117] "RemoveContainer" containerID="11a58e2283072140cedf022783a756a7f6fd620fd21755031cb7ecf3b2fae16f"
Sep 09 21:47:53.412162 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:53.412134    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-config-deletion-unblocked-after-service-delet-de3351af/e2e-del-unblock-svc-kserve-5486858f86-qzlsc"]
Sep 09 21:47:53.417133 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:53.417102    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-config-deletion-unblocked-after-service-delet-de3351af/e2e-del-unblock-svc-kserve-5486858f86-qzlsc"]
Sep 09 21:47:55.408762 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:55.408717    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a3d2d01-5bcb-4b53-9a3f-9b21e16cdac5" path="/var/lib/kubelet/pods/2a3d2d01-5bcb-4b53-9a3f-9b21e16cdac5/volumes"
Sep 09 21:47:56.918638 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:47:56.918606    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:48:02.303993 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:48:02.303902    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-557b9dcccf-csrxs" podUID="a3014647-a76e-4ffc-88f6-27206f6d5088" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": dial tcp 10.132.0.76:8000: connect: connection refused"
Sep 09 21:48:05.586349 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:48:05.586312    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-8669f5b8c5-cvsht"]
Sep 09 21:48:05.586811 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:48:05.586659    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a3d2d01-5bcb-4b53-9a3f-9b21e16cdac5" containerName="storage-initializer"
Sep 09 21:48:05.586811 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:48:05.586671    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a3d2d01-5bcb-4b53-9a3f-9b21e16cdac5" containerName="storage-initializer"
Sep 09 21:48:05.586811 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:48:05.586728    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a3d2d01-5bcb-4b53-9a3f-9b21e16cdac5" containerName="storage-initializer"
Sep 09 21:48:05.591143 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:48:05.591125    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-8669f5b8c5-cvsht"
Sep 09 21:48:05.593726 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:48:05.593700    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b\"/\"kube-root-ca.crt\""
Sep 09 21:48:05.593837 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:48:05.593700    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b\"/\"seaweedfs-s3-creds\""
Sep 09 21:48:05.594611 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:48:05.594591    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b\"/\"default-dockercfg-97ft5\""
Sep 09 21:48:05.594745 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:48:05.594626    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b\"/\"openshift-service-ca.crt\""
Sep 09 21:48:05.599828 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:48:05.599807    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-8669f5b8c5-cvsht"]
Sep 09 21:48:05.665405 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:48:05.665347    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:48:05.753856 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:48:05.753814    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/43c81f83-6469-4f72-b625-b2999e74f720-kserve-provision-location\") pod \"e2e-wk-cfg-implicit-svc-kserve-8669f5b8c5-cvsht\" (UID: \"43c81f83-6469-4f72-b625-b2999e74f720\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-8669f5b8c5-cvsht"
Sep 09 21:48:05.754069 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:48:05.753900    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtkpb\" (UniqueName: \"kubernetes.io/projected/43c81f83-6469-4f72-b625-b2999e74f720-kube-api-access-qtkpb\") pod \"e2e-wk-cfg-implicit-svc-kserve-8669f5b8c5-cvsht\" (UID: \"43c81f83-6469-4f72-b625-b2999e74f720\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-8669f5b8c5-cvsht"
Sep 09 21:48:05.846961 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:48:05.846870    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:48:05.854760 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:48:05.854724    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtkpb\" (UniqueName: \"kubernetes.io/projected/43c81f83-6469-4f72-b625-b2999e74f720-kube-api-access-qtkpb\") pod \"e2e-wk-cfg-implicit-svc-kserve-8669f5b8c5-cvsht\" (UID: \"43c81f83-6469-4f72-b625-b2999e74f720\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-8669f5b8c5-cvsht"
Sep 09 21:48:05.854902 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:48:05.854799    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/43c81f83-6469-4f72-b625-b2999e74f720-kserve-provision-location\") pod \"e2e-wk-cfg-implicit-svc-kserve-8669f5b8c5-cvsht\" (UID: \"43c81f83-6469-4f72-b625-b2999e74f720\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-8669f5b8c5-cvsht"
Sep 09 21:48:05.855183 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:48:05.855162    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/43c81f83-6469-4f72-b625-b2999e74f720-kserve-provision-location\") pod \"e2e-wk-cfg-implicit-svc-kserve-8669f5b8c5-cvsht\" (UID: \"43c81f83-6469-4f72-b625-b2999e74f720\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-8669f5b8c5-cvsht"
Sep 09 21:48:05.863574 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:48:05.863550    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtkpb\" (UniqueName: \"kubernetes.io/projected/43c81f83-6469-4f72-b625-b2999e74f720-kube-api-access-qtkpb\") pod \"e2e-wk-cfg-implicit-svc-kserve-8669f5b8c5-cvsht\" (UID: \"43c81f83-6469-4f72-b625-b2999e74f720\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-8669f5b8c5-cvsht"
Sep 09 21:48:05.902184 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:48:05.902150    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-8669f5b8c5-cvsht"
Sep 09 21:48:06.175270 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:48:06.175247    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-8669f5b8c5-cvsht"]
Sep 09 21:48:06.177751 ip-10-0-134-72 kubenswrapper[2576]: W0909 21:48:06.177728    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43c81f83_6469_4f72_b625_b2999e74f720.slice/crio-4ec8f4e2c453b0ce67299b3ff86363de7ff9195327e4cb09b16e39543b2fef1c WatchSource:0}: Error finding container 4ec8f4e2c453b0ce67299b3ff86363de7ff9195327e4cb09b16e39543b2fef1c: Status 404 returned error can't find the container with id 4ec8f4e2c453b0ce67299b3ff86363de7ff9195327e4cb09b16e39543b2fef1c
Sep 09 21:48:06.444226 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:48:06.444126    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-8669f5b8c5-cvsht" event={"ID":"43c81f83-6469-4f72-b625-b2999e74f720","Type":"ContainerStarted","Data":"83f55037932ccd601fa8299b7c2ebdfcc8d03bd095264569f84beb7be1bb16f5"}
Sep 09 21:48:06.444226 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:48:06.444196    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-8669f5b8c5-cvsht" event={"ID":"43c81f83-6469-4f72-b625-b2999e74f720","Type":"ContainerStarted","Data":"4ec8f4e2c453b0ce67299b3ff86363de7ff9195327e4cb09b16e39543b2fef1c"}
Sep 09 21:48:09.306080 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:48:09.306045    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-8669f5b8c5-cvsht"]
Sep 09 21:48:09.306578 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:48:09.306285    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-8669f5b8c5-cvsht" podUID="43c81f83-6469-4f72-b625-b2999e74f720" containerName="storage-initializer" containerID="cri-o://83f55037932ccd601fa8299b7c2ebdfcc8d03bd095264569f84beb7be1bb16f5" gracePeriod=30
Sep 09 21:48:09.309687 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:48:09.309657    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:48:09.411611 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:48:09.411550    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:48:10.897634 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:48:10.897596    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:48:10.997754 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:48:10.997711    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:48:11.264881 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:48:11.264854    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-8669f5b8c5-cvsht"
Sep 09 21:48:11.297465 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:48:11.297429    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qtkpb\" (UniqueName: \"kubernetes.io/projected/43c81f83-6469-4f72-b625-b2999e74f720-kube-api-access-qtkpb\") pod \"43c81f83-6469-4f72-b625-b2999e74f720\" (UID: \"43c81f83-6469-4f72-b625-b2999e74f720\") "
Sep 09 21:48:11.297626 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:48:11.297532    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/43c81f83-6469-4f72-b625-b2999e74f720-kserve-provision-location\") pod \"43c81f83-6469-4f72-b625-b2999e74f720\" (UID: \"43c81f83-6469-4f72-b625-b2999e74f720\") "
Sep 09 21:48:11.299851 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:48:11.299818    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43c81f83-6469-4f72-b625-b2999e74f720-kube-api-access-qtkpb" (OuterVolumeSpecName: "kube-api-access-qtkpb") pod "43c81f83-6469-4f72-b625-b2999e74f720" (UID: "43c81f83-6469-4f72-b625-b2999e74f720"). InnerVolumeSpecName "kube-api-access-qtkpb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:48:11.366580 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:48:11.366535    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43c81f83-6469-4f72-b625-b2999e74f720-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "43c81f83-6469-4f72-b625-b2999e74f720" (UID: "43c81f83-6469-4f72-b625-b2999e74f720"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:48:11.398593 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:48:11.398564    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/43c81f83-6469-4f72-b625-b2999e74f720-kserve-provision-location\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:48:11.398593 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:48:11.398592    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtkpb\" (UniqueName: \"kubernetes.io/projected/43c81f83-6469-4f72-b625-b2999e74f720-kube-api-access-qtkpb\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:48:11.478764 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:48:11.478674    2576 generic.go:356] "Generic (PLEG): container finished" podID="43c81f83-6469-4f72-b625-b2999e74f720" containerID="83f55037932ccd601fa8299b7c2ebdfcc8d03bd095264569f84beb7be1bb16f5" exitCode=0
Sep 09 21:48:11.478764 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:48:11.478727    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-8669f5b8c5-cvsht"
Sep 09 21:48:11.478952 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:48:11.478768    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-8669f5b8c5-cvsht" event={"ID":"43c81f83-6469-4f72-b625-b2999e74f720","Type":"ContainerDied","Data":"83f55037932ccd601fa8299b7c2ebdfcc8d03bd095264569f84beb7be1bb16f5"}
Sep 09 21:48:11.478952 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:48:11.478822    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-8669f5b8c5-cvsht" event={"ID":"43c81f83-6469-4f72-b625-b2999e74f720","Type":"ContainerDied","Data":"4ec8f4e2c453b0ce67299b3ff86363de7ff9195327e4cb09b16e39543b2fef1c"}
Sep 09 21:48:11.478952 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:48:11.478844    2576 scope.go:117] "RemoveContainer" containerID="83f55037932ccd601fa8299b7c2ebdfcc8d03bd095264569f84beb7be1bb16f5"
Sep 09 21:48:11.488348 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:48:11.488329    2576 scope.go:117] "RemoveContainer" containerID="83f55037932ccd601fa8299b7c2ebdfcc8d03bd095264569f84beb7be1bb16f5"
Sep 09 21:48:11.488660 ip-10-0-134-72 kubenswrapper[2576]: E0909 21:48:11.488640    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83f55037932ccd601fa8299b7c2ebdfcc8d03bd095264569f84beb7be1bb16f5\": container with ID starting with 83f55037932ccd601fa8299b7c2ebdfcc8d03bd095264569f84beb7be1bb16f5 not found: ID does not exist" containerID="83f55037932ccd601fa8299b7c2ebdfcc8d03bd095264569f84beb7be1bb16f5"
Sep 09 21:48:11.488735 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:48:11.488669    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83f55037932ccd601fa8299b7c2ebdfcc8d03bd095264569f84beb7be1bb16f5"} err="failed to get container status \"83f55037932ccd601fa8299b7c2ebdfcc8d03bd095264569f84beb7be1bb16f5\": rpc error: code = NotFound desc = could not find container \"83f55037932ccd601fa8299b7c2ebdfcc8d03bd095264569f84beb7be1bb16f5\": container with ID starting with 83f55037932ccd601fa8299b7c2ebdfcc8d03bd095264569f84beb7be1bb16f5 not found: ID does not exist"
Sep 09 21:48:11.565523 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:48:11.565490    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-8669f5b8c5-cvsht"]
Sep 09 21:48:11.579445 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:48:11.579414    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-8669f5b8c5-cvsht"]
Sep 09 21:48:12.303453 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:48:12.303417    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-557b9dcccf-csrxs" podUID="a3014647-a76e-4ffc-88f6-27206f6d5088" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": dial tcp 10.132.0.76:8000: connect: connection refused"
Sep 09 21:48:13.409027 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:48:13.408998    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="43c81f83-6469-4f72-b625-b2999e74f720" path="/var/lib/kubelet/pods/43c81f83-6469-4f72-b625-b2999e74f720/volumes"
Sep 09 21:48:15.917950 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:48:15.917914    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:48:17.773138 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:48:17.773099    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7644fc97d-z9s46"]
Sep 09 21:48:17.773634 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:48:17.773613    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43c81f83-6469-4f72-b625-b2999e74f720" containerName="storage-initializer"
Sep 09 21:48:17.773716 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:48:17.773639    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="43c81f83-6469-4f72-b625-b2999e74f720" containerName="storage-initializer"
Sep 09 21:48:17.773776 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:48:17.773743    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="43c81f83-6469-4f72-b625-b2999e74f720" containerName="storage-initializer"
Sep 09 21:48:17.779280 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:48:17.779257    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7644fc97d-z9s46"
Sep 09 21:48:17.781917 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:48:17.781892    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"openshift-service-ca.crt\""
Sep 09 21:48:17.782008 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:48:17.781979    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"odh-kserve-custom-ca-bundle\""
Sep 09 21:48:17.782252 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:48:17.782231    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"seaweedfs-s3-creds\""
Sep 09 21:48:17.782349 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:48:17.782280    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"default-dockercfg-lnjzv\""
Sep 09 21:48:17.782349 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:48:17.782323    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-kserve-self-signed-certs\""
Sep 09 21:48:17.782486 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:48:17.782437    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"kube-root-ca.crt\""
Sep 09 21:48:17.791604 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:48:17.791583    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7644fc97d-z9s46"]
Sep 09 21:48:17.810939 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:48:17.810907    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:48:17.856014 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:48:17.855985    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ef9fb10-bd61-4107-8393-7a272150bd8d-model-cache\") pod \"stop-feature-test-kserve-7644fc97d-z9s46\" (UID: \"1ef9fb10-bd61-4107-8393-7a272150bd8d\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7644fc97d-z9s46"
Sep 09 21:48:17.856130 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:48:17.856021    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1ef9fb10-bd61-4107-8393-7a272150bd8d-cabundle-cert\") pod \"stop-feature-test-kserve-7644fc97d-z9s46\" (UID: \"1ef9fb10-bd61-4107-8393-7a272150bd8d\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7644fc97d-z9s46"
Sep 09 21:48:17.856130 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:48:17.856044    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ef9fb10-bd61-4107-8393-7a272150bd8d-tls-certs\") pod \"stop-feature-test-kserve-7644fc97d-z9s46\" (UID: \"1ef9fb10-bd61-4107-8393-7a272150bd8d\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7644fc97d-z9s46"
Sep 09 21:48:17.856130 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:48:17.856069    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1ef9fb10-bd61-4107-8393-7a272150bd8d-kserve-provision-location\") pod \"stop-feature-test-kserve-7644fc97d-z9s46\" (UID: \"1ef9fb10-bd61-4107-8393-7a272150bd8d\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7644fc97d-z9s46"
Sep 09 21:48:17.856130 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:48:17.856108    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z974n\" (UniqueName: \"kubernetes.io/projected/1ef9fb10-bd61-4107-8393-7a272150bd8d-kube-api-access-z974n\") pod \"stop-feature-test-kserve-7644fc97d-z9s46\" (UID: \"1ef9fb10-bd61-4107-8393-7a272150bd8d\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7644fc97d-z9s46"
Sep 09 21:48:17.856273 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:48:17.856133    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ef9fb10-bd61-4107-8393-7a272150bd8d-home\") pod \"stop-feature-test-kserve-7644fc97d-z9s46\" (UID: \"1ef9fb10-bd61-4107-8393-7a272150bd8d\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7644fc97d-z9s46"
Sep 09 21:48:17.856273 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:48:17.856166    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ef9fb10-bd61-4107-8393-7a272150bd8d-tmp-dir\") pod \"stop-feature-test-kserve-7644fc97d-z9s46\" (UID: \"1ef9fb10-bd61-4107-8393-7a272150bd8d\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7644fc97d-z9s46"
Sep 09 21:48:17.856273 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:48:17.856198    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ef9fb10-bd61-4107-8393-7a272150bd8d-dshm\") pod \"stop-feature-test-kserve-7644fc97d-z9s46\" (UID: \"1ef9fb10-bd61-4107-8393-7a272150bd8d\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7644fc97d-z9s46"
Sep 09 21:48:17.957374 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:48:17.957324    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ef9fb10-bd61-4107-8393-7a272150bd8d-model-cache\") pod \"stop-feature-test-kserve-7644fc97d-z9s46\" (UID: \"1ef9fb10-bd61-4107-8393-7a272150bd8d\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7644fc97d-z9s46"
Sep 09 21:48:17.957547 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:48:17.957403    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1ef9fb10-bd61-4107-8393-7a272150bd8d-cabundle-cert\") pod \"stop-feature-test-kserve-7644fc97d-z9s46\" (UID: \"1ef9fb10-bd61-4107-8393-7a272150bd8d\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7644fc97d-z9s46"
Sep 09 21:48:17.957547 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:48:17.957434    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ef9fb10-bd61-4107-8393-7a272150bd8d-tls-certs\") pod \"stop-feature-test-kserve-7644fc97d-z9s46\" (UID: \"1ef9fb10-bd61-4107-8393-7a272150bd8d\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7644fc97d-z9s46"
Sep 09 21:48:17.957547 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:48:17.957457    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1ef9fb10-bd61-4107-8393-7a272150bd8d-kserve-provision-location\") pod \"stop-feature-test-kserve-7644fc97d-z9s46\" (UID: \"1ef9fb10-bd61-4107-8393-7a272150bd8d\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7644fc97d-z9s46"
Sep 09 21:48:17.957547 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:48:17.957487    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z974n\" (UniqueName: \"kubernetes.io/projected/1ef9fb10-bd61-4107-8393-7a272150bd8d-kube-api-access-z974n\") pod \"stop-feature-test-kserve-7644fc97d-z9s46\" (UID: \"1ef9fb10-bd61-4107-8393-7a272150bd8d\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7644fc97d-z9s46"
Sep 09 21:48:17.957742 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:48:17.957648    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ef9fb10-bd61-4107-8393-7a272150bd8d-home\") pod \"stop-feature-test-kserve-7644fc97d-z9s46\" (UID: \"1ef9fb10-bd61-4107-8393-7a272150bd8d\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7644fc97d-z9s46"
Sep 09 21:48:17.957742 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:48:17.957711    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ef9fb10-bd61-4107-8393-7a272150bd8d-tmp-dir\") pod \"stop-feature-test-kserve-7644fc97d-z9s46\" (UID: \"1ef9fb10-bd61-4107-8393-7a272150bd8d\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7644fc97d-z9s46"
Sep 09 21:48:17.957852 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:48:17.957785    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ef9fb10-bd61-4107-8393-7a272150bd8d-dshm\") pod \"stop-feature-test-kserve-7644fc97d-z9s46\" (UID: \"1ef9fb10-bd61-4107-8393-7a272150bd8d\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7644fc97d-z9s46"
Sep 09 21:48:17.957852 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:48:17.957784    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ef9fb10-bd61-4107-8393-7a272150bd8d-model-cache\") pod \"stop-feature-test-kserve-7644fc97d-z9s46\" (UID: \"1ef9fb10-bd61-4107-8393-7a272150bd8d\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7644fc97d-z9s46"
Sep 09 21:48:17.957940 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:48:17.957871    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1ef9fb10-bd61-4107-8393-7a272150bd8d-kserve-provision-location\") pod \"stop-feature-test-kserve-7644fc97d-z9s46\" (UID: \"1ef9fb10-bd61-4107-8393-7a272150bd8d\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7644fc97d-z9s46"
Sep 09 21:48:17.957976 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:48:17.957953    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ef9fb10-bd61-4107-8393-7a272150bd8d-home\") pod \"stop-feature-test-kserve-7644fc97d-z9s46\" (UID: \"1ef9fb10-bd61-4107-8393-7a272150bd8d\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7644fc97d-z9s46"
Sep 09 21:48:17.958034 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:48:17.958016    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ef9fb10-bd61-4107-8393-7a272150bd8d-tmp-dir\") pod \"stop-feature-test-kserve-7644fc97d-z9s46\" (UID: \"1ef9fb10-bd61-4107-8393-7a272150bd8d\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7644fc97d-z9s46"
Sep 09 21:48:17.958202 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:48:17.958171    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1ef9fb10-bd61-4107-8393-7a272150bd8d-cabundle-cert\") pod \"stop-feature-test-kserve-7644fc97d-z9s46\" (UID: \"1ef9fb10-bd61-4107-8393-7a272150bd8d\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7644fc97d-z9s46"
Sep 09 21:48:17.960196 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:48:17.960167    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ef9fb10-bd61-4107-8393-7a272150bd8d-dshm\") pod \"stop-feature-test-kserve-7644fc97d-z9s46\" (UID: \"1ef9fb10-bd61-4107-8393-7a272150bd8d\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7644fc97d-z9s46"
Sep 09 21:48:17.960436 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:48:17.960419    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ef9fb10-bd61-4107-8393-7a272150bd8d-tls-certs\") pod \"stop-feature-test-kserve-7644fc97d-z9s46\" (UID: \"1ef9fb10-bd61-4107-8393-7a272150bd8d\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7644fc97d-z9s46"
Sep 09 21:48:17.971483 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:48:17.971458    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z974n\" (UniqueName: \"kubernetes.io/projected/1ef9fb10-bd61-4107-8393-7a272150bd8d-kube-api-access-z974n\") pod \"stop-feature-test-kserve-7644fc97d-z9s46\" (UID: \"1ef9fb10-bd61-4107-8393-7a272150bd8d\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7644fc97d-z9s46"
Sep 09 21:48:18.067201 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:48:18.067123    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:48:18.090861 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:48:18.090828    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7644fc97d-z9s46"
Sep 09 21:48:18.386231 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:48:18.386200    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7644fc97d-z9s46"]
Sep 09 21:48:18.387425 ip-10-0-134-72 kubenswrapper[2576]: W0909 21:48:18.387391    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ef9fb10_bd61_4107_8393_7a272150bd8d.slice/crio-e4b09ad02153bf1d7f4ddc63b52443919b542139e8546afe5b5fd36b9b176c14 WatchSource:0}: Error finding container e4b09ad02153bf1d7f4ddc63b52443919b542139e8546afe5b5fd36b9b176c14: Status 404 returned error can't find the container with id e4b09ad02153bf1d7f4ddc63b52443919b542139e8546afe5b5fd36b9b176c14
Sep 09 21:48:18.518493 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:48:18.518460    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7644fc97d-z9s46" event={"ID":"1ef9fb10-bd61-4107-8393-7a272150bd8d","Type":"ContainerStarted","Data":"2e0d611d737f6c169af25da7bf987f46496a61a07fb5e008e17cc1dabcd0a2ef"}
Sep 09 21:48:18.518658 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:48:18.518510    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7644fc97d-z9s46" event={"ID":"1ef9fb10-bd61-4107-8393-7a272150bd8d","Type":"ContainerStarted","Data":"e4b09ad02153bf1d7f4ddc63b52443919b542139e8546afe5b5fd36b9b176c14"}
Sep 09 21:48:21.536985 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:48:21.536947    2576 generic.go:356] "Generic (PLEG): container finished" podID="1ef9fb10-bd61-4107-8393-7a272150bd8d" containerID="2e0d611d737f6c169af25da7bf987f46496a61a07fb5e008e17cc1dabcd0a2ef" exitCode=0
Sep 09 21:48:21.537473 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:48:21.536995    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7644fc97d-z9s46" event={"ID":"1ef9fb10-bd61-4107-8393-7a272150bd8d","Type":"ContainerDied","Data":"2e0d611d737f6c169af25da7bf987f46496a61a07fb5e008e17cc1dabcd0a2ef"}
Sep 09 21:48:22.303663 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:48:22.303615    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-557b9dcccf-csrxs" podUID="a3014647-a76e-4ffc-88f6-27206f6d5088" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": dial tcp 10.132.0.76:8000: connect: connection refused"
Sep 09 21:48:22.546004 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:48:22.545968    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7644fc97d-z9s46" event={"ID":"1ef9fb10-bd61-4107-8393-7a272150bd8d","Type":"ContainerStarted","Data":"ee7aa0cbda2d5ce991c5b90adfbbb6d7aa476898c273264d0e293ef7ccd4e550"}
Sep 09 21:48:22.570470 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:48:22.570329    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7644fc97d-z9s46" podStartSLOduration=3.305371089 podStartE2EDuration="5.570309195s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.264938106s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:48:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:48:22.567341674 +0000 UTC m=+3345.681027530" watchObservedRunningTime="2026-09-09 21:48:22.570309195 +0000 UTC m=+3345.683995056"
Sep 09 21:48:23.091352 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:48:23.091298    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7644fc97d-z9s46"
Sep 09 21:48:28.091527 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:48:28.091484    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7644fc97d-z9s46"
Sep 09 21:48:28.093078 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:48:28.093047    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7644fc97d-z9s46" podUID="1ef9fb10-bd61-4107-8393-7a272150bd8d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": dial tcp 10.132.0.80:8000: connect: connection refused"
Sep 09 21:48:32.313931 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:48:32.313897    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-557b9dcccf-csrxs"
Sep 09 21:48:32.330592 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:48:32.330559    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-557b9dcccf-csrxs"
Sep 09 21:48:38.091977 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:48:38.091938    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7644fc97d-z9s46" podUID="1ef9fb10-bd61-4107-8393-7a272150bd8d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": dial tcp 10.132.0.80:8000: connect: connection refused"
Sep 09 21:48:41.533652 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:48:41.533615    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:48:44.277216 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:48:44.277174    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:48:47.147167 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:48:47.147126    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:48:47.154575 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:48:47.154541    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-557b9dcccf-csrxs"]
Sep 09 21:48:47.209386 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:48:47.209331    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:48:48.091782 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:48:48.091727    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7644fc97d-z9s46" podUID="1ef9fb10-bd61-4107-8393-7a272150bd8d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": dial tcp 10.132.0.80:8000: connect: connection refused"
Sep 09 21:48:58.091656 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:48:58.091608    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7644fc97d-z9s46" podUID="1ef9fb10-bd61-4107-8393-7a272150bd8d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": dial tcp 10.132.0.80:8000: connect: connection refused"
Sep 09 21:49:02.180348 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:49:02.180301    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-557b9dcccf-csrxs" podUID="a3014647-a76e-4ffc-88f6-27206f6d5088" containerName="main" containerID="cri-o://501d770895d5807b95183744f8d5ecc83d89d2bc6450e3c10d96cb925e44f8dd" gracePeriod=45
Sep 09 21:49:02.315331 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:49:02.315271    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-557b9dcccf-csrxs" podUID="a3014647-a76e-4ffc-88f6-27206f6d5088" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": dial tcp 10.132.0.76:8000: connect: connection refused"
Sep 09 21:49:02.962865 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:49:02.962839    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-557b9dcccf-csrxs"
Sep 09 21:49:03.074869 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:49:03.074767    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3014647-a76e-4ffc-88f6-27206f6d5088-tmp-dir\") pod \"a3014647-a76e-4ffc-88f6-27206f6d5088\" (UID: \"a3014647-a76e-4ffc-88f6-27206f6d5088\") "
Sep 09 21:49:03.074869 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:49:03.074814    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3014647-a76e-4ffc-88f6-27206f6d5088-model-cache\") pod \"a3014647-a76e-4ffc-88f6-27206f6d5088\" (UID: \"a3014647-a76e-4ffc-88f6-27206f6d5088\") "
Sep 09 21:49:03.074869 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:49:03.074853    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a3014647-a76e-4ffc-88f6-27206f6d5088-kserve-provision-location\") pod \"a3014647-a76e-4ffc-88f6-27206f6d5088\" (UID: \"a3014647-a76e-4ffc-88f6-27206f6d5088\") "
Sep 09 21:49:03.075207 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:49:03.074876    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3014647-a76e-4ffc-88f6-27206f6d5088-dshm\") pod \"a3014647-a76e-4ffc-88f6-27206f6d5088\" (UID: \"a3014647-a76e-4ffc-88f6-27206f6d5088\") "
Sep 09 21:49:03.075207 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:49:03.074922    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3014647-a76e-4ffc-88f6-27206f6d5088-home\") pod \"a3014647-a76e-4ffc-88f6-27206f6d5088\" (UID: \"a3014647-a76e-4ffc-88f6-27206f6d5088\") "
Sep 09 21:49:03.075207 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:49:03.075000    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3014647-a76e-4ffc-88f6-27206f6d5088-tls-certs\") pod \"a3014647-a76e-4ffc-88f6-27206f6d5088\" (UID: \"a3014647-a76e-4ffc-88f6-27206f6d5088\") "
Sep 09 21:49:03.075207 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:49:03.075038    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m2h2k\" (UniqueName: \"kubernetes.io/projected/a3014647-a76e-4ffc-88f6-27206f6d5088-kube-api-access-m2h2k\") pod \"a3014647-a76e-4ffc-88f6-27206f6d5088\" (UID: \"a3014647-a76e-4ffc-88f6-27206f6d5088\") "
Sep 09 21:49:03.075207 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:49:03.075071    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a3014647-a76e-4ffc-88f6-27206f6d5088-cabundle-cert\") pod \"a3014647-a76e-4ffc-88f6-27206f6d5088\" (UID: \"a3014647-a76e-4ffc-88f6-27206f6d5088\") "
Sep 09 21:49:03.075207 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:49:03.075135    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3014647-a76e-4ffc-88f6-27206f6d5088-model-cache" (OuterVolumeSpecName: "model-cache") pod "a3014647-a76e-4ffc-88f6-27206f6d5088" (UID: "a3014647-a76e-4ffc-88f6-27206f6d5088"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:49:03.075563 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:49:03.075423    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3014647-a76e-4ffc-88f6-27206f6d5088-model-cache\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:49:03.076023 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:49:03.075996    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3014647-a76e-4ffc-88f6-27206f6d5088-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a3014647-a76e-4ffc-88f6-27206f6d5088" (UID: "a3014647-a76e-4ffc-88f6-27206f6d5088"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:49:03.076188 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:49:03.076049    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3014647-a76e-4ffc-88f6-27206f6d5088-home" (OuterVolumeSpecName: "home") pod "a3014647-a76e-4ffc-88f6-27206f6d5088" (UID: "a3014647-a76e-4ffc-88f6-27206f6d5088"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:49:03.077777 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:49:03.077689    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3014647-a76e-4ffc-88f6-27206f6d5088-dshm" (OuterVolumeSpecName: "dshm") pod "a3014647-a76e-4ffc-88f6-27206f6d5088" (UID: "a3014647-a76e-4ffc-88f6-27206f6d5088"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:49:03.078154 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:49:03.078126    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3014647-a76e-4ffc-88f6-27206f6d5088-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a3014647-a76e-4ffc-88f6-27206f6d5088" (UID: "a3014647-a76e-4ffc-88f6-27206f6d5088"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:49:03.078427 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:49:03.078400    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3014647-a76e-4ffc-88f6-27206f6d5088-kube-api-access-m2h2k" (OuterVolumeSpecName: "kube-api-access-m2h2k") pod "a3014647-a76e-4ffc-88f6-27206f6d5088" (UID: "a3014647-a76e-4ffc-88f6-27206f6d5088"). InnerVolumeSpecName "kube-api-access-m2h2k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:49:03.090617 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:49:03.090584    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3014647-a76e-4ffc-88f6-27206f6d5088-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a3014647-a76e-4ffc-88f6-27206f6d5088" (UID: "a3014647-a76e-4ffc-88f6-27206f6d5088"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:49:03.090828 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:49:03.090805    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3014647-a76e-4ffc-88f6-27206f6d5088-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a3014647-a76e-4ffc-88f6-27206f6d5088" (UID: "a3014647-a76e-4ffc-88f6-27206f6d5088"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:49:03.176080 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:49:03.176047    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3014647-a76e-4ffc-88f6-27206f6d5088-home\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:49:03.176080 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:49:03.176078    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3014647-a76e-4ffc-88f6-27206f6d5088-tls-certs\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:49:03.176080 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:49:03.176088    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2h2k\" (UniqueName: \"kubernetes.io/projected/a3014647-a76e-4ffc-88f6-27206f6d5088-kube-api-access-m2h2k\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:49:03.176319 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:49:03.176098    2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a3014647-a76e-4ffc-88f6-27206f6d5088-cabundle-cert\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:49:03.176319 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:49:03.176127    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3014647-a76e-4ffc-88f6-27206f6d5088-tmp-dir\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:49:03.176319 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:49:03.176135    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a3014647-a76e-4ffc-88f6-27206f6d5088-kserve-provision-location\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:49:03.176319 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:49:03.176143    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3014647-a76e-4ffc-88f6-27206f6d5088-dshm\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:49:03.788077 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:49:03.787989    2576 generic.go:356] "Generic (PLEG): container finished" podID="a3014647-a76e-4ffc-88f6-27206f6d5088" containerID="501d770895d5807b95183744f8d5ecc83d89d2bc6450e3c10d96cb925e44f8dd" exitCode=0
Sep 09 21:49:03.788077 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:49:03.788056    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-557b9dcccf-csrxs" event={"ID":"a3014647-a76e-4ffc-88f6-27206f6d5088","Type":"ContainerDied","Data":"501d770895d5807b95183744f8d5ecc83d89d2bc6450e3c10d96cb925e44f8dd"}
Sep 09 21:49:03.788555 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:49:03.788076    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-557b9dcccf-csrxs"
Sep 09 21:49:03.788555 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:49:03.788105    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-557b9dcccf-csrxs" event={"ID":"a3014647-a76e-4ffc-88f6-27206f6d5088","Type":"ContainerDied","Data":"77178d015d38763a02db829d60f4ea887a21032a67f7881de5391f5b0c04fa75"}
Sep 09 21:49:03.788555 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:49:03.788127    2576 scope.go:117] "RemoveContainer" containerID="501d770895d5807b95183744f8d5ecc83d89d2bc6450e3c10d96cb925e44f8dd"
Sep 09 21:49:03.798103 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:49:03.798079    2576 scope.go:117] "RemoveContainer" containerID="89353801495d897348aa12f447399c0d6cc335baa15c96263067f0cee2bdb450"
Sep 09 21:49:03.806084 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:49:03.806066    2576 scope.go:117] "RemoveContainer" containerID="501d770895d5807b95183744f8d5ecc83d89d2bc6450e3c10d96cb925e44f8dd"
Sep 09 21:49:03.806578 ip-10-0-134-72 kubenswrapper[2576]: E0909 21:49:03.806552    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"501d770895d5807b95183744f8d5ecc83d89d2bc6450e3c10d96cb925e44f8dd\": container with ID starting with 501d770895d5807b95183744f8d5ecc83d89d2bc6450e3c10d96cb925e44f8dd not found: ID does not exist" containerID="501d770895d5807b95183744f8d5ecc83d89d2bc6450e3c10d96cb925e44f8dd"
Sep 09 21:49:03.806678 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:49:03.806589    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"501d770895d5807b95183744f8d5ecc83d89d2bc6450e3c10d96cb925e44f8dd"} err="failed to get container status \"501d770895d5807b95183744f8d5ecc83d89d2bc6450e3c10d96cb925e44f8dd\": rpc error: code = NotFound desc = could not find container \"501d770895d5807b95183744f8d5ecc83d89d2bc6450e3c10d96cb925e44f8dd\": container with ID starting with 501d770895d5807b95183744f8d5ecc83d89d2bc6450e3c10d96cb925e44f8dd not found: ID does not exist"
Sep 09 21:49:03.806678 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:49:03.806613    2576 scope.go:117] "RemoveContainer" containerID="89353801495d897348aa12f447399c0d6cc335baa15c96263067f0cee2bdb450"
Sep 09 21:49:03.806955 ip-10-0-134-72 kubenswrapper[2576]: E0909 21:49:03.806935    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89353801495d897348aa12f447399c0d6cc335baa15c96263067f0cee2bdb450\": container with ID starting with 89353801495d897348aa12f447399c0d6cc335baa15c96263067f0cee2bdb450 not found: ID does not exist" containerID="89353801495d897348aa12f447399c0d6cc335baa15c96263067f0cee2bdb450"
Sep 09 21:49:03.807043 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:49:03.806964    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89353801495d897348aa12f447399c0d6cc335baa15c96263067f0cee2bdb450"} err="failed to get container status \"89353801495d897348aa12f447399c0d6cc335baa15c96263067f0cee2bdb450\": rpc error: code = NotFound desc = could not find container \"89353801495d897348aa12f447399c0d6cc335baa15c96263067f0cee2bdb450\": container with ID starting with 89353801495d897348aa12f447399c0d6cc335baa15c96263067f0cee2bdb450 not found: ID does not exist"
Sep 09 21:49:03.809190 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:49:03.809168    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-557b9dcccf-csrxs"]
Sep 09 21:49:03.813532 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:49:03.813512    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-557b9dcccf-csrxs"]
Sep 09 21:49:04.917042 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:49:04.916999    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:49:05.409980 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:49:05.409946    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a3014647-a76e-4ffc-88f6-27206f6d5088" path="/var/lib/kubelet/pods/a3014647-a76e-4ffc-88f6-27206f6d5088/volumes"
Sep 09 21:49:08.092155 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:49:08.092117    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7644fc97d-z9s46" podUID="1ef9fb10-bd61-4107-8393-7a272150bd8d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": dial tcp 10.132.0.80:8000: connect: connection refused"
Sep 09 21:49:18.091614 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:49:18.091571    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7644fc97d-z9s46" podUID="1ef9fb10-bd61-4107-8393-7a272150bd8d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": dial tcp 10.132.0.80:8000: connect: connection refused"
Sep 09 21:49:24.102224 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:49:24.102171    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:49:24.229586 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:49:24.229545    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:49:24.332741 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:49:24.332708    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:49:28.092322 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:49:28.092279    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7644fc97d-z9s46" podUID="1ef9fb10-bd61-4107-8393-7a272150bd8d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": dial tcp 10.132.0.80:8000: connect: connection refused"
Sep 09 21:49:37.443801 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:49:37.443757    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:49:37.525833 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:49:37.525801    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:49:37.588603 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:49:37.588560    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:49:38.092038 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:49:38.091984    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7644fc97d-z9s46" podUID="1ef9fb10-bd61-4107-8393-7a272150bd8d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": dial tcp 10.132.0.80:8000: connect: connection refused"
Sep 09 21:49:40.840820 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:49:40.840769    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:49:44.024572 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:49:44.024520    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:49:48.091413 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:49:48.091331    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7644fc97d-z9s46" podUID="1ef9fb10-bd61-4107-8393-7a272150bd8d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": dial tcp 10.132.0.80:8000: connect: connection refused"
Sep 09 21:49:49.803338 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:49:49.803285    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:49:49.861890 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:49:49.861852    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:49:53.861585 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:49:53.861546    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5c69d4c4c5-jpqmz"]
Sep 09 21:49:53.862068 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:49:53.861901    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3014647-a76e-4ffc-88f6-27206f6d5088" containerName="storage-initializer"
Sep 09 21:49:53.862068 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:49:53.861916    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3014647-a76e-4ffc-88f6-27206f6d5088" containerName="storage-initializer"
Sep 09 21:49:53.862068 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:49:53.861941    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3014647-a76e-4ffc-88f6-27206f6d5088" containerName="main"
Sep 09 21:49:53.862068 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:49:53.861947    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3014647-a76e-4ffc-88f6-27206f6d5088" containerName="main"
Sep 09 21:49:53.862068 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:49:53.862001    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3014647-a76e-4ffc-88f6-27206f6d5088" containerName="main"
Sep 09 21:49:53.865177 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:49:53.865154    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5c69d4c4c5-jpqmz"
Sep 09 21:49:53.867666 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:49:53.867645    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"odh-kserve-custom-ca-bundle\""
Sep 09 21:49:53.867882 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:49:53.867863    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"kube-root-ca.crt\""
Sep 09 21:49:53.867978 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:49:53.867956    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"openshift-service-ca.crt\""
Sep 09 21:49:53.868033 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:49:53.867975    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"router-with-refs-test-kserve-self-signed-certs\""
Sep 09 21:49:53.868087 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:49:53.868034    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"default-dockercfg-nzfp4\""
Sep 09 21:49:53.868087 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:49:53.867977    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"seaweedfs-s3-creds\""
Sep 09 21:49:53.877722 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:49:53.877703    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5c69d4c4c5-jpqmz"]
Sep 09 21:49:53.894061 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:49:53.894031    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:49:53.906927 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:49:53.906898    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/040118fb-9722-4e0d-99fb-c636fb82c3dd-dshm\") pod \"router-with-refs-test-kserve-5c69d4c4c5-jpqmz\" (UID: \"040118fb-9722-4e0d-99fb-c636fb82c3dd\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5c69d4c4c5-jpqmz"
Sep 09 21:49:53.907043 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:49:53.906942    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/040118fb-9722-4e0d-99fb-c636fb82c3dd-model-cache\") pod \"router-with-refs-test-kserve-5c69d4c4c5-jpqmz\" (UID: \"040118fb-9722-4e0d-99fb-c636fb82c3dd\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5c69d4c4c5-jpqmz"
Sep 09 21:49:53.907043 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:49:53.906972    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/040118fb-9722-4e0d-99fb-c636fb82c3dd-kserve-provision-location\") pod \"router-with-refs-test-kserve-5c69d4c4c5-jpqmz\" (UID: \"040118fb-9722-4e0d-99fb-c636fb82c3dd\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5c69d4c4c5-jpqmz"
Sep 09 21:49:53.907043 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:49:53.907029    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/040118fb-9722-4e0d-99fb-c636fb82c3dd-home\") pod \"router-with-refs-test-kserve-5c69d4c4c5-jpqmz\" (UID: \"040118fb-9722-4e0d-99fb-c636fb82c3dd\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5c69d4c4c5-jpqmz"
Sep 09 21:49:53.907179 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:49:53.907072    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7qs4\" (UniqueName: \"kubernetes.io/projected/040118fb-9722-4e0d-99fb-c636fb82c3dd-kube-api-access-n7qs4\") pod \"router-with-refs-test-kserve-5c69d4c4c5-jpqmz\" (UID: \"040118fb-9722-4e0d-99fb-c636fb82c3dd\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5c69d4c4c5-jpqmz"
Sep 09 21:49:53.907179 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:49:53.907103    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/040118fb-9722-4e0d-99fb-c636fb82c3dd-cabundle-cert\") pod \"router-with-refs-test-kserve-5c69d4c4c5-jpqmz\" (UID: \"040118fb-9722-4e0d-99fb-c636fb82c3dd\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5c69d4c4c5-jpqmz"
Sep 09 21:49:53.907179 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:49:53.907121    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/040118fb-9722-4e0d-99fb-c636fb82c3dd-tls-certs\") pod \"router-with-refs-test-kserve-5c69d4c4c5-jpqmz\" (UID: \"040118fb-9722-4e0d-99fb-c636fb82c3dd\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5c69d4c4c5-jpqmz"
Sep 09 21:49:53.907179 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:49:53.907149    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/040118fb-9722-4e0d-99fb-c636fb82c3dd-tmp-dir\") pod \"router-with-refs-test-kserve-5c69d4c4c5-jpqmz\" (UID: \"040118fb-9722-4e0d-99fb-c636fb82c3dd\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5c69d4c4c5-jpqmz"
Sep 09 21:49:54.007531 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:49:54.007500    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/040118fb-9722-4e0d-99fb-c636fb82c3dd-dshm\") pod \"router-with-refs-test-kserve-5c69d4c4c5-jpqmz\" (UID: \"040118fb-9722-4e0d-99fb-c636fb82c3dd\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5c69d4c4c5-jpqmz"
Sep 09 21:49:54.007531 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:49:54.007536    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/040118fb-9722-4e0d-99fb-c636fb82c3dd-model-cache\") pod \"router-with-refs-test-kserve-5c69d4c4c5-jpqmz\" (UID: \"040118fb-9722-4e0d-99fb-c636fb82c3dd\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5c69d4c4c5-jpqmz"
Sep 09 21:49:54.007729 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:49:54.007552    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/040118fb-9722-4e0d-99fb-c636fb82c3dd-kserve-provision-location\") pod \"router-with-refs-test-kserve-5c69d4c4c5-jpqmz\" (UID: \"040118fb-9722-4e0d-99fb-c636fb82c3dd\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5c69d4c4c5-jpqmz"
Sep 09 21:49:54.007729 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:49:54.007667    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/040118fb-9722-4e0d-99fb-c636fb82c3dd-home\") pod \"router-with-refs-test-kserve-5c69d4c4c5-jpqmz\" (UID: \"040118fb-9722-4e0d-99fb-c636fb82c3dd\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5c69d4c4c5-jpqmz"
Sep 09 21:49:54.007729 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:49:54.007712    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7qs4\" (UniqueName: \"kubernetes.io/projected/040118fb-9722-4e0d-99fb-c636fb82c3dd-kube-api-access-n7qs4\") pod \"router-with-refs-test-kserve-5c69d4c4c5-jpqmz\" (UID: \"040118fb-9722-4e0d-99fb-c636fb82c3dd\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5c69d4c4c5-jpqmz"
Sep 09 21:49:54.007872 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:49:54.007754    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/040118fb-9722-4e0d-99fb-c636fb82c3dd-cabundle-cert\") pod \"router-with-refs-test-kserve-5c69d4c4c5-jpqmz\" (UID: \"040118fb-9722-4e0d-99fb-c636fb82c3dd\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5c69d4c4c5-jpqmz"
Sep 09 21:49:54.007872 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:49:54.007783    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/040118fb-9722-4e0d-99fb-c636fb82c3dd-tls-certs\") pod \"router-with-refs-test-kserve-5c69d4c4c5-jpqmz\" (UID: \"040118fb-9722-4e0d-99fb-c636fb82c3dd\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5c69d4c4c5-jpqmz"
Sep 09 21:49:54.007872 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:49:54.007825    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/040118fb-9722-4e0d-99fb-c636fb82c3dd-tmp-dir\") pod \"router-with-refs-test-kserve-5c69d4c4c5-jpqmz\" (UID: \"040118fb-9722-4e0d-99fb-c636fb82c3dd\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5c69d4c4c5-jpqmz"
Sep 09 21:49:54.008048 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:49:54.007926    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/040118fb-9722-4e0d-99fb-c636fb82c3dd-model-cache\") pod \"router-with-refs-test-kserve-5c69d4c4c5-jpqmz\" (UID: \"040118fb-9722-4e0d-99fb-c636fb82c3dd\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5c69d4c4c5-jpqmz"
Sep 09 21:49:54.008048 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:49:54.008019    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/040118fb-9722-4e0d-99fb-c636fb82c3dd-kserve-provision-location\") pod \"router-with-refs-test-kserve-5c69d4c4c5-jpqmz\" (UID: \"040118fb-9722-4e0d-99fb-c636fb82c3dd\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5c69d4c4c5-jpqmz"
Sep 09 21:49:54.008160 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:49:54.008059    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/040118fb-9722-4e0d-99fb-c636fb82c3dd-home\") pod \"router-with-refs-test-kserve-5c69d4c4c5-jpqmz\" (UID: \"040118fb-9722-4e0d-99fb-c636fb82c3dd\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5c69d4c4c5-jpqmz"
Sep 09 21:49:54.008209 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:49:54.008175    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/040118fb-9722-4e0d-99fb-c636fb82c3dd-tmp-dir\") pod \"router-with-refs-test-kserve-5c69d4c4c5-jpqmz\" (UID: \"040118fb-9722-4e0d-99fb-c636fb82c3dd\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5c69d4c4c5-jpqmz"
Sep 09 21:49:54.008537 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:49:54.008504    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/040118fb-9722-4e0d-99fb-c636fb82c3dd-cabundle-cert\") pod \"router-with-refs-test-kserve-5c69d4c4c5-jpqmz\" (UID: \"040118fb-9722-4e0d-99fb-c636fb82c3dd\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5c69d4c4c5-jpqmz"
Sep 09 21:49:54.010089 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:49:54.010067    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/040118fb-9722-4e0d-99fb-c636fb82c3dd-dshm\") pod \"router-with-refs-test-kserve-5c69d4c4c5-jpqmz\" (UID: \"040118fb-9722-4e0d-99fb-c636fb82c3dd\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5c69d4c4c5-jpqmz"
Sep 09 21:49:54.010723 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:49:54.010702    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/040118fb-9722-4e0d-99fb-c636fb82c3dd-tls-certs\") pod \"router-with-refs-test-kserve-5c69d4c4c5-jpqmz\" (UID: \"040118fb-9722-4e0d-99fb-c636fb82c3dd\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5c69d4c4c5-jpqmz"
Sep 09 21:49:54.016573 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:49:54.016542    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7qs4\" (UniqueName: \"kubernetes.io/projected/040118fb-9722-4e0d-99fb-c636fb82c3dd-kube-api-access-n7qs4\") pod \"router-with-refs-test-kserve-5c69d4c4c5-jpqmz\" (UID: \"040118fb-9722-4e0d-99fb-c636fb82c3dd\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5c69d4c4c5-jpqmz"
Sep 09 21:49:54.176788 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:49:54.176708    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5c69d4c4c5-jpqmz"
Sep 09 21:49:54.361490 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:49:54.361455    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:49:54.455842 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:49:54.455807    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5c69d4c4c5-jpqmz"]
Sep 09 21:49:54.456234 ip-10-0-134-72 kubenswrapper[2576]: W0909 21:49:54.456208    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod040118fb_9722_4e0d_99fb_c636fb82c3dd.slice/crio-ec30bd02a41d796beed73934f1dd733459795e12dd4964b6744af161e874c024 WatchSource:0}: Error finding container ec30bd02a41d796beed73934f1dd733459795e12dd4964b6744af161e874c024: Status 404 returned error can't find the container with id ec30bd02a41d796beed73934f1dd733459795e12dd4964b6744af161e874c024
Sep 09 21:49:54.472255 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:49:54.472221    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:49:55.071187 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:49:55.071138    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5c69d4c4c5-jpqmz" event={"ID":"040118fb-9722-4e0d-99fb-c636fb82c3dd","Type":"ContainerStarted","Data":"37a2b4f84a7b57fdfcc9c7a6aaa211027f052fdba37be058ed20f9e8a0a2ed20"}
Sep 09 21:49:55.071635 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:49:55.071245    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5c69d4c4c5-jpqmz" event={"ID":"040118fb-9722-4e0d-99fb-c636fb82c3dd","Type":"ContainerStarted","Data":"ec30bd02a41d796beed73934f1dd733459795e12dd4964b6744af161e874c024"}
Sep 09 21:49:56.741402 ip-10-0-134-72 kubenswrapper[2576]: E0909 21:49:56.741359    2576 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod040118fb_9722_4e0d_99fb_c636fb82c3dd.slice/crio-37a2b4f84a7b57fdfcc9c7a6aaa211027f052fdba37be058ed20f9e8a0a2ed20.scope\": RecentStats: unable to find data in memory cache]"
Sep 09 21:49:57.084720 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:49:57.084627    2576 generic.go:356] "Generic (PLEG): container finished" podID="040118fb-9722-4e0d-99fb-c636fb82c3dd" containerID="37a2b4f84a7b57fdfcc9c7a6aaa211027f052fdba37be058ed20f9e8a0a2ed20" exitCode=0
Sep 09 21:49:57.084891 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:49:57.084708    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5c69d4c4c5-jpqmz" event={"ID":"040118fb-9722-4e0d-99fb-c636fb82c3dd","Type":"ContainerDied","Data":"37a2b4f84a7b57fdfcc9c7a6aaa211027f052fdba37be058ed20f9e8a0a2ed20"}
Sep 09 21:49:58.091689 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:49:58.091644    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7644fc97d-z9s46" podUID="1ef9fb10-bd61-4107-8393-7a272150bd8d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": dial tcp 10.132.0.80:8000: connect: connection refused"
Sep 09 21:49:58.093908 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:49:58.093885    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5c69d4c4c5-jpqmz" event={"ID":"040118fb-9722-4e0d-99fb-c636fb82c3dd","Type":"ContainerStarted","Data":"03e1cf7ed4d0ed3089dd73d0ce42e4731a53484ae73145a9018f7b130f3cffe4"}
Sep 09 21:49:58.118780 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:49:58.118722    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5c69d4c4c5-jpqmz" podStartSLOduration=2.919892971 podStartE2EDuration="5.118703032s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.198810061s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:49:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:49:58.114765216 +0000 UTC m=+3441.228451075" watchObservedRunningTime="2026-09-09 21:49:58.118703032 +0000 UTC m=+3441.232388893"
Sep 09 21:49:58.177542 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:49:58.177499    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5c69d4c4c5-jpqmz"
Sep 09 21:50:04.177397 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:50:04.177323    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5c69d4c4c5-jpqmz"
Sep 09 21:50:04.179166 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:50:04.179126    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5c69d4c4c5-jpqmz" podUID="040118fb-9722-4e0d-99fb-c636fb82c3dd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/health\": dial tcp 10.132.0.81:8000: connect: connection refused"
Sep 09 21:50:08.091288 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:50:08.091242    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7644fc97d-z9s46" podUID="1ef9fb10-bd61-4107-8393-7a272150bd8d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": dial tcp 10.132.0.80:8000: connect: connection refused"
Sep 09 21:50:14.178187 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:50:14.178132    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5c69d4c4c5-jpqmz" podUID="040118fb-9722-4e0d-99fb-c636fb82c3dd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/health\": dial tcp 10.132.0.81:8000: connect: connection refused"
Sep 09 21:50:18.102280 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:50:18.102242    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7644fc97d-z9s46"
Sep 09 21:50:18.110580 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:50:18.110555    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7644fc97d-z9s46"
Sep 09 21:50:19.988328 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:50:19.988294    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7644fc97d-z9s46"]
Sep 09 21:50:20.005411 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:50:20.005378    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:50:20.105225 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:50:20.105183    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:50:20.162025 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:50:20.161986    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:50:24.177415 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:50:24.177356    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5c69d4c4c5-jpqmz" podUID="040118fb-9722-4e0d-99fb-c636fb82c3dd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/health\": dial tcp 10.132.0.81:8000: connect: connection refused"
Sep 09 21:50:29.613244 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:50:29.613205    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:50:29.779490 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:50:29.779456    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:50:33.992396 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:50:33.992342    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:50:34.093425 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:50:34.093393    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:50:34.177167 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:50:34.177122    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5c69d4c4c5-jpqmz" podUID="040118fb-9722-4e0d-99fb-c636fb82c3dd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/health\": dial tcp 10.132.0.81:8000: connect: connection refused"
Sep 09 21:50:35.012455 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:50:35.012403    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7644fc97d-z9s46" podUID="1ef9fb10-bd61-4107-8393-7a272150bd8d" containerName="main" containerID="cri-o://ee7aa0cbda2d5ce991c5b90adfbbb6d7aa476898c273264d0e293ef7ccd4e550" gracePeriod=45
Sep 09 21:50:35.103634 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:50:35.103586    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7644fc97d-z9s46" podUID="1ef9fb10-bd61-4107-8393-7a272150bd8d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": dial tcp 10.132.0.80:8000: connect: connection refused"
Sep 09 21:50:35.769695 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:50:35.769670    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7644fc97d-z9s46"
Sep 09 21:50:35.788209 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:50:35.788177    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z974n\" (UniqueName: \"kubernetes.io/projected/1ef9fb10-bd61-4107-8393-7a272150bd8d-kube-api-access-z974n\") pod \"1ef9fb10-bd61-4107-8393-7a272150bd8d\" (UID: \"1ef9fb10-bd61-4107-8393-7a272150bd8d\") "
Sep 09 21:50:35.788463 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:50:35.788219    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ef9fb10-bd61-4107-8393-7a272150bd8d-tls-certs\") pod \"1ef9fb10-bd61-4107-8393-7a272150bd8d\" (UID: \"1ef9fb10-bd61-4107-8393-7a272150bd8d\") "
Sep 09 21:50:35.788463 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:50:35.788238    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ef9fb10-bd61-4107-8393-7a272150bd8d-home\") pod \"1ef9fb10-bd61-4107-8393-7a272150bd8d\" (UID: \"1ef9fb10-bd61-4107-8393-7a272150bd8d\") "
Sep 09 21:50:35.788463 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:50:35.788266    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ef9fb10-bd61-4107-8393-7a272150bd8d-tmp-dir\") pod \"1ef9fb10-bd61-4107-8393-7a272150bd8d\" (UID: \"1ef9fb10-bd61-4107-8393-7a272150bd8d\") "
Sep 09 21:50:35.788463 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:50:35.788294    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1ef9fb10-bd61-4107-8393-7a272150bd8d-kserve-provision-location\") pod \"1ef9fb10-bd61-4107-8393-7a272150bd8d\" (UID: \"1ef9fb10-bd61-4107-8393-7a272150bd8d\") "
Sep 09 21:50:35.788463 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:50:35.788328    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ef9fb10-bd61-4107-8393-7a272150bd8d-model-cache\") pod \"1ef9fb10-bd61-4107-8393-7a272150bd8d\" (UID: \"1ef9fb10-bd61-4107-8393-7a272150bd8d\") "
Sep 09 21:50:35.788463 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:50:35.788355    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1ef9fb10-bd61-4107-8393-7a272150bd8d-cabundle-cert\") pod \"1ef9fb10-bd61-4107-8393-7a272150bd8d\" (UID: \"1ef9fb10-bd61-4107-8393-7a272150bd8d\") "
Sep 09 21:50:35.788912 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:50:35.788883    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ef9fb10-bd61-4107-8393-7a272150bd8d-model-cache" (OuterVolumeSpecName: "model-cache") pod "1ef9fb10-bd61-4107-8393-7a272150bd8d" (UID: "1ef9fb10-bd61-4107-8393-7a272150bd8d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:50:35.788967 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:50:35.788888    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ef9fb10-bd61-4107-8393-7a272150bd8d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "1ef9fb10-bd61-4107-8393-7a272150bd8d" (UID: "1ef9fb10-bd61-4107-8393-7a272150bd8d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:50:35.789032 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:50:35.788991    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ef9fb10-bd61-4107-8393-7a272150bd8d-home" (OuterVolumeSpecName: "home") pod "1ef9fb10-bd61-4107-8393-7a272150bd8d" (UID: "1ef9fb10-bd61-4107-8393-7a272150bd8d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:50:35.791209 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:50:35.791182    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ef9fb10-bd61-4107-8393-7a272150bd8d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1ef9fb10-bd61-4107-8393-7a272150bd8d" (UID: "1ef9fb10-bd61-4107-8393-7a272150bd8d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:50:35.791209 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:50:35.791196    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ef9fb10-bd61-4107-8393-7a272150bd8d-kube-api-access-z974n" (OuterVolumeSpecName: "kube-api-access-z974n") pod "1ef9fb10-bd61-4107-8393-7a272150bd8d" (UID: "1ef9fb10-bd61-4107-8393-7a272150bd8d"). InnerVolumeSpecName "kube-api-access-z974n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:50:35.805965 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:50:35.805932    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ef9fb10-bd61-4107-8393-7a272150bd8d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1ef9fb10-bd61-4107-8393-7a272150bd8d" (UID: "1ef9fb10-bd61-4107-8393-7a272150bd8d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:50:35.819787 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:50:35.819692    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ef9fb10-bd61-4107-8393-7a272150bd8d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1ef9fb10-bd61-4107-8393-7a272150bd8d" (UID: "1ef9fb10-bd61-4107-8393-7a272150bd8d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:50:35.889324 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:50:35.889293    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ef9fb10-bd61-4107-8393-7a272150bd8d-dshm\") pod \"1ef9fb10-bd61-4107-8393-7a272150bd8d\" (UID: \"1ef9fb10-bd61-4107-8393-7a272150bd8d\") "
Sep 09 21:50:35.889518 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:50:35.889502    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z974n\" (UniqueName: \"kubernetes.io/projected/1ef9fb10-bd61-4107-8393-7a272150bd8d-kube-api-access-z974n\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:50:35.889563 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:50:35.889523    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ef9fb10-bd61-4107-8393-7a272150bd8d-tls-certs\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:50:35.889563 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:50:35.889533    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ef9fb10-bd61-4107-8393-7a272150bd8d-home\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:50:35.889563 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:50:35.889541    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ef9fb10-bd61-4107-8393-7a272150bd8d-tmp-dir\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:50:35.889563 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:50:35.889549    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1ef9fb10-bd61-4107-8393-7a272150bd8d-kserve-provision-location\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:50:35.889563 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:50:35.889557    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ef9fb10-bd61-4107-8393-7a272150bd8d-model-cache\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:50:35.889563 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:50:35.889564    2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1ef9fb10-bd61-4107-8393-7a272150bd8d-cabundle-cert\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:50:35.891411 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:50:35.891388    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ef9fb10-bd61-4107-8393-7a272150bd8d-dshm" (OuterVolumeSpecName: "dshm") pod "1ef9fb10-bd61-4107-8393-7a272150bd8d" (UID: "1ef9fb10-bd61-4107-8393-7a272150bd8d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:50:35.990073 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:50:35.990047    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ef9fb10-bd61-4107-8393-7a272150bd8d-dshm\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:50:36.311646 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:50:36.311603    2576 generic.go:356] "Generic (PLEG): container finished" podID="1ef9fb10-bd61-4107-8393-7a272150bd8d" containerID="ee7aa0cbda2d5ce991c5b90adfbbb6d7aa476898c273264d0e293ef7ccd4e550" exitCode=0
Sep 09 21:50:36.312039 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:50:36.311656    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7644fc97d-z9s46"
Sep 09 21:50:36.312039 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:50:36.311668    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7644fc97d-z9s46" event={"ID":"1ef9fb10-bd61-4107-8393-7a272150bd8d","Type":"ContainerDied","Data":"ee7aa0cbda2d5ce991c5b90adfbbb6d7aa476898c273264d0e293ef7ccd4e550"}
Sep 09 21:50:36.312039 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:50:36.311726    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7644fc97d-z9s46" event={"ID":"1ef9fb10-bd61-4107-8393-7a272150bd8d","Type":"ContainerDied","Data":"e4b09ad02153bf1d7f4ddc63b52443919b542139e8546afe5b5fd36b9b176c14"}
Sep 09 21:50:36.312039 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:50:36.311757    2576 scope.go:117] "RemoveContainer" containerID="ee7aa0cbda2d5ce991c5b90adfbbb6d7aa476898c273264d0e293ef7ccd4e550"
Sep 09 21:50:36.322339 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:50:36.322319    2576 scope.go:117] "RemoveContainer" containerID="2e0d611d737f6c169af25da7bf987f46496a61a07fb5e008e17cc1dabcd0a2ef"
Sep 09 21:50:36.330338 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:50:36.330321    2576 scope.go:117] "RemoveContainer" containerID="ee7aa0cbda2d5ce991c5b90adfbbb6d7aa476898c273264d0e293ef7ccd4e550"
Sep 09 21:50:36.330667 ip-10-0-134-72 kubenswrapper[2576]: E0909 21:50:36.330647    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee7aa0cbda2d5ce991c5b90adfbbb6d7aa476898c273264d0e293ef7ccd4e550\": container with ID starting with ee7aa0cbda2d5ce991c5b90adfbbb6d7aa476898c273264d0e293ef7ccd4e550 not found: ID does not exist" containerID="ee7aa0cbda2d5ce991c5b90adfbbb6d7aa476898c273264d0e293ef7ccd4e550"
Sep 09 21:50:36.330745 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:50:36.330672    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee7aa0cbda2d5ce991c5b90adfbbb6d7aa476898c273264d0e293ef7ccd4e550"} err="failed to get container status \"ee7aa0cbda2d5ce991c5b90adfbbb6d7aa476898c273264d0e293ef7ccd4e550\": rpc error: code = NotFound desc = could not find container \"ee7aa0cbda2d5ce991c5b90adfbbb6d7aa476898c273264d0e293ef7ccd4e550\": container with ID starting with ee7aa0cbda2d5ce991c5b90adfbbb6d7aa476898c273264d0e293ef7ccd4e550 not found: ID does not exist"
Sep 09 21:50:36.330745 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:50:36.330686    2576 scope.go:117] "RemoveContainer" containerID="2e0d611d737f6c169af25da7bf987f46496a61a07fb5e008e17cc1dabcd0a2ef"
Sep 09 21:50:36.330931 ip-10-0-134-72 kubenswrapper[2576]: E0909 21:50:36.330915    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e0d611d737f6c169af25da7bf987f46496a61a07fb5e008e17cc1dabcd0a2ef\": container with ID starting with 2e0d611d737f6c169af25da7bf987f46496a61a07fb5e008e17cc1dabcd0a2ef not found: ID does not exist" containerID="2e0d611d737f6c169af25da7bf987f46496a61a07fb5e008e17cc1dabcd0a2ef"
Sep 09 21:50:36.330971 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:50:36.330936    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e0d611d737f6c169af25da7bf987f46496a61a07fb5e008e17cc1dabcd0a2ef"} err="failed to get container status \"2e0d611d737f6c169af25da7bf987f46496a61a07fb5e008e17cc1dabcd0a2ef\": rpc error: code = NotFound desc = could not find container \"2e0d611d737f6c169af25da7bf987f46496a61a07fb5e008e17cc1dabcd0a2ef\": container with ID starting with 2e0d611d737f6c169af25da7bf987f46496a61a07fb5e008e17cc1dabcd0a2ef not found: ID does not exist"
Sep 09 21:50:36.335683 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:50:36.335660    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7644fc97d-z9s46"]
Sep 09 21:50:36.340197 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:50:36.340175    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7644fc97d-z9s46"]
Sep 09 21:50:37.409710 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:50:37.409677    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1ef9fb10-bd61-4107-8393-7a272150bd8d" path="/var/lib/kubelet/pods/1ef9fb10-bd61-4107-8393-7a272150bd8d/volumes"
Sep 09 21:50:38.341967 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:50:38.341926    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:50:44.178043 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:50:44.177999    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5c69d4c4c5-jpqmz" podUID="040118fb-9722-4e0d-99fb-c636fb82c3dd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/health\": dial tcp 10.132.0.81:8000: connect: connection refused"
Sep 09 21:50:54.178156 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:50:54.178111    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5c69d4c4c5-jpqmz" podUID="040118fb-9722-4e0d-99fb-c636fb82c3dd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/health\": dial tcp 10.132.0.81:8000: connect: connection refused"
Sep 09 21:51:04.177420 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:51:04.177358    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5c69d4c4c5-jpqmz" podUID="040118fb-9722-4e0d-99fb-c636fb82c3dd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/health\": dial tcp 10.132.0.81:8000: connect: connection refused"
Sep 09 21:51:14.177444 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:51:14.177395    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5c69d4c4c5-jpqmz" podUID="040118fb-9722-4e0d-99fb-c636fb82c3dd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/health\": dial tcp 10.132.0.81:8000: connect: connection refused"
Sep 09 21:51:24.177970 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:51:24.177924    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5c69d4c4c5-jpqmz" podUID="040118fb-9722-4e0d-99fb-c636fb82c3dd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/health\": dial tcp 10.132.0.81:8000: connect: connection refused"
Sep 09 21:51:34.178062 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:51:34.178020    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5c69d4c4c5-jpqmz" podUID="040118fb-9722-4e0d-99fb-c636fb82c3dd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/health\": dial tcp 10.132.0.81:8000: connect: connection refused"
Sep 09 21:51:36.944952 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:51:36.944905    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:51:44.177909 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:51:44.177855    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5c69d4c4c5-jpqmz" podUID="040118fb-9722-4e0d-99fb-c636fb82c3dd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/health\": dial tcp 10.132.0.81:8000: connect: connection refused"
Sep 09 21:51:54.187427 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:51:54.187387    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5c69d4c4c5-jpqmz"
Sep 09 21:51:54.194986 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:51:54.194965    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5c69d4c4c5-jpqmz"
Sep 09 21:51:58.542981 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:51:58.542942    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:51:58.553155 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:51:58.553120    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5c69d4c4c5-jpqmz"]
Sep 09 21:51:58.619456 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:51:58.619421    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:52:13.576963 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:52:13.576906    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5c69d4c4c5-jpqmz" podUID="040118fb-9722-4e0d-99fb-c636fb82c3dd" containerName="main" containerID="cri-o://03e1cf7ed4d0ed3089dd73d0ce42e4731a53484ae73145a9018f7b130f3cffe4" gracePeriod=45
Sep 09 21:52:14.188779 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:52:14.188742    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5c69d4c4c5-jpqmz" podUID="040118fb-9722-4e0d-99fb-c636fb82c3dd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/health\": dial tcp 10.132.0.81:8000: connect: connection refused"
Sep 09 21:52:14.323089 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:52:14.323065    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5c69d4c4c5-jpqmz"
Sep 09 21:52:14.445825 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:52:14.445742    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/040118fb-9722-4e0d-99fb-c636fb82c3dd-cabundle-cert\") pod \"040118fb-9722-4e0d-99fb-c636fb82c3dd\" (UID: \"040118fb-9722-4e0d-99fb-c636fb82c3dd\") "
Sep 09 21:52:14.445825 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:52:14.445796    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n7qs4\" (UniqueName: \"kubernetes.io/projected/040118fb-9722-4e0d-99fb-c636fb82c3dd-kube-api-access-n7qs4\") pod \"040118fb-9722-4e0d-99fb-c636fb82c3dd\" (UID: \"040118fb-9722-4e0d-99fb-c636fb82c3dd\") "
Sep 09 21:52:14.445825 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:52:14.445824    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/040118fb-9722-4e0d-99fb-c636fb82c3dd-tls-certs\") pod \"040118fb-9722-4e0d-99fb-c636fb82c3dd\" (UID: \"040118fb-9722-4e0d-99fb-c636fb82c3dd\") "
Sep 09 21:52:14.446100 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:52:14.445861    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/040118fb-9722-4e0d-99fb-c636fb82c3dd-dshm\") pod \"040118fb-9722-4e0d-99fb-c636fb82c3dd\" (UID: \"040118fb-9722-4e0d-99fb-c636fb82c3dd\") "
Sep 09 21:52:14.446100 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:52:14.445893    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/040118fb-9722-4e0d-99fb-c636fb82c3dd-kserve-provision-location\") pod \"040118fb-9722-4e0d-99fb-c636fb82c3dd\" (UID: \"040118fb-9722-4e0d-99fb-c636fb82c3dd\") "
Sep 09 21:52:14.446100 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:52:14.445939    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/040118fb-9722-4e0d-99fb-c636fb82c3dd-model-cache\") pod \"040118fb-9722-4e0d-99fb-c636fb82c3dd\" (UID: \"040118fb-9722-4e0d-99fb-c636fb82c3dd\") "
Sep 09 21:52:14.446100 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:52:14.445970    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/040118fb-9722-4e0d-99fb-c636fb82c3dd-home\") pod \"040118fb-9722-4e0d-99fb-c636fb82c3dd\" (UID: \"040118fb-9722-4e0d-99fb-c636fb82c3dd\") "
Sep 09 21:52:14.446100 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:52:14.446003    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/040118fb-9722-4e0d-99fb-c636fb82c3dd-tmp-dir\") pod \"040118fb-9722-4e0d-99fb-c636fb82c3dd\" (UID: \"040118fb-9722-4e0d-99fb-c636fb82c3dd\") "
Sep 09 21:52:14.446397 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:52:14.446183    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/040118fb-9722-4e0d-99fb-c636fb82c3dd-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "040118fb-9722-4e0d-99fb-c636fb82c3dd" (UID: "040118fb-9722-4e0d-99fb-c636fb82c3dd"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:52:14.446397 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:52:14.446235    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/040118fb-9722-4e0d-99fb-c636fb82c3dd-model-cache" (OuterVolumeSpecName: "model-cache") pod "040118fb-9722-4e0d-99fb-c636fb82c3dd" (UID: "040118fb-9722-4e0d-99fb-c636fb82c3dd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:52:14.447083 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:52:14.447051    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/040118fb-9722-4e0d-99fb-c636fb82c3dd-home" (OuterVolumeSpecName: "home") pod "040118fb-9722-4e0d-99fb-c636fb82c3dd" (UID: "040118fb-9722-4e0d-99fb-c636fb82c3dd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:52:14.448357 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:52:14.448331    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/040118fb-9722-4e0d-99fb-c636fb82c3dd-dshm" (OuterVolumeSpecName: "dshm") pod "040118fb-9722-4e0d-99fb-c636fb82c3dd" (UID: "040118fb-9722-4e0d-99fb-c636fb82c3dd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:52:14.448505 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:52:14.448484    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/040118fb-9722-4e0d-99fb-c636fb82c3dd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "040118fb-9722-4e0d-99fb-c636fb82c3dd" (UID: "040118fb-9722-4e0d-99fb-c636fb82c3dd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:52:14.448785 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:52:14.448769    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/040118fb-9722-4e0d-99fb-c636fb82c3dd-kube-api-access-n7qs4" (OuterVolumeSpecName: "kube-api-access-n7qs4") pod "040118fb-9722-4e0d-99fb-c636fb82c3dd" (UID: "040118fb-9722-4e0d-99fb-c636fb82c3dd"). InnerVolumeSpecName "kube-api-access-n7qs4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:52:14.460050 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:52:14.460002    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/040118fb-9722-4e0d-99fb-c636fb82c3dd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "040118fb-9722-4e0d-99fb-c636fb82c3dd" (UID: "040118fb-9722-4e0d-99fb-c636fb82c3dd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:52:14.467056 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:52:14.467034    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/040118fb-9722-4e0d-99fb-c636fb82c3dd-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "040118fb-9722-4e0d-99fb-c636fb82c3dd" (UID: "040118fb-9722-4e0d-99fb-c636fb82c3dd"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:52:14.547025 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:52:14.547000    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/040118fb-9722-4e0d-99fb-c636fb82c3dd-model-cache\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:52:14.547025 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:52:14.547024    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/040118fb-9722-4e0d-99fb-c636fb82c3dd-home\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:52:14.547175 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:52:14.547034    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/040118fb-9722-4e0d-99fb-c636fb82c3dd-tmp-dir\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:52:14.547175 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:52:14.547041    2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/040118fb-9722-4e0d-99fb-c636fb82c3dd-cabundle-cert\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:52:14.547175 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:52:14.547049    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7qs4\" (UniqueName: \"kubernetes.io/projected/040118fb-9722-4e0d-99fb-c636fb82c3dd-kube-api-access-n7qs4\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:52:14.547175 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:52:14.547060    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/040118fb-9722-4e0d-99fb-c636fb82c3dd-tls-certs\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:52:14.547175 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:52:14.547068    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/040118fb-9722-4e0d-99fb-c636fb82c3dd-dshm\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:52:14.547175 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:52:14.547076    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/040118fb-9722-4e0d-99fb-c636fb82c3dd-kserve-provision-location\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:52:14.843388 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:52:14.843333    2576 generic.go:356] "Generic (PLEG): container finished" podID="040118fb-9722-4e0d-99fb-c636fb82c3dd" containerID="03e1cf7ed4d0ed3089dd73d0ce42e4731a53484ae73145a9018f7b130f3cffe4" exitCode=0
Sep 09 21:52:14.843820 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:52:14.843408    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5c69d4c4c5-jpqmz"
Sep 09 21:52:14.843820 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:52:14.843423    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5c69d4c4c5-jpqmz" event={"ID":"040118fb-9722-4e0d-99fb-c636fb82c3dd","Type":"ContainerDied","Data":"03e1cf7ed4d0ed3089dd73d0ce42e4731a53484ae73145a9018f7b130f3cffe4"}
Sep 09 21:52:14.843820 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:52:14.843486    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5c69d4c4c5-jpqmz" event={"ID":"040118fb-9722-4e0d-99fb-c636fb82c3dd","Type":"ContainerDied","Data":"ec30bd02a41d796beed73934f1dd733459795e12dd4964b6744af161e874c024"}
Sep 09 21:52:14.843820 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:52:14.843516    2576 scope.go:117] "RemoveContainer" containerID="03e1cf7ed4d0ed3089dd73d0ce42e4731a53484ae73145a9018f7b130f3cffe4"
Sep 09 21:52:14.853052 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:52:14.853031    2576 scope.go:117] "RemoveContainer" containerID="37a2b4f84a7b57fdfcc9c7a6aaa211027f052fdba37be058ed20f9e8a0a2ed20"
Sep 09 21:52:14.861694 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:52:14.861676    2576 scope.go:117] "RemoveContainer" containerID="03e1cf7ed4d0ed3089dd73d0ce42e4731a53484ae73145a9018f7b130f3cffe4"
Sep 09 21:52:14.861964 ip-10-0-134-72 kubenswrapper[2576]: E0909 21:52:14.861946    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03e1cf7ed4d0ed3089dd73d0ce42e4731a53484ae73145a9018f7b130f3cffe4\": container with ID starting with 03e1cf7ed4d0ed3089dd73d0ce42e4731a53484ae73145a9018f7b130f3cffe4 not found: ID does not exist" containerID="03e1cf7ed4d0ed3089dd73d0ce42e4731a53484ae73145a9018f7b130f3cffe4"
Sep 09 21:52:14.862027 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:52:14.861972    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03e1cf7ed4d0ed3089dd73d0ce42e4731a53484ae73145a9018f7b130f3cffe4"} err="failed to get container status \"03e1cf7ed4d0ed3089dd73d0ce42e4731a53484ae73145a9018f7b130f3cffe4\": rpc error: code = NotFound desc = could not find container \"03e1cf7ed4d0ed3089dd73d0ce42e4731a53484ae73145a9018f7b130f3cffe4\": container with ID starting with 03e1cf7ed4d0ed3089dd73d0ce42e4731a53484ae73145a9018f7b130f3cffe4 not found: ID does not exist"
Sep 09 21:52:14.862027 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:52:14.861988    2576 scope.go:117] "RemoveContainer" containerID="37a2b4f84a7b57fdfcc9c7a6aaa211027f052fdba37be058ed20f9e8a0a2ed20"
Sep 09 21:52:14.862252 ip-10-0-134-72 kubenswrapper[2576]: E0909 21:52:14.862236    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37a2b4f84a7b57fdfcc9c7a6aaa211027f052fdba37be058ed20f9e8a0a2ed20\": container with ID starting with 37a2b4f84a7b57fdfcc9c7a6aaa211027f052fdba37be058ed20f9e8a0a2ed20 not found: ID does not exist" containerID="37a2b4f84a7b57fdfcc9c7a6aaa211027f052fdba37be058ed20f9e8a0a2ed20"
Sep 09 21:52:14.862297 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:52:14.862259    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37a2b4f84a7b57fdfcc9c7a6aaa211027f052fdba37be058ed20f9e8a0a2ed20"} err="failed to get container status \"37a2b4f84a7b57fdfcc9c7a6aaa211027f052fdba37be058ed20f9e8a0a2ed20\": rpc error: code = NotFound desc = could not find container \"37a2b4f84a7b57fdfcc9c7a6aaa211027f052fdba37be058ed20f9e8a0a2ed20\": container with ID starting with 37a2b4f84a7b57fdfcc9c7a6aaa211027f052fdba37be058ed20f9e8a0a2ed20 not found: ID does not exist"
Sep 09 21:52:14.865878 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:52:14.865856    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5c69d4c4c5-jpqmz"]
Sep 09 21:52:14.869647 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:52:14.869628    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5c69d4c4c5-jpqmz"]
Sep 09 21:52:15.410634 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:52:15.410599    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="040118fb-9722-4e0d-99fb-c636fb82c3dd" path="/var/lib/kubelet/pods/040118fb-9722-4e0d-99fb-c636fb82c3dd/volumes"
Sep 09 21:52:31.435148 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:52:31.435058    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:52:31.537947 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:52:31.537914    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:52:40.921338 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:52:40.921305    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:52:48.759850 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:52:48.759810    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:52:48.859090 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:52:48.859058    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:52:53.755159 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:52:53.755128    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:52:54.041640 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:52:54.041558    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-dbf8fcf48-7n8hn"]
Sep 09 21:52:54.041940 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:52:54.041922    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="040118fb-9722-4e0d-99fb-c636fb82c3dd" containerName="main"
Sep 09 21:52:54.042032 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:52:54.041945    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="040118fb-9722-4e0d-99fb-c636fb82c3dd" containerName="main"
Sep 09 21:52:54.042032 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:52:54.041962    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ef9fb10-bd61-4107-8393-7a272150bd8d" containerName="main"
Sep 09 21:52:54.042032 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:52:54.041970    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ef9fb10-bd61-4107-8393-7a272150bd8d" containerName="main"
Sep 09 21:52:54.042032 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:52:54.041990    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ef9fb10-bd61-4107-8393-7a272150bd8d" containerName="storage-initializer"
Sep 09 21:52:54.042032 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:52:54.041998    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ef9fb10-bd61-4107-8393-7a272150bd8d" containerName="storage-initializer"
Sep 09 21:52:54.042032 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:52:54.042011    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="040118fb-9722-4e0d-99fb-c636fb82c3dd" containerName="storage-initializer"
Sep 09 21:52:54.042032 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:52:54.042018    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="040118fb-9722-4e0d-99fb-c636fb82c3dd" containerName="storage-initializer"
Sep 09 21:52:54.042405 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:52:54.042094    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="040118fb-9722-4e0d-99fb-c636fb82c3dd" containerName="main"
Sep 09 21:52:54.042405 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:52:54.042107    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ef9fb10-bd61-4107-8393-7a272150bd8d" containerName="main"
Sep 09 21:52:54.045376 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:52:54.045341    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-dbf8fcf48-7n8hn"
Sep 09 21:52:54.045693 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:52:54.045632    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f9vfp5"]
Sep 09 21:52:54.047753 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:52:54.047715    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"seaweedfs-s3-creds\""
Sep 09 21:52:54.047922 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:52:54.047896    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"openshift-service-ca.crt\""
Sep 09 21:52:54.048057 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:52:54.048036    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-dockercfg-2kblx\""
Sep 09 21:52:54.048131 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:52:54.047734    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"kube-root-ca.crt\""
Sep 09 21:52:54.048131 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:52:54.047770    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"default-dockercfg-kbmx8\""
Sep 09 21:52:54.048268 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:52:54.048246    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"odh-kserve-custom-ca-bundle\""
Sep 09 21:52:54.048503 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:52:54.048485    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisv44d181485fad85e662eb092f3749502f-kserve-self-signed-certs\""
Sep 09 21:52:54.049131 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:52:54.049118    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f9vfp5"
Sep 09 21:52:54.060982 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:52:54.060959    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-dbf8fcf48-7n8hn"]
Sep 09 21:52:54.078196 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:52:54.078169    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f9vfp5"]
Sep 09 21:52:54.089288 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:52:54.089243    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:52:54.160004 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:52:54.159973    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/22f4548d-6510-4dda-9f67-cce061fc1686-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f9vfp5\" (UID: \"22f4548d-6510-4dda-9f67-cce061fc1686\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f9vfp5"
Sep 09 21:52:54.160004 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:52:54.160004    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22f4548d-6510-4dda-9f67-cce061fc1686-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f9vfp5\" (UID: \"22f4548d-6510-4dda-9f67-cce061fc1686\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f9vfp5"
Sep 09 21:52:54.160175 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:52:54.160029    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/77e7a640-a9c5-414f-92a4-fda5f4467ffe-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-dbf8fcf48-7n8hn\" (UID: \"77e7a640-a9c5-414f-92a4-fda5f4467ffe\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-dbf8fcf48-7n8hn"
Sep 09 21:52:54.160175 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:52:54.160056    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22f4548d-6510-4dda-9f67-cce061fc1686-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f9vfp5\" (UID: \"22f4548d-6510-4dda-9f67-cce061fc1686\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f9vfp5"
Sep 09 21:52:54.160175 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:52:54.160102    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22f4548d-6510-4dda-9f67-cce061fc1686-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f9vfp5\" (UID: \"22f4548d-6510-4dda-9f67-cce061fc1686\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f9vfp5"
Sep 09 21:52:54.160175 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:52:54.160169    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77e7a640-a9c5-414f-92a4-fda5f4467ffe-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-dbf8fcf48-7n8hn\" (UID: \"77e7a640-a9c5-414f-92a4-fda5f4467ffe\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-dbf8fcf48-7n8hn"
Sep 09 21:52:54.160309 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:52:54.160188    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77e7a640-a9c5-414f-92a4-fda5f4467ffe-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-dbf8fcf48-7n8hn\" (UID: \"77e7a640-a9c5-414f-92a4-fda5f4467ffe\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-dbf8fcf48-7n8hn"
Sep 09 21:52:54.160309 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:52:54.160217    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/22f4548d-6510-4dda-9f67-cce061fc1686-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f9vfp5\" (UID: \"22f4548d-6510-4dda-9f67-cce061fc1686\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f9vfp5"
Sep 09 21:52:54.160309 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:52:54.160238    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77e7a640-a9c5-414f-92a4-fda5f4467ffe-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-dbf8fcf48-7n8hn\" (UID: \"77e7a640-a9c5-414f-92a4-fda5f4467ffe\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-dbf8fcf48-7n8hn"
Sep 09 21:52:54.160309 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:52:54.160287    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22f4548d-6510-4dda-9f67-cce061fc1686-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f9vfp5\" (UID: \"22f4548d-6510-4dda-9f67-cce061fc1686\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f9vfp5"
Sep 09 21:52:54.160309 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:52:54.160307    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22f4548d-6510-4dda-9f67-cce061fc1686-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f9vfp5\" (UID: \"22f4548d-6510-4dda-9f67-cce061fc1686\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f9vfp5"
Sep 09 21:52:54.160517 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:52:54.160328    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kbg8\" (UniqueName: \"kubernetes.io/projected/22f4548d-6510-4dda-9f67-cce061fc1686-kube-api-access-8kbg8\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f9vfp5\" (UID: \"22f4548d-6510-4dda-9f67-cce061fc1686\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f9vfp5"
Sep 09 21:52:54.160517 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:52:54.160359    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77e7a640-a9c5-414f-92a4-fda5f4467ffe-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-dbf8fcf48-7n8hn\" (UID: \"77e7a640-a9c5-414f-92a4-fda5f4467ffe\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-dbf8fcf48-7n8hn"
Sep 09 21:52:54.160517 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:52:54.160413    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/77e7a640-a9c5-414f-92a4-fda5f4467ffe-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-dbf8fcf48-7n8hn\" (UID: \"77e7a640-a9c5-414f-92a4-fda5f4467ffe\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-dbf8fcf48-7n8hn"
Sep 09 21:52:54.160517 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:52:54.160439    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77e7a640-a9c5-414f-92a4-fda5f4467ffe-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-dbf8fcf48-7n8hn\" (UID: \"77e7a640-a9c5-414f-92a4-fda5f4467ffe\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-dbf8fcf48-7n8hn"
Sep 09 21:52:54.160517 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:52:54.160483    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8zph\" (UniqueName: \"kubernetes.io/projected/77e7a640-a9c5-414f-92a4-fda5f4467ffe-kube-api-access-p8zph\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-dbf8fcf48-7n8hn\" (UID: \"77e7a640-a9c5-414f-92a4-fda5f4467ffe\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-dbf8fcf48-7n8hn"
Sep 09 21:52:54.260800 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:52:54.260767    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77e7a640-a9c5-414f-92a4-fda5f4467ffe-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-dbf8fcf48-7n8hn\" (UID: \"77e7a640-a9c5-414f-92a4-fda5f4467ffe\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-dbf8fcf48-7n8hn"
Sep 09 21:52:54.260800 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:52:54.260801    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77e7a640-a9c5-414f-92a4-fda5f4467ffe-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-dbf8fcf48-7n8hn\" (UID: \"77e7a640-a9c5-414f-92a4-fda5f4467ffe\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-dbf8fcf48-7n8hn"
Sep 09 21:52:54.260998 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:52:54.260823    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/22f4548d-6510-4dda-9f67-cce061fc1686-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f9vfp5\" (UID: \"22f4548d-6510-4dda-9f67-cce061fc1686\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f9vfp5"
Sep 09 21:52:54.260998 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:52:54.260842    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77e7a640-a9c5-414f-92a4-fda5f4467ffe-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-dbf8fcf48-7n8hn\" (UID: \"77e7a640-a9c5-414f-92a4-fda5f4467ffe\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-dbf8fcf48-7n8hn"
Sep 09 21:52:54.260998 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:52:54.260869    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22f4548d-6510-4dda-9f67-cce061fc1686-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f9vfp5\" (UID: \"22f4548d-6510-4dda-9f67-cce061fc1686\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f9vfp5"
Sep 09 21:52:54.260998 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:52:54.260895    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22f4548d-6510-4dda-9f67-cce061fc1686-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f9vfp5\" (UID: \"22f4548d-6510-4dda-9f67-cce061fc1686\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f9vfp5"
Sep 09 21:52:54.260998 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:52:54.260921    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kbg8\" (UniqueName: \"kubernetes.io/projected/22f4548d-6510-4dda-9f67-cce061fc1686-kube-api-access-8kbg8\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f9vfp5\" (UID: \"22f4548d-6510-4dda-9f67-cce061fc1686\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f9vfp5"
Sep 09 21:52:54.260998 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:52:54.260944    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77e7a640-a9c5-414f-92a4-fda5f4467ffe-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-dbf8fcf48-7n8hn\" (UID: \"77e7a640-a9c5-414f-92a4-fda5f4467ffe\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-dbf8fcf48-7n8hn"
Sep 09 21:52:54.260998 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:52:54.260967    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/77e7a640-a9c5-414f-92a4-fda5f4467ffe-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-dbf8fcf48-7n8hn\" (UID: \"77e7a640-a9c5-414f-92a4-fda5f4467ffe\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-dbf8fcf48-7n8hn"
Sep 09 21:52:54.260998 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:52:54.260990    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77e7a640-a9c5-414f-92a4-fda5f4467ffe-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-dbf8fcf48-7n8hn\" (UID: \"77e7a640-a9c5-414f-92a4-fda5f4467ffe\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-dbf8fcf48-7n8hn"
Sep 09 21:52:54.261562 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:52:54.261007    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8zph\" (UniqueName: \"kubernetes.io/projected/77e7a640-a9c5-414f-92a4-fda5f4467ffe-kube-api-access-p8zph\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-dbf8fcf48-7n8hn\" (UID: \"77e7a640-a9c5-414f-92a4-fda5f4467ffe\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-dbf8fcf48-7n8hn"
Sep 09 21:52:54.261562 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:52:54.261029    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/22f4548d-6510-4dda-9f67-cce061fc1686-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f9vfp5\" (UID: \"22f4548d-6510-4dda-9f67-cce061fc1686\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f9vfp5"
Sep 09 21:52:54.261562 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:52:54.261055    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22f4548d-6510-4dda-9f67-cce061fc1686-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f9vfp5\" (UID: \"22f4548d-6510-4dda-9f67-cce061fc1686\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f9vfp5"
Sep 09 21:52:54.261562 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:52:54.261073    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/77e7a640-a9c5-414f-92a4-fda5f4467ffe-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-dbf8fcf48-7n8hn\" (UID: \"77e7a640-a9c5-414f-92a4-fda5f4467ffe\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-dbf8fcf48-7n8hn"
Sep 09 21:52:54.261562 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:52:54.261090    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22f4548d-6510-4dda-9f67-cce061fc1686-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f9vfp5\" (UID: \"22f4548d-6510-4dda-9f67-cce061fc1686\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f9vfp5"
Sep 09 21:52:54.261562 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:52:54.261105    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22f4548d-6510-4dda-9f67-cce061fc1686-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f9vfp5\" (UID: \"22f4548d-6510-4dda-9f67-cce061fc1686\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f9vfp5"
Sep 09 21:52:54.261562 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:52:54.261195    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/22f4548d-6510-4dda-9f67-cce061fc1686-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f9vfp5\" (UID: \"22f4548d-6510-4dda-9f67-cce061fc1686\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f9vfp5"
Sep 09 21:52:54.261562 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:52:54.261218    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77e7a640-a9c5-414f-92a4-fda5f4467ffe-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-dbf8fcf48-7n8hn\" (UID: \"77e7a640-a9c5-414f-92a4-fda5f4467ffe\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-dbf8fcf48-7n8hn"
Sep 09 21:52:54.261562 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:52:54.261277    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77e7a640-a9c5-414f-92a4-fda5f4467ffe-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-dbf8fcf48-7n8hn\" (UID: \"77e7a640-a9c5-414f-92a4-fda5f4467ffe\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-dbf8fcf48-7n8hn"
Sep 09 21:52:54.262005 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:52:54.261591    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22f4548d-6510-4dda-9f67-cce061fc1686-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f9vfp5\" (UID: \"22f4548d-6510-4dda-9f67-cce061fc1686\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f9vfp5"
Sep 09 21:52:54.262005 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:52:54.261600    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/77e7a640-a9c5-414f-92a4-fda5f4467ffe-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-dbf8fcf48-7n8hn\" (UID: \"77e7a640-a9c5-414f-92a4-fda5f4467ffe\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-dbf8fcf48-7n8hn"
Sep 09 21:52:54.262005 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:52:54.261830    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77e7a640-a9c5-414f-92a4-fda5f4467ffe-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-dbf8fcf48-7n8hn\" (UID: \"77e7a640-a9c5-414f-92a4-fda5f4467ffe\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-dbf8fcf48-7n8hn"
Sep 09 21:52:54.262005 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:52:54.261878    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22f4548d-6510-4dda-9f67-cce061fc1686-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f9vfp5\" (UID: \"22f4548d-6510-4dda-9f67-cce061fc1686\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f9vfp5"
Sep 09 21:52:54.262208 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:52:54.262185    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22f4548d-6510-4dda-9f67-cce061fc1686-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f9vfp5\" (UID: \"22f4548d-6510-4dda-9f67-cce061fc1686\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f9vfp5"
Sep 09 21:52:54.262664 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:52:54.262636    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/77e7a640-a9c5-414f-92a4-fda5f4467ffe-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-dbf8fcf48-7n8hn\" (UID: \"77e7a640-a9c5-414f-92a4-fda5f4467ffe\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-dbf8fcf48-7n8hn"
Sep 09 21:52:54.262792 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:52:54.262736    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/22f4548d-6510-4dda-9f67-cce061fc1686-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f9vfp5\" (UID: \"22f4548d-6510-4dda-9f67-cce061fc1686\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f9vfp5"
Sep 09 21:52:54.263544 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:52:54.263514    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77e7a640-a9c5-414f-92a4-fda5f4467ffe-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-dbf8fcf48-7n8hn\" (UID: \"77e7a640-a9c5-414f-92a4-fda5f4467ffe\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-dbf8fcf48-7n8hn"
Sep 09 21:52:54.263829 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:52:54.263812    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22f4548d-6510-4dda-9f67-cce061fc1686-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f9vfp5\" (UID: \"22f4548d-6510-4dda-9f67-cce061fc1686\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f9vfp5"
Sep 09 21:52:54.263943 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:52:54.263924    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22f4548d-6510-4dda-9f67-cce061fc1686-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f9vfp5\" (UID: \"22f4548d-6510-4dda-9f67-cce061fc1686\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f9vfp5"
Sep 09 21:52:54.264459 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:52:54.264439    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77e7a640-a9c5-414f-92a4-fda5f4467ffe-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-dbf8fcf48-7n8hn\" (UID: \"77e7a640-a9c5-414f-92a4-fda5f4467ffe\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-dbf8fcf48-7n8hn"
Sep 09 21:52:54.269242 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:52:54.269219    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8zph\" (UniqueName: \"kubernetes.io/projected/77e7a640-a9c5-414f-92a4-fda5f4467ffe-kube-api-access-p8zph\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-dbf8fcf48-7n8hn\" (UID: \"77e7a640-a9c5-414f-92a4-fda5f4467ffe\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-dbf8fcf48-7n8hn"
Sep 09 21:52:54.269349 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:52:54.269332    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kbg8\" (UniqueName: \"kubernetes.io/projected/22f4548d-6510-4dda-9f67-cce061fc1686-kube-api-access-8kbg8\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f9vfp5\" (UID: \"22f4548d-6510-4dda-9f67-cce061fc1686\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f9vfp5"
Sep 09 21:52:54.344027 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:52:54.343990    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:52:54.356608 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:52:54.356582    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-dbf8fcf48-7n8hn"
Sep 09 21:52:54.363880 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:52:54.363865    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f9vfp5"
Sep 09 21:52:54.640617 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:52:54.640590    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-dbf8fcf48-7n8hn"]
Sep 09 21:52:54.641738 ip-10-0-134-72 kubenswrapper[2576]: W0909 21:52:54.641615    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77e7a640_a9c5_414f_92a4_fda5f4467ffe.slice/crio-8b35dc10b9df055a66924272118ca717d8f7774953b99f8a8be75508aab72ed5 WatchSource:0}: Error finding container 8b35dc10b9df055a66924272118ca717d8f7774953b99f8a8be75508aab72ed5: Status 404 returned error can't find the container with id 8b35dc10b9df055a66924272118ca717d8f7774953b99f8a8be75508aab72ed5
Sep 09 21:52:54.643121 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:52:54.643096    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f9vfp5"]
Sep 09 21:52:54.644105 ip-10-0-134-72 kubenswrapper[2576]: W0909 21:52:54.644054    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22f4548d_6510_4dda_9f67_cce061fc1686.slice/crio-a847dcaaf0dc141c0db0747dd622df50c5ed0166c6bae614f8e94bd1222d6b7b WatchSource:0}: Error finding container a847dcaaf0dc141c0db0747dd622df50c5ed0166c6bae614f8e94bd1222d6b7b: Status 404 returned error can't find the container with id a847dcaaf0dc141c0db0747dd622df50c5ed0166c6bae614f8e94bd1222d6b7b
Sep 09 21:52:55.046647 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:52:55.046554    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f9vfp5" event={"ID":"22f4548d-6510-4dda-9f67-cce061fc1686","Type":"ContainerStarted","Data":"9b865adb658402471de367a2e75d068722dba9d4e0e965e3ae8931f6391b9fd7"}
Sep 09 21:52:55.046647 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:52:55.046622    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f9vfp5" event={"ID":"22f4548d-6510-4dda-9f67-cce061fc1686","Type":"ContainerStarted","Data":"a847dcaaf0dc141c0db0747dd622df50c5ed0166c6bae614f8e94bd1222d6b7b"}
Sep 09 21:52:55.048433 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:52:55.048401    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-dbf8fcf48-7n8hn" event={"ID":"77e7a640-a9c5-414f-92a4-fda5f4467ffe","Type":"ContainerStarted","Data":"0bdb3864f8eeff8b14d4e1438becc389547cf69a6901ee0273618702145962ba"}
Sep 09 21:52:55.048557 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:52:55.048460    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-dbf8fcf48-7n8hn" event={"ID":"77e7a640-a9c5-414f-92a4-fda5f4467ffe","Type":"ContainerStarted","Data":"8b35dc10b9df055a66924272118ca717d8f7774953b99f8a8be75508aab72ed5"}
Sep 09 21:52:55.048557 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:52:55.048494    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-dbf8fcf48-7n8hn"
Sep 09 21:52:56.061611 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:52:56.061570    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-dbf8fcf48-7n8hn" event={"ID":"77e7a640-a9c5-414f-92a4-fda5f4467ffe","Type":"ContainerStarted","Data":"00a1dbe6c6c28c4a4c69e3711c9ca7fabf3606fad554c714257843ad8e5e62d9"}
Sep 09 21:52:58.073898 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:52:58.073861    2576 generic.go:356] "Generic (PLEG): container finished" podID="22f4548d-6510-4dda-9f67-cce061fc1686" containerID="9b865adb658402471de367a2e75d068722dba9d4e0e965e3ae8931f6391b9fd7" exitCode=0
Sep 09 21:52:58.074300 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:52:58.073930    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f9vfp5" event={"ID":"22f4548d-6510-4dda-9f67-cce061fc1686","Type":"ContainerDied","Data":"9b865adb658402471de367a2e75d068722dba9d4e0e965e3ae8931f6391b9fd7"}
Sep 09 21:52:58.075881 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:52:58.075858    2576 generic.go:356] "Generic (PLEG): container finished" podID="77e7a640-a9c5-414f-92a4-fda5f4467ffe" containerID="00a1dbe6c6c28c4a4c69e3711c9ca7fabf3606fad554c714257843ad8e5e62d9" exitCode=0
Sep 09 21:52:58.075990 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:52:58.075899    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-dbf8fcf48-7n8hn" event={"ID":"77e7a640-a9c5-414f-92a4-fda5f4467ffe","Type":"ContainerDied","Data":"00a1dbe6c6c28c4a4c69e3711c9ca7fabf3606fad554c714257843ad8e5e62d9"}
Sep 09 21:52:59.085100 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:52:59.085058    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f9vfp5" event={"ID":"22f4548d-6510-4dda-9f67-cce061fc1686","Type":"ContainerStarted","Data":"369c9b6c27eb282acd36f18710ddb2ff4533cd97eadc38620d46cc0c1ba7fb2b"}
Sep 09 21:52:59.087617 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:52:59.087587    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-dbf8fcf48-7n8hn" event={"ID":"77e7a640-a9c5-414f-92a4-fda5f4467ffe","Type":"ContainerStarted","Data":"9023eb7bccf0b8822018837b95a5cd14acaeae1e2844ab7038c017acc28ea302"}
Sep 09 21:52:59.109007 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:52:59.108947    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f9vfp5" podStartSLOduration=2.670667248 podStartE2EDuration="5.108933532s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.438266284s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:52:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:52:59.107536274 +0000 UTC m=+3622.221222173" watchObservedRunningTime="2026-09-09 21:52:59.108933532 +0000 UTC m=+3622.222619406"
Sep 09 21:52:59.131742 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:52:59.131696    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-dbf8fcf48-7n8hn" podStartSLOduration=2.718334907 podStartE2EDuration="5.131677069s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.413342162s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:52:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:52:59.128342182 +0000 UTC m=+3622.242028043" watchObservedRunningTime="2026-09-09 21:52:59.131677069 +0000 UTC m=+3622.245362962"
Sep 09 21:53:01.137692 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:53:01.137650    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:53:01.195137 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:53:01.195103    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:53:04.356832 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:53:04.356790    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-dbf8fcf48-7n8hn"
Sep 09 21:53:04.357361 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:53:04.356872    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-dbf8fcf48-7n8hn"
Sep 09 21:53:04.358469 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:53:04.358424    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-dbf8fcf48-7n8hn" podUID="77e7a640-a9c5-414f-92a4-fda5f4467ffe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8001/health\": dial tcp 10.132.0.82:8001: connect: connection refused"
Sep 09 21:53:04.364920 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:53:04.364898    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f9vfp5"
Sep 09 21:53:04.365033 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:53:04.364955    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f9vfp5"
Sep 09 21:53:04.366164 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:53:04.366140    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f9vfp5" podUID="22f4548d-6510-4dda-9f67-cce061fc1686" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/health\": dial tcp 10.132.0.83:8000: connect: connection refused"
Sep 09 21:53:04.375958 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:53:04.375940    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-dbf8fcf48-7n8hn"
Sep 09 21:53:09.487791 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:53:09.487750    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:53:14.357666 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:53:14.357607    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-dbf8fcf48-7n8hn" podUID="77e7a640-a9c5-414f-92a4-fda5f4467ffe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8001/health\": dial tcp 10.132.0.82:8001: connect: connection refused"
Sep 09 21:53:14.364347 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:53:14.364308    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f9vfp5" podUID="22f4548d-6510-4dda-9f67-cce061fc1686" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/health\": dial tcp 10.132.0.83:8000: connect: connection refused"
Sep 09 21:53:15.176219 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:53:15.176173    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:53:16.225723 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:53:16.225678    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:53:19.009563 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:53:19.009508    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:53:20.903203 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:53:20.903165    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:53:21.007094 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:53:21.007059    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:53:21.111149 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:53:21.111112    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:53:22.464209 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:53:22.464173    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:53:24.357550 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:53:24.357480    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-dbf8fcf48-7n8hn" podUID="77e7a640-a9c5-414f-92a4-fda5f4467ffe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8001/health\": dial tcp 10.132.0.82:8001: connect: connection refused"
Sep 09 21:53:24.364490 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:53:24.364455    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f9vfp5" podUID="22f4548d-6510-4dda-9f67-cce061fc1686" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/health\": dial tcp 10.132.0.83:8000: connect: connection refused"
Sep 09 21:53:27.409959 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:53:27.409926    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:53:33.113999 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:53:33.113931    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:53:33.526437 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:53:33.526343    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:53:33.590294 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:53:33.590256    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:53:34.357507 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:53:34.357464    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-dbf8fcf48-7n8hn" podUID="77e7a640-a9c5-414f-92a4-fda5f4467ffe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8001/health\": dial tcp 10.132.0.82:8001: connect: connection refused"
Sep 09 21:53:34.365039 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:53:34.365002    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f9vfp5" podUID="22f4548d-6510-4dda-9f67-cce061fc1686" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/health\": dial tcp 10.132.0.83:8000: connect: connection refused"
Sep 09 21:53:37.168871 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:53:37.168838    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:53:44.357592 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:53:44.357546    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-dbf8fcf48-7n8hn" podUID="77e7a640-a9c5-414f-92a4-fda5f4467ffe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8001/health\": dial tcp 10.132.0.82:8001: connect: connection refused"
Sep 09 21:53:44.365196 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:53:44.365156    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f9vfp5" podUID="22f4548d-6510-4dda-9f67-cce061fc1686" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/health\": dial tcp 10.132.0.83:8000: connect: connection refused"
Sep 09 21:53:46.127436 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:53:46.127387    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:53:46.227468 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:53:46.227424    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:53:54.357188 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:53:54.357139    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-dbf8fcf48-7n8hn" podUID="77e7a640-a9c5-414f-92a4-fda5f4467ffe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8001/health\": dial tcp 10.132.0.82:8001: connect: connection refused"
Sep 09 21:53:54.364832 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:53:54.364791    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f9vfp5" podUID="22f4548d-6510-4dda-9f67-cce061fc1686" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/health\": dial tcp 10.132.0.83:8000: connect: connection refused"
Sep 09 21:54:04.357465 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:54:04.357199    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-dbf8fcf48-7n8hn" podUID="77e7a640-a9c5-414f-92a4-fda5f4467ffe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8001/health\": dial tcp 10.132.0.82:8001: connect: connection refused"
Sep 09 21:54:04.364871 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:54:04.364833    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f9vfp5" podUID="22f4548d-6510-4dda-9f67-cce061fc1686" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/health\": dial tcp 10.132.0.83:8000: connect: connection refused"
Sep 09 21:54:14.357469 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:54:14.357413    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-dbf8fcf48-7n8hn" podUID="77e7a640-a9c5-414f-92a4-fda5f4467ffe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8001/health\": dial tcp 10.132.0.82:8001: connect: connection refused"
Sep 09 21:54:14.364793 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:54:14.364757    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f9vfp5" podUID="22f4548d-6510-4dda-9f67-cce061fc1686" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/health\": dial tcp 10.132.0.83:8000: connect: connection refused"
Sep 09 21:54:24.357286 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:54:24.357224    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-dbf8fcf48-7n8hn" podUID="77e7a640-a9c5-414f-92a4-fda5f4467ffe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8001/health\": dial tcp 10.132.0.82:8001: connect: connection refused"
Sep 09 21:54:24.364427 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:54:24.364393    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f9vfp5" podUID="22f4548d-6510-4dda-9f67-cce061fc1686" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/health\": dial tcp 10.132.0.83:8000: connect: connection refused"
Sep 09 21:54:34.357901 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:54:34.357854    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-dbf8fcf48-7n8hn" podUID="77e7a640-a9c5-414f-92a4-fda5f4467ffe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8001/health\": dial tcp 10.132.0.82:8001: connect: connection refused"
Sep 09 21:54:34.364474 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:54:34.364428    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f9vfp5" podUID="22f4548d-6510-4dda-9f67-cce061fc1686" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/health\": dial tcp 10.132.0.83:8000: connect: connection refused"
Sep 09 21:54:44.357528 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:54:44.357476    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-dbf8fcf48-7n8hn" podUID="77e7a640-a9c5-414f-92a4-fda5f4467ffe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8001/health\": dial tcp 10.132.0.82:8001: connect: connection refused"
Sep 09 21:54:44.364919 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:54:44.364887    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f9vfp5" podUID="22f4548d-6510-4dda-9f67-cce061fc1686" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/health\": dial tcp 10.132.0.83:8000: connect: connection refused"
Sep 09 21:54:54.358004 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:54:54.357950    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-dbf8fcf48-7n8hn" podUID="77e7a640-a9c5-414f-92a4-fda5f4467ffe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8001/health\": dial tcp 10.132.0.82:8001: connect: connection refused"
Sep 09 21:54:54.364554 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:54:54.364524    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f9vfp5" podUID="22f4548d-6510-4dda-9f67-cce061fc1686" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/health\": dial tcp 10.132.0.83:8000: connect: connection refused"
Sep 09 21:55:04.357972 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:55:04.357922    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-dbf8fcf48-7n8hn" podUID="77e7a640-a9c5-414f-92a4-fda5f4467ffe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8001/health\": dial tcp 10.132.0.82:8001: connect: connection refused"
Sep 09 21:55:04.365078 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:55:04.365037    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f9vfp5" podUID="22f4548d-6510-4dda-9f67-cce061fc1686" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/health\": dial tcp 10.132.0.83:8000: connect: connection refused"
Sep 09 21:55:14.357888 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:55:14.357835    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-dbf8fcf48-7n8hn" podUID="77e7a640-a9c5-414f-92a4-fda5f4467ffe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8001/health\": dial tcp 10.132.0.82:8001: connect: connection refused"
Sep 09 21:55:14.365029 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:55:14.364999    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f9vfp5" podUID="22f4548d-6510-4dda-9f67-cce061fc1686" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/health\": dial tcp 10.132.0.83:8000: connect: connection refused"
Sep 09 21:55:21.055042 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:55:21.055001    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:55:24.357132 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:55:24.357072    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-dbf8fcf48-7n8hn" podUID="77e7a640-a9c5-414f-92a4-fda5f4467ffe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8001/health\": dial tcp 10.132.0.82:8001: connect: connection refused"
Sep 09 21:55:24.364890 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:55:24.364860    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f9vfp5" podUID="22f4548d-6510-4dda-9f67-cce061fc1686" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/health\": dial tcp 10.132.0.83:8000: connect: connection refused"
Sep 09 21:55:34.367510 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:55:34.367425    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-dbf8fcf48-7n8hn"
Sep 09 21:55:34.377040 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:55:34.377012    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f9vfp5"
Sep 09 21:55:34.387971 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:55:34.387940    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-dbf8fcf48-7n8hn"
Sep 09 21:55:34.389526 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:55:34.389501    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f9vfp5"
Sep 09 21:55:41.022232 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:55:41.022173    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:55:41.123055 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:55:41.123008    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:55:41.226262 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:55:41.226210    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:55:47.502968 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:55:47.502938    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:55:47.619540 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:55:47.619499    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:55:54.204634 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:55:54.204591    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:55:56.193862 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:55:56.193817    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:55:56.212758 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:55:56.212721    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f9vfp5"]
Sep 09 21:55:56.215094 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:55:56.215065    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-dbf8fcf48-7n8hn"]
Sep 09 21:55:56.356128 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:55:56.356084    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:56:11.236740 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:11.236697    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f9vfp5" podUID="22f4548d-6510-4dda-9f67-cce061fc1686" containerName="main" containerID="cri-o://369c9b6c27eb282acd36f18710ddb2ff4533cd97eadc38620d46cc0c1ba7fb2b" gracePeriod=45
Sep 09 21:56:11.238295 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:11.238250    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-dbf8fcf48-7n8hn" podUID="77e7a640-a9c5-414f-92a4-fda5f4467ffe" containerName="main" containerID="cri-o://9023eb7bccf0b8822018837b95a5cd14acaeae1e2844ab7038c017acc28ea302" gracePeriod=45
Sep 09 21:56:12.062118 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:12.062082    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-dbf8fcf48-7n8hn" podUID="77e7a640-a9c5-414f-92a4-fda5f4467ffe" containerName="llm-d-routing-sidecar" containerID="cri-o://0bdb3864f8eeff8b14d4e1438becc389547cf69a6901ee0273618702145962ba" gracePeriod=45
Sep 09 21:56:12.192590 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:12.192562    2576 generic.go:356] "Generic (PLEG): container finished" podID="22f4548d-6510-4dda-9f67-cce061fc1686" containerID="369c9b6c27eb282acd36f18710ddb2ff4533cd97eadc38620d46cc0c1ba7fb2b" exitCode=0
Sep 09 21:56:12.192728 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:12.192639    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f9vfp5" event={"ID":"22f4548d-6510-4dda-9f67-cce061fc1686","Type":"ContainerDied","Data":"369c9b6c27eb282acd36f18710ddb2ff4533cd97eadc38620d46cc0c1ba7fb2b"}
Sep 09 21:56:12.192728 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:12.192703    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f9vfp5" event={"ID":"22f4548d-6510-4dda-9f67-cce061fc1686","Type":"ContainerDied","Data":"a847dcaaf0dc141c0db0747dd622df50c5ed0166c6bae614f8e94bd1222d6b7b"}
Sep 09 21:56:12.192844 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:12.192729    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a847dcaaf0dc141c0db0747dd622df50c5ed0166c6bae614f8e94bd1222d6b7b"
Sep 09 21:56:12.192844 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:12.192824    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f9vfp5"
Sep 09 21:56:12.195098 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:12.195078    2576 generic.go:356] "Generic (PLEG): container finished" podID="77e7a640-a9c5-414f-92a4-fda5f4467ffe" containerID="9023eb7bccf0b8822018837b95a5cd14acaeae1e2844ab7038c017acc28ea302" exitCode=0
Sep 09 21:56:12.195098 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:12.195096    2576 generic.go:356] "Generic (PLEG): container finished" podID="77e7a640-a9c5-414f-92a4-fda5f4467ffe" containerID="0bdb3864f8eeff8b14d4e1438becc389547cf69a6901ee0273618702145962ba" exitCode=0
Sep 09 21:56:12.195258 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:12.195155    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-dbf8fcf48-7n8hn" event={"ID":"77e7a640-a9c5-414f-92a4-fda5f4467ffe","Type":"ContainerDied","Data":"9023eb7bccf0b8822018837b95a5cd14acaeae1e2844ab7038c017acc28ea302"}
Sep 09 21:56:12.195258 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:12.195198    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-dbf8fcf48-7n8hn" event={"ID":"77e7a640-a9c5-414f-92a4-fda5f4467ffe","Type":"ContainerDied","Data":"0bdb3864f8eeff8b14d4e1438becc389547cf69a6901ee0273618702145962ba"}
Sep 09 21:56:12.259898 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:12.259826    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22f4548d-6510-4dda-9f67-cce061fc1686-dshm\") pod \"22f4548d-6510-4dda-9f67-cce061fc1686\" (UID: \"22f4548d-6510-4dda-9f67-cce061fc1686\") "
Sep 09 21:56:12.260316 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:12.259908    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22f4548d-6510-4dda-9f67-cce061fc1686-home\") pod \"22f4548d-6510-4dda-9f67-cce061fc1686\" (UID: \"22f4548d-6510-4dda-9f67-cce061fc1686\") "
Sep 09 21:56:12.260316 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:12.259978    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/22f4548d-6510-4dda-9f67-cce061fc1686-kserve-provision-location\") pod \"22f4548d-6510-4dda-9f67-cce061fc1686\" (UID: \"22f4548d-6510-4dda-9f67-cce061fc1686\") "
Sep 09 21:56:12.260316 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:12.260006    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22f4548d-6510-4dda-9f67-cce061fc1686-tmp-dir\") pod \"22f4548d-6510-4dda-9f67-cce061fc1686\" (UID: \"22f4548d-6510-4dda-9f67-cce061fc1686\") "
Sep 09 21:56:12.260316 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:12.260036    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22f4548d-6510-4dda-9f67-cce061fc1686-model-cache\") pod \"22f4548d-6510-4dda-9f67-cce061fc1686\" (UID: \"22f4548d-6510-4dda-9f67-cce061fc1686\") "
Sep 09 21:56:12.260316 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:12.260074    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/22f4548d-6510-4dda-9f67-cce061fc1686-cabundle-cert\") pod \"22f4548d-6510-4dda-9f67-cce061fc1686\" (UID: \"22f4548d-6510-4dda-9f67-cce061fc1686\") "
Sep 09 21:56:12.260316 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:12.260099    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8kbg8\" (UniqueName: \"kubernetes.io/projected/22f4548d-6510-4dda-9f67-cce061fc1686-kube-api-access-8kbg8\") pod \"22f4548d-6510-4dda-9f67-cce061fc1686\" (UID: \"22f4548d-6510-4dda-9f67-cce061fc1686\") "
Sep 09 21:56:12.260316 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:12.260133    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22f4548d-6510-4dda-9f67-cce061fc1686-tls-certs\") pod \"22f4548d-6510-4dda-9f67-cce061fc1686\" (UID: \"22f4548d-6510-4dda-9f67-cce061fc1686\") "
Sep 09 21:56:12.260787 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:12.260757    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22f4548d-6510-4dda-9f67-cce061fc1686-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "22f4548d-6510-4dda-9f67-cce061fc1686" (UID: "22f4548d-6510-4dda-9f67-cce061fc1686"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:56:12.260890 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:12.260809    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22f4548d-6510-4dda-9f67-cce061fc1686-home" (OuterVolumeSpecName: "home") pod "22f4548d-6510-4dda-9f67-cce061fc1686" (UID: "22f4548d-6510-4dda-9f67-cce061fc1686"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:56:12.261130 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:12.261104    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22f4548d-6510-4dda-9f67-cce061fc1686-model-cache" (OuterVolumeSpecName: "model-cache") pod "22f4548d-6510-4dda-9f67-cce061fc1686" (UID: "22f4548d-6510-4dda-9f67-cce061fc1686"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:56:12.262738 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:12.262707    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22f4548d-6510-4dda-9f67-cce061fc1686-dshm" (OuterVolumeSpecName: "dshm") pod "22f4548d-6510-4dda-9f67-cce061fc1686" (UID: "22f4548d-6510-4dda-9f67-cce061fc1686"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:56:12.262879 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:12.262860    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22f4548d-6510-4dda-9f67-cce061fc1686-kube-api-access-8kbg8" (OuterVolumeSpecName: "kube-api-access-8kbg8") pod "22f4548d-6510-4dda-9f67-cce061fc1686" (UID: "22f4548d-6510-4dda-9f67-cce061fc1686"). InnerVolumeSpecName "kube-api-access-8kbg8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:56:12.263097 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:12.263078    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22f4548d-6510-4dda-9f67-cce061fc1686-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "22f4548d-6510-4dda-9f67-cce061fc1686" (UID: "22f4548d-6510-4dda-9f67-cce061fc1686"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:56:12.266239 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:12.266214    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22f4548d-6510-4dda-9f67-cce061fc1686-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "22f4548d-6510-4dda-9f67-cce061fc1686" (UID: "22f4548d-6510-4dda-9f67-cce061fc1686"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:56:12.273542 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:12.273517    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22f4548d-6510-4dda-9f67-cce061fc1686-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "22f4548d-6510-4dda-9f67-cce061fc1686" (UID: "22f4548d-6510-4dda-9f67-cce061fc1686"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:56:12.297884 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:12.297862    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-dbf8fcf48-7n8hn"
Sep 09 21:56:12.360984 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:12.360956    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/77e7a640-a9c5-414f-92a4-fda5f4467ffe-kserve-provision-location\") pod \"77e7a640-a9c5-414f-92a4-fda5f4467ffe\" (UID: \"77e7a640-a9c5-414f-92a4-fda5f4467ffe\") "
Sep 09 21:56:12.361131 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:12.360994    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77e7a640-a9c5-414f-92a4-fda5f4467ffe-dshm\") pod \"77e7a640-a9c5-414f-92a4-fda5f4467ffe\" (UID: \"77e7a640-a9c5-414f-92a4-fda5f4467ffe\") "
Sep 09 21:56:12.361131 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:12.361052    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p8zph\" (UniqueName: \"kubernetes.io/projected/77e7a640-a9c5-414f-92a4-fda5f4467ffe-kube-api-access-p8zph\") pod \"77e7a640-a9c5-414f-92a4-fda5f4467ffe\" (UID: \"77e7a640-a9c5-414f-92a4-fda5f4467ffe\") "
Sep 09 21:56:12.361131 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:12.361096    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/77e7a640-a9c5-414f-92a4-fda5f4467ffe-cabundle-cert\") pod \"77e7a640-a9c5-414f-92a4-fda5f4467ffe\" (UID: \"77e7a640-a9c5-414f-92a4-fda5f4467ffe\") "
Sep 09 21:56:12.361310 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:12.361134    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77e7a640-a9c5-414f-92a4-fda5f4467ffe-tmp-dir\") pod \"77e7a640-a9c5-414f-92a4-fda5f4467ffe\" (UID: \"77e7a640-a9c5-414f-92a4-fda5f4467ffe\") "
Sep 09 21:56:12.361310 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:12.361174    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77e7a640-a9c5-414f-92a4-fda5f4467ffe-tls-certs\") pod \"77e7a640-a9c5-414f-92a4-fda5f4467ffe\" (UID: \"77e7a640-a9c5-414f-92a4-fda5f4467ffe\") "
Sep 09 21:56:12.361310 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:12.361198    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77e7a640-a9c5-414f-92a4-fda5f4467ffe-home\") pod \"77e7a640-a9c5-414f-92a4-fda5f4467ffe\" (UID: \"77e7a640-a9c5-414f-92a4-fda5f4467ffe\") "
Sep 09 21:56:12.361310 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:12.361256    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77e7a640-a9c5-414f-92a4-fda5f4467ffe-model-cache\") pod \"77e7a640-a9c5-414f-92a4-fda5f4467ffe\" (UID: \"77e7a640-a9c5-414f-92a4-fda5f4467ffe\") "
Sep 09 21:56:12.361561 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:12.361484    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77e7a640-a9c5-414f-92a4-fda5f4467ffe-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "77e7a640-a9c5-414f-92a4-fda5f4467ffe" (UID: "77e7a640-a9c5-414f-92a4-fda5f4467ffe"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:56:12.361561 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:12.361547    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22f4548d-6510-4dda-9f67-cce061fc1686-dshm\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:56:12.361655 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:12.361567    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22f4548d-6510-4dda-9f67-cce061fc1686-home\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:56:12.361655 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:12.361582    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/22f4548d-6510-4dda-9f67-cce061fc1686-kserve-provision-location\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:56:12.361655 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:12.361597    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22f4548d-6510-4dda-9f67-cce061fc1686-tmp-dir\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:56:12.361655 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:12.361611    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22f4548d-6510-4dda-9f67-cce061fc1686-model-cache\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:56:12.361655 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:12.361625    2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/22f4548d-6510-4dda-9f67-cce061fc1686-cabundle-cert\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:56:12.361655 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:12.361635    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8kbg8\" (UniqueName: \"kubernetes.io/projected/22f4548d-6510-4dda-9f67-cce061fc1686-kube-api-access-8kbg8\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:56:12.361655 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:12.361645    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22f4548d-6510-4dda-9f67-cce061fc1686-tls-certs\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:56:12.362170 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:12.361846    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77e7a640-a9c5-414f-92a4-fda5f4467ffe-model-cache" (OuterVolumeSpecName: "model-cache") pod "77e7a640-a9c5-414f-92a4-fda5f4467ffe" (UID: "77e7a640-a9c5-414f-92a4-fda5f4467ffe"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:56:12.362311 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:12.362287    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77e7a640-a9c5-414f-92a4-fda5f4467ffe-home" (OuterVolumeSpecName: "home") pod "77e7a640-a9c5-414f-92a4-fda5f4467ffe" (UID: "77e7a640-a9c5-414f-92a4-fda5f4467ffe"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:56:12.363562 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:12.363533    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77e7a640-a9c5-414f-92a4-fda5f4467ffe-kube-api-access-p8zph" (OuterVolumeSpecName: "kube-api-access-p8zph") pod "77e7a640-a9c5-414f-92a4-fda5f4467ffe" (UID: "77e7a640-a9c5-414f-92a4-fda5f4467ffe"). InnerVolumeSpecName "kube-api-access-p8zph". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:56:12.363669 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:12.363632    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77e7a640-a9c5-414f-92a4-fda5f4467ffe-dshm" (OuterVolumeSpecName: "dshm") pod "77e7a640-a9c5-414f-92a4-fda5f4467ffe" (UID: "77e7a640-a9c5-414f-92a4-fda5f4467ffe"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:56:12.363917 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:12.363898    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77e7a640-a9c5-414f-92a4-fda5f4467ffe-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "77e7a640-a9c5-414f-92a4-fda5f4467ffe" (UID: "77e7a640-a9c5-414f-92a4-fda5f4467ffe"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:56:12.366518 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:12.366496    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77e7a640-a9c5-414f-92a4-fda5f4467ffe-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "77e7a640-a9c5-414f-92a4-fda5f4467ffe" (UID: "77e7a640-a9c5-414f-92a4-fda5f4467ffe"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:56:12.373463 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:12.373442    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77e7a640-a9c5-414f-92a4-fda5f4467ffe-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "77e7a640-a9c5-414f-92a4-fda5f4467ffe" (UID: "77e7a640-a9c5-414f-92a4-fda5f4467ffe"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:56:12.462723 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:12.462701    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8zph\" (UniqueName: \"kubernetes.io/projected/77e7a640-a9c5-414f-92a4-fda5f4467ffe-kube-api-access-p8zph\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:56:12.462723 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:12.462722    2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/77e7a640-a9c5-414f-92a4-fda5f4467ffe-cabundle-cert\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:56:12.462843 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:12.462733    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77e7a640-a9c5-414f-92a4-fda5f4467ffe-tmp-dir\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:56:12.462843 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:12.462740    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77e7a640-a9c5-414f-92a4-fda5f4467ffe-tls-certs\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:56:12.462843 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:12.462748    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77e7a640-a9c5-414f-92a4-fda5f4467ffe-home\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:56:12.462843 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:12.462755    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77e7a640-a9c5-414f-92a4-fda5f4467ffe-model-cache\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:56:12.462843 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:12.462763    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/77e7a640-a9c5-414f-92a4-fda5f4467ffe-kserve-provision-location\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:56:12.462843 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:12.462770    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77e7a640-a9c5-414f-92a4-fda5f4467ffe-dshm\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:56:13.202491 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:13.202463    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-dbf8fcf48-7n8hn"
Sep 09 21:56:13.202664 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:13.202489    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-dbf8fcf48-7n8hn" event={"ID":"77e7a640-a9c5-414f-92a4-fda5f4467ffe","Type":"ContainerDied","Data":"8b35dc10b9df055a66924272118ca717d8f7774953b99f8a8be75508aab72ed5"}
Sep 09 21:56:13.202664 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:13.202545    2576 scope.go:117] "RemoveContainer" containerID="9023eb7bccf0b8822018837b95a5cd14acaeae1e2844ab7038c017acc28ea302"
Sep 09 21:56:13.202664 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:13.202465    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f9vfp5"
Sep 09 21:56:13.212446 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:13.212422    2576 scope.go:117] "RemoveContainer" containerID="00a1dbe6c6c28c4a4c69e3711c9ca7fabf3606fad554c714257843ad8e5e62d9"
Sep 09 21:56:13.220527 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:13.220509    2576 scope.go:117] "RemoveContainer" containerID="0bdb3864f8eeff8b14d4e1438becc389547cf69a6901ee0273618702145962ba"
Sep 09 21:56:13.227896 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:13.227877    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-dbf8fcf48-7n8hn"]
Sep 09 21:56:13.233145 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:13.233125    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-dbf8fcf48-7n8hn"]
Sep 09 21:56:13.245702 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:13.245679    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f9vfp5"]
Sep 09 21:56:13.251138 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:13.251119    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f9vfp5"]
Sep 09 21:56:13.409203 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:13.409172    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="22f4548d-6510-4dda-9f67-cce061fc1686" path="/var/lib/kubelet/pods/22f4548d-6510-4dda-9f67-cce061fc1686/volumes"
Sep 09 21:56:13.409642 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:13.409629    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="77e7a640-a9c5-414f-92a4-fda5f4467ffe" path="/var/lib/kubelet/pods/77e7a640-a9c5-414f-92a4-fda5f4467ffe/volumes"
Sep 09 21:56:26.434033 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:26.433993    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:56:44.042707 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:44.042659    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:56:45.543490 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:45.543453    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:56:52.214448 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:52.214412    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:56:52.477351 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:52.477271    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-684c688995-bqsvh"]
Sep 09 21:56:52.477767 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:52.477746    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77e7a640-a9c5-414f-92a4-fda5f4467ffe" containerName="main"
Sep 09 21:56:52.477838 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:52.477772    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="77e7a640-a9c5-414f-92a4-fda5f4467ffe" containerName="main"
Sep 09 21:56:52.477838 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:52.477792    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22f4548d-6510-4dda-9f67-cce061fc1686" containerName="main"
Sep 09 21:56:52.477838 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:52.477801    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="22f4548d-6510-4dda-9f67-cce061fc1686" containerName="main"
Sep 09 21:56:52.477838 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:52.477822    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77e7a640-a9c5-414f-92a4-fda5f4467ffe" containerName="storage-initializer"
Sep 09 21:56:52.477838 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:52.477831    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="77e7a640-a9c5-414f-92a4-fda5f4467ffe" containerName="storage-initializer"
Sep 09 21:56:52.478013 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:52.477846    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77e7a640-a9c5-414f-92a4-fda5f4467ffe" containerName="llm-d-routing-sidecar"
Sep 09 21:56:52.478013 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:52.477855    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="77e7a640-a9c5-414f-92a4-fda5f4467ffe" containerName="llm-d-routing-sidecar"
Sep 09 21:56:52.478013 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:52.477872    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22f4548d-6510-4dda-9f67-cce061fc1686" containerName="storage-initializer"
Sep 09 21:56:52.478013 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:52.477880    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="22f4548d-6510-4dda-9f67-cce061fc1686" containerName="storage-initializer"
Sep 09 21:56:52.478013 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:52.477952    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="77e7a640-a9c5-414f-92a4-fda5f4467ffe" containerName="main"
Sep 09 21:56:52.478013 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:52.477963    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="77e7a640-a9c5-414f-92a4-fda5f4467ffe" containerName="llm-d-routing-sidecar"
Sep 09 21:56:52.478013 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:52.477971    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="22f4548d-6510-4dda-9f67-cce061fc1686" containerName="main"
Sep 09 21:56:52.482583 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:52.482553    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-684c688995-bqsvh"
Sep 09 21:56:52.488023 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:52.487996    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"odh-kserve-custom-ca-bundle\""
Sep 09 21:56:52.488162 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:52.488028    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-kserve-dockercfg-vnwlt\""
Sep 09 21:56:52.488533 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:52.488514    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"default-dockercfg-c4t85\""
Sep 09 21:56:52.488626 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:52.488547    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"kube-root-ca.crt\""
Sep 09 21:56:52.488626 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:52.488562    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"seaweedfs-s3-creds\""
Sep 09 21:56:52.489645 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:52.489443    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-kserve-self-signed-certs\""
Sep 09 21:56:52.489645 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:52.489593    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"openshift-service-ca.crt\""
Sep 09 21:56:52.513418 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:52.513395    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-65c8845bdf-t6625"]
Sep 09 21:56:52.516632 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:52.516616    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-65c8845bdf-t6625"
Sep 09 21:56:52.530710 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:52.530687    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-684c688995-bqsvh"]
Sep 09 21:56:52.554691 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:52.554662    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-65c8845bdf-t6625"]
Sep 09 21:56:52.584800 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:52.584772    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:56:52.603034 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:52.602998    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/897e3e04-a7d6-4670-baf2-d8f91d40cabf-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-684c688995-bqsvh\" (UID: \"897e3e04-a7d6-4670-baf2-d8f91d40cabf\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-684c688995-bqsvh"
Sep 09 21:56:52.603176 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:52.603047    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ef59f263-d3c3-4201-a6af-0f0992e3fd25-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-prefill-65c8845bdf-t6625\" (UID: \"ef59f263-d3c3-4201-a6af-0f0992e3fd25\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-65c8845bdf-t6625"
Sep 09 21:56:52.603176 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:52.603067    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef59f263-d3c3-4201-a6af-0f0992e3fd25-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-65c8845bdf-t6625\" (UID: \"ef59f263-d3c3-4201-a6af-0f0992e3fd25\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-65c8845bdf-t6625"
Sep 09 21:56:52.603176 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:52.603106    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef59f263-d3c3-4201-a6af-0f0992e3fd25-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-65c8845bdf-t6625\" (UID: \"ef59f263-d3c3-4201-a6af-0f0992e3fd25\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-65c8845bdf-t6625"
Sep 09 21:56:52.603176 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:52.603139    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/897e3e04-a7d6-4670-baf2-d8f91d40cabf-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-684c688995-bqsvh\" (UID: \"897e3e04-a7d6-4670-baf2-d8f91d40cabf\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-684c688995-bqsvh"
Sep 09 21:56:52.603176 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:52.603163    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/897e3e04-a7d6-4670-baf2-d8f91d40cabf-home\") pod \"custom-route-timeout-pd-test-kserve-684c688995-bqsvh\" (UID: \"897e3e04-a7d6-4670-baf2-d8f91d40cabf\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-684c688995-bqsvh"
Sep 09 21:56:52.603390 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:52.603187    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/897e3e04-a7d6-4670-baf2-d8f91d40cabf-dshm\") pod \"custom-route-timeout-pd-test-kserve-684c688995-bqsvh\" (UID: \"897e3e04-a7d6-4670-baf2-d8f91d40cabf\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-684c688995-bqsvh"
Sep 09 21:56:52.603390 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:52.603209    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ef59f263-d3c3-4201-a6af-0f0992e3fd25-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-65c8845bdf-t6625\" (UID: \"ef59f263-d3c3-4201-a6af-0f0992e3fd25\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-65c8845bdf-t6625"
Sep 09 21:56:52.603390 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:52.603239    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/897e3e04-a7d6-4670-baf2-d8f91d40cabf-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-684c688995-bqsvh\" (UID: \"897e3e04-a7d6-4670-baf2-d8f91d40cabf\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-684c688995-bqsvh"
Sep 09 21:56:52.603390 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:52.603288    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef59f263-d3c3-4201-a6af-0f0992e3fd25-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-65c8845bdf-t6625\" (UID: \"ef59f263-d3c3-4201-a6af-0f0992e3fd25\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-65c8845bdf-t6625"
Sep 09 21:56:52.603390 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:52.603328    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef59f263-d3c3-4201-a6af-0f0992e3fd25-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-65c8845bdf-t6625\" (UID: \"ef59f263-d3c3-4201-a6af-0f0992e3fd25\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-65c8845bdf-t6625"
Sep 09 21:56:52.603390 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:52.603349    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef59f263-d3c3-4201-a6af-0f0992e3fd25-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-65c8845bdf-t6625\" (UID: \"ef59f263-d3c3-4201-a6af-0f0992e3fd25\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-65c8845bdf-t6625"
Sep 09 21:56:52.603633 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:52.603391    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swfzz\" (UniqueName: \"kubernetes.io/projected/897e3e04-a7d6-4670-baf2-d8f91d40cabf-kube-api-access-swfzz\") pod \"custom-route-timeout-pd-test-kserve-684c688995-bqsvh\" (UID: \"897e3e04-a7d6-4670-baf2-d8f91d40cabf\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-684c688995-bqsvh"
Sep 09 21:56:52.603633 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:52.603458    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbnw8\" (UniqueName: \"kubernetes.io/projected/ef59f263-d3c3-4201-a6af-0f0992e3fd25-kube-api-access-jbnw8\") pod \"custom-route-timeout-pd-test-kserve-prefill-65c8845bdf-t6625\" (UID: \"ef59f263-d3c3-4201-a6af-0f0992e3fd25\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-65c8845bdf-t6625"
Sep 09 21:56:52.603633 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:52.603507    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/897e3e04-a7d6-4670-baf2-d8f91d40cabf-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-684c688995-bqsvh\" (UID: \"897e3e04-a7d6-4670-baf2-d8f91d40cabf\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-684c688995-bqsvh"
Sep 09 21:56:52.603633 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:52.603536    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/897e3e04-a7d6-4670-baf2-d8f91d40cabf-model-cache\") pod \"custom-route-timeout-pd-test-kserve-684c688995-bqsvh\" (UID: \"897e3e04-a7d6-4670-baf2-d8f91d40cabf\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-684c688995-bqsvh"
Sep 09 21:56:52.704143 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:52.704106    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/897e3e04-a7d6-4670-baf2-d8f91d40cabf-model-cache\") pod \"custom-route-timeout-pd-test-kserve-684c688995-bqsvh\" (UID: \"897e3e04-a7d6-4670-baf2-d8f91d40cabf\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-684c688995-bqsvh"
Sep 09 21:56:52.704143 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:52.704151    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/897e3e04-a7d6-4670-baf2-d8f91d40cabf-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-684c688995-bqsvh\" (UID: \"897e3e04-a7d6-4670-baf2-d8f91d40cabf\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-684c688995-bqsvh"
Sep 09 21:56:52.704348 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:52.704178    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ef59f263-d3c3-4201-a6af-0f0992e3fd25-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-prefill-65c8845bdf-t6625\" (UID: \"ef59f263-d3c3-4201-a6af-0f0992e3fd25\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-65c8845bdf-t6625"
Sep 09 21:56:52.704348 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:52.704289    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef59f263-d3c3-4201-a6af-0f0992e3fd25-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-65c8845bdf-t6625\" (UID: \"ef59f263-d3c3-4201-a6af-0f0992e3fd25\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-65c8845bdf-t6625"
Sep 09 21:56:52.704348 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:52.704329    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef59f263-d3c3-4201-a6af-0f0992e3fd25-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-65c8845bdf-t6625\" (UID: \"ef59f263-d3c3-4201-a6af-0f0992e3fd25\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-65c8845bdf-t6625"
Sep 09 21:56:52.704537 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:52.704357    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/897e3e04-a7d6-4670-baf2-d8f91d40cabf-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-684c688995-bqsvh\" (UID: \"897e3e04-a7d6-4670-baf2-d8f91d40cabf\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-684c688995-bqsvh"
Sep 09 21:56:52.704537 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:52.704500    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/897e3e04-a7d6-4670-baf2-d8f91d40cabf-home\") pod \"custom-route-timeout-pd-test-kserve-684c688995-bqsvh\" (UID: \"897e3e04-a7d6-4670-baf2-d8f91d40cabf\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-684c688995-bqsvh"
Sep 09 21:56:52.704653 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:52.704537    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/897e3e04-a7d6-4670-baf2-d8f91d40cabf-dshm\") pod \"custom-route-timeout-pd-test-kserve-684c688995-bqsvh\" (UID: \"897e3e04-a7d6-4670-baf2-d8f91d40cabf\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-684c688995-bqsvh"
Sep 09 21:56:52.704653 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:52.704569    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ef59f263-d3c3-4201-a6af-0f0992e3fd25-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-65c8845bdf-t6625\" (UID: \"ef59f263-d3c3-4201-a6af-0f0992e3fd25\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-65c8845bdf-t6625"
Sep 09 21:56:52.704653 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:52.704589    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/897e3e04-a7d6-4670-baf2-d8f91d40cabf-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-684c688995-bqsvh\" (UID: \"897e3e04-a7d6-4670-baf2-d8f91d40cabf\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-684c688995-bqsvh"
Sep 09 21:56:52.704653 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:52.704623    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/897e3e04-a7d6-4670-baf2-d8f91d40cabf-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-684c688995-bqsvh\" (UID: \"897e3e04-a7d6-4670-baf2-d8f91d40cabf\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-684c688995-bqsvh"
Sep 09 21:56:52.704653 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:52.704641    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/897e3e04-a7d6-4670-baf2-d8f91d40cabf-model-cache\") pod \"custom-route-timeout-pd-test-kserve-684c688995-bqsvh\" (UID: \"897e3e04-a7d6-4670-baf2-d8f91d40cabf\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-684c688995-bqsvh"
Sep 09 21:56:52.704917 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:52.704674    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef59f263-d3c3-4201-a6af-0f0992e3fd25-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-65c8845bdf-t6625\" (UID: \"ef59f263-d3c3-4201-a6af-0f0992e3fd25\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-65c8845bdf-t6625"
Sep 09 21:56:52.704917 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:52.704768    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef59f263-d3c3-4201-a6af-0f0992e3fd25-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-65c8845bdf-t6625\" (UID: \"ef59f263-d3c3-4201-a6af-0f0992e3fd25\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-65c8845bdf-t6625"
Sep 09 21:56:52.704917 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:52.704817    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef59f263-d3c3-4201-a6af-0f0992e3fd25-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-65c8845bdf-t6625\" (UID: \"ef59f263-d3c3-4201-a6af-0f0992e3fd25\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-65c8845bdf-t6625"
Sep 09 21:56:52.704917 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:52.704856    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef59f263-d3c3-4201-a6af-0f0992e3fd25-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-65c8845bdf-t6625\" (UID: \"ef59f263-d3c3-4201-a6af-0f0992e3fd25\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-65c8845bdf-t6625"
Sep 09 21:56:52.704917 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:52.704881    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef59f263-d3c3-4201-a6af-0f0992e3fd25-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-65c8845bdf-t6625\" (UID: \"ef59f263-d3c3-4201-a6af-0f0992e3fd25\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-65c8845bdf-t6625"
Sep 09 21:56:52.704917 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:52.704907    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swfzz\" (UniqueName: \"kubernetes.io/projected/897e3e04-a7d6-4670-baf2-d8f91d40cabf-kube-api-access-swfzz\") pod \"custom-route-timeout-pd-test-kserve-684c688995-bqsvh\" (UID: \"897e3e04-a7d6-4670-baf2-d8f91d40cabf\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-684c688995-bqsvh"
Sep 09 21:56:52.704917 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:52.704910    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/897e3e04-a7d6-4670-baf2-d8f91d40cabf-home\") pod \"custom-route-timeout-pd-test-kserve-684c688995-bqsvh\" (UID: \"897e3e04-a7d6-4670-baf2-d8f91d40cabf\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-684c688995-bqsvh"
Sep 09 21:56:52.705279 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:52.704922    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ef59f263-d3c3-4201-a6af-0f0992e3fd25-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-prefill-65c8845bdf-t6625\" (UID: \"ef59f263-d3c3-4201-a6af-0f0992e3fd25\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-65c8845bdf-t6625"
Sep 09 21:56:52.705279 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:52.704947    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbnw8\" (UniqueName: \"kubernetes.io/projected/ef59f263-d3c3-4201-a6af-0f0992e3fd25-kube-api-access-jbnw8\") pod \"custom-route-timeout-pd-test-kserve-prefill-65c8845bdf-t6625\" (UID: \"ef59f263-d3c3-4201-a6af-0f0992e3fd25\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-65c8845bdf-t6625"
Sep 09 21:56:52.705279 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:52.704963    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ef59f263-d3c3-4201-a6af-0f0992e3fd25-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-65c8845bdf-t6625\" (UID: \"ef59f263-d3c3-4201-a6af-0f0992e3fd25\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-65c8845bdf-t6625"
Sep 09 21:56:52.705279 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:52.704988    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/897e3e04-a7d6-4670-baf2-d8f91d40cabf-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-684c688995-bqsvh\" (UID: \"897e3e04-a7d6-4670-baf2-d8f91d40cabf\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-684c688995-bqsvh"
Sep 09 21:56:52.705279 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:52.705103    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/897e3e04-a7d6-4670-baf2-d8f91d40cabf-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-684c688995-bqsvh\" (UID: \"897e3e04-a7d6-4670-baf2-d8f91d40cabf\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-684c688995-bqsvh"
Sep 09 21:56:52.705279 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:52.705159    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef59f263-d3c3-4201-a6af-0f0992e3fd25-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-65c8845bdf-t6625\" (UID: \"ef59f263-d3c3-4201-a6af-0f0992e3fd25\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-65c8845bdf-t6625"
Sep 09 21:56:52.705573 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:52.705282    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/897e3e04-a7d6-4670-baf2-d8f91d40cabf-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-684c688995-bqsvh\" (UID: \"897e3e04-a7d6-4670-baf2-d8f91d40cabf\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-684c688995-bqsvh"
Sep 09 21:56:52.707032 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:52.707008    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/897e3e04-a7d6-4670-baf2-d8f91d40cabf-dshm\") pod \"custom-route-timeout-pd-test-kserve-684c688995-bqsvh\" (UID: \"897e3e04-a7d6-4670-baf2-d8f91d40cabf\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-684c688995-bqsvh"
Sep 09 21:56:52.707387 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:52.707347    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef59f263-d3c3-4201-a6af-0f0992e3fd25-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-65c8845bdf-t6625\" (UID: \"ef59f263-d3c3-4201-a6af-0f0992e3fd25\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-65c8845bdf-t6625"
Sep 09 21:56:52.707810 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:52.707794    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/897e3e04-a7d6-4670-baf2-d8f91d40cabf-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-684c688995-bqsvh\" (UID: \"897e3e04-a7d6-4670-baf2-d8f91d40cabf\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-684c688995-bqsvh"
Sep 09 21:56:52.707934 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:52.707913    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef59f263-d3c3-4201-a6af-0f0992e3fd25-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-65c8845bdf-t6625\" (UID: \"ef59f263-d3c3-4201-a6af-0f0992e3fd25\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-65c8845bdf-t6625"
Sep 09 21:56:52.716107 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:52.716081    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-swfzz\" (UniqueName: \"kubernetes.io/projected/897e3e04-a7d6-4670-baf2-d8f91d40cabf-kube-api-access-swfzz\") pod \"custom-route-timeout-pd-test-kserve-684c688995-bqsvh\" (UID: \"897e3e04-a7d6-4670-baf2-d8f91d40cabf\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-684c688995-bqsvh"
Sep 09 21:56:52.716346 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:52.716321    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbnw8\" (UniqueName: \"kubernetes.io/projected/ef59f263-d3c3-4201-a6af-0f0992e3fd25-kube-api-access-jbnw8\") pod \"custom-route-timeout-pd-test-kserve-prefill-65c8845bdf-t6625\" (UID: \"ef59f263-d3c3-4201-a6af-0f0992e3fd25\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-65c8845bdf-t6625"
Sep 09 21:56:52.793219 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:52.793140    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-684c688995-bqsvh"
Sep 09 21:56:52.825661 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:52.825618    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-65c8845bdf-t6625"
Sep 09 21:56:52.907293 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:52.907259    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:56:53.088541 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:53.088515    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-684c688995-bqsvh"]
Sep 09 21:56:53.091094 ip-10-0-134-72 kubenswrapper[2576]: W0909 21:56:53.091070    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod897e3e04_a7d6_4670_baf2_d8f91d40cabf.slice/crio-f623a50b747ec14fff6427f513aadafec053ab4c7aa3373bffe8b176a7a3e6e5 WatchSource:0}: Error finding container f623a50b747ec14fff6427f513aadafec053ab4c7aa3373bffe8b176a7a3e6e5: Status 404 returned error can't find the container with id f623a50b747ec14fff6427f513aadafec053ab4c7aa3373bffe8b176a7a3e6e5
Sep 09 21:56:53.124415 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:53.124393    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-65c8845bdf-t6625"]
Sep 09 21:56:53.125740 ip-10-0-134-72 kubenswrapper[2576]: W0909 21:56:53.125709    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef59f263_d3c3_4201_a6af_0f0992e3fd25.slice/crio-ba89cd99eae9bf056c855ae1c910a57bd39fb1fed0045f61df881d6c3df22e7a WatchSource:0}: Error finding container ba89cd99eae9bf056c855ae1c910a57bd39fb1fed0045f61df881d6c3df22e7a: Status 404 returned error can't find the container with id ba89cd99eae9bf056c855ae1c910a57bd39fb1fed0045f61df881d6c3df22e7a
Sep 09 21:56:53.409695 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:53.409670    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-684c688995-bqsvh"
Sep 09 21:56:53.410003 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:53.409726    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-684c688995-bqsvh" event={"ID":"897e3e04-a7d6-4670-baf2-d8f91d40cabf","Type":"ContainerStarted","Data":"0899c4f53f535398749e6d2d8974994461531f3ccceebd3c36638b2028517604"}
Sep 09 21:56:53.410003 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:53.409754    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-684c688995-bqsvh" event={"ID":"897e3e04-a7d6-4670-baf2-d8f91d40cabf","Type":"ContainerStarted","Data":"f623a50b747ec14fff6427f513aadafec053ab4c7aa3373bffe8b176a7a3e6e5"}
Sep 09 21:56:53.410003 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:53.409780    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-65c8845bdf-t6625" event={"ID":"ef59f263-d3c3-4201-a6af-0f0992e3fd25","Type":"ContainerStarted","Data":"154bd45ee58259db58eb761af585283901533a4d4cf77d38359a070d53562137"}
Sep 09 21:56:53.410003 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:53.409810    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-65c8845bdf-t6625" event={"ID":"ef59f263-d3c3-4201-a6af-0f0992e3fd25","Type":"ContainerStarted","Data":"ba89cd99eae9bf056c855ae1c910a57bd39fb1fed0045f61df881d6c3df22e7a"}
Sep 09 21:56:54.413070 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:54.413038    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-684c688995-bqsvh" event={"ID":"897e3e04-a7d6-4670-baf2-d8f91d40cabf","Type":"ContainerStarted","Data":"c9168e7f67f68723344d8eb289d963a8594825b9caf60a6e2415f5b04af0b60a"}
Sep 09 21:56:56.426780 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:56.426746    2576 generic.go:356] "Generic (PLEG): container finished" podID="ef59f263-d3c3-4201-a6af-0f0992e3fd25" containerID="154bd45ee58259db58eb761af585283901533a4d4cf77d38359a070d53562137" exitCode=0
Sep 09 21:56:56.427319 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:56.426827    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-65c8845bdf-t6625" event={"ID":"ef59f263-d3c3-4201-a6af-0f0992e3fd25","Type":"ContainerDied","Data":"154bd45ee58259db58eb761af585283901533a4d4cf77d38359a070d53562137"}
Sep 09 21:56:56.428953 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:56.428931    2576 generic.go:356] "Generic (PLEG): container finished" podID="897e3e04-a7d6-4670-baf2-d8f91d40cabf" containerID="c9168e7f67f68723344d8eb289d963a8594825b9caf60a6e2415f5b04af0b60a" exitCode=0
Sep 09 21:56:56.429041 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:56.428998    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-684c688995-bqsvh" event={"ID":"897e3e04-a7d6-4670-baf2-d8f91d40cabf","Type":"ContainerDied","Data":"c9168e7f67f68723344d8eb289d963a8594825b9caf60a6e2415f5b04af0b60a"}
Sep 09 21:56:57.437647 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:57.437606    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-684c688995-bqsvh" event={"ID":"897e3e04-a7d6-4670-baf2-d8f91d40cabf","Type":"ContainerStarted","Data":"38e892034b712611a00c8ccf683813bbbd6cd43e183de4cef41c3ad036eb00c2"}
Sep 09 21:56:57.439771 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:57.439744    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-65c8845bdf-t6625" event={"ID":"ef59f263-d3c3-4201-a6af-0f0992e3fd25","Type":"ContainerStarted","Data":"3b7a6aaddcd001a29ef1ec31bdaadf3350cc6ddeffab54d762f33127dacd9f9c"}
Sep 09 21:56:57.493231 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:57.493176    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-684c688995-bqsvh" podStartSLOduration=3.138576836 podStartE2EDuration="5.493158314s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.354581478s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:56:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:56:57.489826357 +0000 UTC m=+3860.603512215" watchObservedRunningTime="2026-09-09 21:56:57.493158314 +0000 UTC m=+3860.606844176"
Sep 09 21:56:57.542472 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:57.542417    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-65c8845bdf-t6625" podStartSLOduration=3.117801414 podStartE2EDuration="5.542400058s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.424598644s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:56:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:56:57.539276966 +0000 UTC m=+3860.652962827" watchObservedRunningTime="2026-09-09 21:56:57.542400058 +0000 UTC m=+3860.656085913"
Sep 09 21:56:59.401198 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:59.401096    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:56:59.511795 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:56:59.511755    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:57:02.794148 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:57:02.794111    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-684c688995-bqsvh"
Sep 09 21:57:02.794696 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:57:02.794212    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-684c688995-bqsvh"
Sep 09 21:57:02.795415 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:57:02.795357    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-684c688995-bqsvh" podUID="897e3e04-a7d6-4670-baf2-d8f91d40cabf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8001/health\": dial tcp 10.132.0.84:8001: connect: connection refused"
Sep 09 21:57:02.825797 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:57:02.825756    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-65c8845bdf-t6625"
Sep 09 21:57:02.825994 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:57:02.825835    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-65c8845bdf-t6625"
Sep 09 21:57:02.827460 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:57:02.827431    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-65c8845bdf-t6625" podUID="ef59f263-d3c3-4201-a6af-0f0992e3fd25" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/health\": dial tcp 10.132.0.85:8000: connect: connection refused"
Sep 09 21:57:07.286873 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:57:07.286835    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:57:12.794231 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:57:12.794182    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-684c688995-bqsvh" podUID="897e3e04-a7d6-4670-baf2-d8f91d40cabf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8001/health\": dial tcp 10.132.0.84:8001: connect: connection refused"
Sep 09 21:57:12.812078 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:57:12.812049    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-684c688995-bqsvh"
Sep 09 21:57:12.826307 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:57:12.826273    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-65c8845bdf-t6625" podUID="ef59f263-d3c3-4201-a6af-0f0992e3fd25" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/health\": dial tcp 10.132.0.85:8000: connect: connection refused"
Sep 09 21:57:22.793669 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:57:22.793615    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-684c688995-bqsvh" podUID="897e3e04-a7d6-4670-baf2-d8f91d40cabf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8001/health\": dial tcp 10.132.0.84:8001: connect: connection refused"
Sep 09 21:57:22.826725 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:57:22.826677    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-65c8845bdf-t6625" podUID="ef59f263-d3c3-4201-a6af-0f0992e3fd25" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/health\": dial tcp 10.132.0.85:8000: connect: connection refused"
Sep 09 21:57:32.794249 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:57:32.794194    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-684c688995-bqsvh" podUID="897e3e04-a7d6-4670-baf2-d8f91d40cabf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8001/health\": dial tcp 10.132.0.84:8001: connect: connection refused"
Sep 09 21:57:32.826736 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:57:32.826696    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-65c8845bdf-t6625" podUID="ef59f263-d3c3-4201-a6af-0f0992e3fd25" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/health\": dial tcp 10.132.0.85:8000: connect: connection refused"
Sep 09 21:57:42.794286 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:57:42.794234    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-684c688995-bqsvh" podUID="897e3e04-a7d6-4670-baf2-d8f91d40cabf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8001/health\": dial tcp 10.132.0.84:8001: connect: connection refused"
Sep 09 21:57:42.826888 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:57:42.826847    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-65c8845bdf-t6625" podUID="ef59f263-d3c3-4201-a6af-0f0992e3fd25" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/health\": dial tcp 10.132.0.85:8000: connect: connection refused"
Sep 09 21:57:44.012909 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:57:44.012860    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:57:52.794620 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:57:52.794568    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-684c688995-bqsvh" podUID="897e3e04-a7d6-4670-baf2-d8f91d40cabf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8001/health\": dial tcp 10.132.0.84:8001: connect: connection refused"
Sep 09 21:57:52.826348 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:57:52.826312    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-65c8845bdf-t6625" podUID="ef59f263-d3c3-4201-a6af-0f0992e3fd25" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/health\": dial tcp 10.132.0.85:8000: connect: connection refused"
Sep 09 21:58:02.793809 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:58:02.793758    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-684c688995-bqsvh" podUID="897e3e04-a7d6-4670-baf2-d8f91d40cabf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8001/health\": dial tcp 10.132.0.84:8001: connect: connection refused"
Sep 09 21:58:02.826991 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:58:02.826920    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-65c8845bdf-t6625" podUID="ef59f263-d3c3-4201-a6af-0f0992e3fd25" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/health\": dial tcp 10.132.0.85:8000: connect: connection refused"
Sep 09 21:58:04.003438 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:58:04.003390    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:58:04.097767 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:58:04.097710    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:58:10.390749 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:58:10.390715    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:58:10.490112 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:58:10.490078    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:58:12.793747 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:58:12.793692    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-684c688995-bqsvh" podUID="897e3e04-a7d6-4670-baf2-d8f91d40cabf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8001/health\": dial tcp 10.132.0.84:8001: connect: connection refused"
Sep 09 21:58:12.826097 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:58:12.826062    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-65c8845bdf-t6625" podUID="ef59f263-d3c3-4201-a6af-0f0992e3fd25" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/health\": dial tcp 10.132.0.85:8000: connect: connection refused"
Sep 09 21:58:22.794575 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:58:22.794525    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-684c688995-bqsvh" podUID="897e3e04-a7d6-4670-baf2-d8f91d40cabf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8001/health\": dial tcp 10.132.0.84:8001: connect: connection refused"
Sep 09 21:58:22.826151 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:58:22.826103    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-65c8845bdf-t6625" podUID="ef59f263-d3c3-4201-a6af-0f0992e3fd25" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/health\": dial tcp 10.132.0.85:8000: connect: connection refused"
Sep 09 21:58:23.033189 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:58:23.033147    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:58:23.312330 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:58:23.312289    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:58:32.794600 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:58:32.794502    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-684c688995-bqsvh" podUID="897e3e04-a7d6-4670-baf2-d8f91d40cabf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8001/health\": dial tcp 10.132.0.84:8001: connect: connection refused"
Sep 09 21:58:32.826765 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:58:32.826721    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-65c8845bdf-t6625" podUID="ef59f263-d3c3-4201-a6af-0f0992e3fd25" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/health\": dial tcp 10.132.0.85:8000: connect: connection refused"
Sep 09 21:58:38.634620 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:58:38.634584    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:58:42.794296 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:58:42.794243    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-684c688995-bqsvh" podUID="897e3e04-a7d6-4670-baf2-d8f91d40cabf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8001/health\": dial tcp 10.132.0.84:8001: connect: connection refused"
Sep 09 21:58:42.826149 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:58:42.826113    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-65c8845bdf-t6625" podUID="ef59f263-d3c3-4201-a6af-0f0992e3fd25" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/health\": dial tcp 10.132.0.85:8000: connect: connection refused"
Sep 09 21:58:52.794068 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:58:52.794022    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-684c688995-bqsvh" podUID="897e3e04-a7d6-4670-baf2-d8f91d40cabf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8001/health\": dial tcp 10.132.0.84:8001: connect: connection refused"
Sep 09 21:58:52.826156 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:58:52.826107    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-65c8845bdf-t6625" podUID="ef59f263-d3c3-4201-a6af-0f0992e3fd25" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/health\": dial tcp 10.132.0.85:8000: connect: connection refused"
Sep 09 21:59:02.794335 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:59:02.794285    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-684c688995-bqsvh" podUID="897e3e04-a7d6-4670-baf2-d8f91d40cabf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8001/health\": dial tcp 10.132.0.84:8001: connect: connection refused"
Sep 09 21:59:02.826509 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:59:02.826457    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-65c8845bdf-t6625" podUID="ef59f263-d3c3-4201-a6af-0f0992e3fd25" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/health\": dial tcp 10.132.0.85:8000: connect: connection refused"
Sep 09 21:59:12.794381 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:59:12.794309    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-684c688995-bqsvh" podUID="897e3e04-a7d6-4670-baf2-d8f91d40cabf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8001/health\": dial tcp 10.132.0.84:8001: connect: connection refused"
Sep 09 21:59:12.826220 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:59:12.826176    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-65c8845bdf-t6625" podUID="ef59f263-d3c3-4201-a6af-0f0992e3fd25" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/health\": dial tcp 10.132.0.85:8000: connect: connection refused"
Sep 09 21:59:22.794194 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:59:22.794140    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-684c688995-bqsvh" podUID="897e3e04-a7d6-4670-baf2-d8f91d40cabf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8001/health\": dial tcp 10.132.0.84:8001: connect: connection refused"
Sep 09 21:59:22.826361 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:59:22.826309    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-65c8845bdf-t6625" podUID="ef59f263-d3c3-4201-a6af-0f0992e3fd25" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/health\": dial tcp 10.132.0.85:8000: connect: connection refused"
Sep 09 21:59:32.803473 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:59:32.803439    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-684c688995-bqsvh"
Sep 09 21:59:32.818408 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:59:32.818358    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-684c688995-bqsvh"
Sep 09 21:59:32.835452 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:59:32.835427    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-65c8845bdf-t6625"
Sep 09 21:59:32.843597 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:59:32.843573    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-65c8845bdf-t6625"
Sep 09 21:59:40.032939 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:59:40.032899    2576 scope.go:117] "RemoveContainer" containerID="9b865adb658402471de367a2e75d068722dba9d4e0e965e3ae8931f6391b9fd7"
Sep 09 21:59:40.042134 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:59:40.042112    2576 scope.go:117] "RemoveContainer" containerID="369c9b6c27eb282acd36f18710ddb2ff4533cd97eadc38620d46cc0c1ba7fb2b"
Sep 09 21:59:41.909992 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:59:41.909955    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-684c688995-bqsvh"]
Sep 09 21:59:41.915196 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:59:41.915169    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-65c8845bdf-t6625"]
Sep 09 21:59:41.922709 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:59:41.922678    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:59:41.995761 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:59:41.995727    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 21:59:56.935478 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:59:56.935435    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-684c688995-bqsvh" podUID="897e3e04-a7d6-4670-baf2-d8f91d40cabf" containerName="main" containerID="cri-o://38e892034b712611a00c8ccf683813bbbd6cd43e183de4cef41c3ad036eb00c2" gracePeriod=45
Sep 09 21:59:56.938465 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:59:56.938317    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-65c8845bdf-t6625" podUID="ef59f263-d3c3-4201-a6af-0f0992e3fd25" containerName="main" containerID="cri-o://3b7a6aaddcd001a29ef1ec31bdaadf3350cc6ddeffab54d762f33127dacd9f9c" gracePeriod=45
Sep 09 21:59:57.750538 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:59:57.750412    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-684c688995-bqsvh" podUID="897e3e04-a7d6-4670-baf2-d8f91d40cabf" containerName="llm-d-routing-sidecar" containerID="cri-o://0899c4f53f535398749e6d2d8974994461531f3ccceebd3c36638b2028517604" gracePeriod=45
Sep 09 21:59:57.883273 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:59:57.883248    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-65c8845bdf-t6625"
Sep 09 21:59:57.985943 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:59:57.985876    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef59f263-d3c3-4201-a6af-0f0992e3fd25-model-cache\") pod \"ef59f263-d3c3-4201-a6af-0f0992e3fd25\" (UID: \"ef59f263-d3c3-4201-a6af-0f0992e3fd25\") "
Sep 09 21:59:57.985943 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:59:57.985934    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef59f263-d3c3-4201-a6af-0f0992e3fd25-tls-certs\") pod \"ef59f263-d3c3-4201-a6af-0f0992e3fd25\" (UID: \"ef59f263-d3c3-4201-a6af-0f0992e3fd25\") "
Sep 09 21:59:57.986518 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:59:57.985986    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ef59f263-d3c3-4201-a6af-0f0992e3fd25-kserve-provision-location\") pod \"ef59f263-d3c3-4201-a6af-0f0992e3fd25\" (UID: \"ef59f263-d3c3-4201-a6af-0f0992e3fd25\") "
Sep 09 21:59:57.986518 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:59:57.986010    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef59f263-d3c3-4201-a6af-0f0992e3fd25-dshm\") pod \"ef59f263-d3c3-4201-a6af-0f0992e3fd25\" (UID: \"ef59f263-d3c3-4201-a6af-0f0992e3fd25\") "
Sep 09 21:59:57.986518 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:59:57.986051    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ef59f263-d3c3-4201-a6af-0f0992e3fd25-cabundle-cert\") pod \"ef59f263-d3c3-4201-a6af-0f0992e3fd25\" (UID: \"ef59f263-d3c3-4201-a6af-0f0992e3fd25\") "
Sep 09 21:59:57.986518 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:59:57.986097    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jbnw8\" (UniqueName: \"kubernetes.io/projected/ef59f263-d3c3-4201-a6af-0f0992e3fd25-kube-api-access-jbnw8\") pod \"ef59f263-d3c3-4201-a6af-0f0992e3fd25\" (UID: \"ef59f263-d3c3-4201-a6af-0f0992e3fd25\") "
Sep 09 21:59:57.986518 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:59:57.986129    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef59f263-d3c3-4201-a6af-0f0992e3fd25-home\") pod \"ef59f263-d3c3-4201-a6af-0f0992e3fd25\" (UID: \"ef59f263-d3c3-4201-a6af-0f0992e3fd25\") "
Sep 09 21:59:57.986518 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:59:57.986157    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef59f263-d3c3-4201-a6af-0f0992e3fd25-tmp-dir\") pod \"ef59f263-d3c3-4201-a6af-0f0992e3fd25\" (UID: \"ef59f263-d3c3-4201-a6af-0f0992e3fd25\") "
Sep 09 21:59:57.986518 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:59:57.986220    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef59f263-d3c3-4201-a6af-0f0992e3fd25-model-cache" (OuterVolumeSpecName: "model-cache") pod "ef59f263-d3c3-4201-a6af-0f0992e3fd25" (UID: "ef59f263-d3c3-4201-a6af-0f0992e3fd25"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:59:57.986518 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:59:57.986508    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef59f263-d3c3-4201-a6af-0f0992e3fd25-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ef59f263-d3c3-4201-a6af-0f0992e3fd25" (UID: "ef59f263-d3c3-4201-a6af-0f0992e3fd25"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:59:57.987468 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:59:57.986539    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef59f263-d3c3-4201-a6af-0f0992e3fd25-model-cache\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:59:57.987468 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:59:57.987285    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef59f263-d3c3-4201-a6af-0f0992e3fd25-home" (OuterVolumeSpecName: "home") pod "ef59f263-d3c3-4201-a6af-0f0992e3fd25" (UID: "ef59f263-d3c3-4201-a6af-0f0992e3fd25"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:59:57.988966 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:59:57.988936    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef59f263-d3c3-4201-a6af-0f0992e3fd25-dshm" (OuterVolumeSpecName: "dshm") pod "ef59f263-d3c3-4201-a6af-0f0992e3fd25" (UID: "ef59f263-d3c3-4201-a6af-0f0992e3fd25"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:59:57.989053 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:59:57.989020    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef59f263-d3c3-4201-a6af-0f0992e3fd25-kube-api-access-jbnw8" (OuterVolumeSpecName: "kube-api-access-jbnw8") pod "ef59f263-d3c3-4201-a6af-0f0992e3fd25" (UID: "ef59f263-d3c3-4201-a6af-0f0992e3fd25"). InnerVolumeSpecName "kube-api-access-jbnw8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:59:57.989123 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:59:57.989080    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef59f263-d3c3-4201-a6af-0f0992e3fd25-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ef59f263-d3c3-4201-a6af-0f0992e3fd25" (UID: "ef59f263-d3c3-4201-a6af-0f0992e3fd25"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:59:57.998518 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:59:57.998498    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-684c688995-bqsvh"
Sep 09 21:59:58.002003 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:59:58.001978    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef59f263-d3c3-4201-a6af-0f0992e3fd25-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ef59f263-d3c3-4201-a6af-0f0992e3fd25" (UID: "ef59f263-d3c3-4201-a6af-0f0992e3fd25"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:59:58.011574 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:59:58.011553    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef59f263-d3c3-4201-a6af-0f0992e3fd25-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ef59f263-d3c3-4201-a6af-0f0992e3fd25" (UID: "ef59f263-d3c3-4201-a6af-0f0992e3fd25"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:59:58.087184 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:59:58.087160    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/897e3e04-a7d6-4670-baf2-d8f91d40cabf-model-cache\") pod \"897e3e04-a7d6-4670-baf2-d8f91d40cabf\" (UID: \"897e3e04-a7d6-4670-baf2-d8f91d40cabf\") "
Sep 09 21:59:58.087349 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:59:58.087205    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/897e3e04-a7d6-4670-baf2-d8f91d40cabf-tls-certs\") pod \"897e3e04-a7d6-4670-baf2-d8f91d40cabf\" (UID: \"897e3e04-a7d6-4670-baf2-d8f91d40cabf\") "
Sep 09 21:59:58.087349 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:59:58.087230    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/897e3e04-a7d6-4670-baf2-d8f91d40cabf-tmp-dir\") pod \"897e3e04-a7d6-4670-baf2-d8f91d40cabf\" (UID: \"897e3e04-a7d6-4670-baf2-d8f91d40cabf\") "
Sep 09 21:59:58.087349 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:59:58.087248    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/897e3e04-a7d6-4670-baf2-d8f91d40cabf-home\") pod \"897e3e04-a7d6-4670-baf2-d8f91d40cabf\" (UID: \"897e3e04-a7d6-4670-baf2-d8f91d40cabf\") "
Sep 09 21:59:58.087349 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:59:58.087269    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/897e3e04-a7d6-4670-baf2-d8f91d40cabf-dshm\") pod \"897e3e04-a7d6-4670-baf2-d8f91d40cabf\" (UID: \"897e3e04-a7d6-4670-baf2-d8f91d40cabf\") "
Sep 09 21:59:58.087349 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:59:58.087290    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/897e3e04-a7d6-4670-baf2-d8f91d40cabf-cabundle-cert\") pod \"897e3e04-a7d6-4670-baf2-d8f91d40cabf\" (UID: \"897e3e04-a7d6-4670-baf2-d8f91d40cabf\") "
Sep 09 21:59:58.087349 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:59:58.087321    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/897e3e04-a7d6-4670-baf2-d8f91d40cabf-kserve-provision-location\") pod \"897e3e04-a7d6-4670-baf2-d8f91d40cabf\" (UID: \"897e3e04-a7d6-4670-baf2-d8f91d40cabf\") "
Sep 09 21:59:58.087710 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:59:58.087406    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-swfzz\" (UniqueName: \"kubernetes.io/projected/897e3e04-a7d6-4670-baf2-d8f91d40cabf-kube-api-access-swfzz\") pod \"897e3e04-a7d6-4670-baf2-d8f91d40cabf\" (UID: \"897e3e04-a7d6-4670-baf2-d8f91d40cabf\") "
Sep 09 21:59:58.087710 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:59:58.087510    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/897e3e04-a7d6-4670-baf2-d8f91d40cabf-model-cache" (OuterVolumeSpecName: "model-cache") pod "897e3e04-a7d6-4670-baf2-d8f91d40cabf" (UID: "897e3e04-a7d6-4670-baf2-d8f91d40cabf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:59:58.087813 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:59:58.087730    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef59f263-d3c3-4201-a6af-0f0992e3fd25-tls-certs\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:59:58.087813 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:59:58.087753    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ef59f263-d3c3-4201-a6af-0f0992e3fd25-kserve-provision-location\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:59:58.087813 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:59:58.087766    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef59f263-d3c3-4201-a6af-0f0992e3fd25-dshm\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:59:58.087813 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:59:58.087779    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/897e3e04-a7d6-4670-baf2-d8f91d40cabf-model-cache\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:59:58.087813 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:59:58.087790    2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ef59f263-d3c3-4201-a6af-0f0992e3fd25-cabundle-cert\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:59:58.087813 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:59:58.087803    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jbnw8\" (UniqueName: \"kubernetes.io/projected/ef59f263-d3c3-4201-a6af-0f0992e3fd25-kube-api-access-jbnw8\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:59:58.087813 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:59:58.087813    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef59f263-d3c3-4201-a6af-0f0992e3fd25-home\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:59:58.088167 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:59:58.087826    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef59f263-d3c3-4201-a6af-0f0992e3fd25-tmp-dir\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:59:58.088167 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:59:58.088024    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/897e3e04-a7d6-4670-baf2-d8f91d40cabf-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "897e3e04-a7d6-4670-baf2-d8f91d40cabf" (UID: "897e3e04-a7d6-4670-baf2-d8f91d40cabf"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:59:58.088167 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:59:58.088044    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/897e3e04-a7d6-4670-baf2-d8f91d40cabf-home" (OuterVolumeSpecName: "home") pod "897e3e04-a7d6-4670-baf2-d8f91d40cabf" (UID: "897e3e04-a7d6-4670-baf2-d8f91d40cabf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:59:58.089793 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:59:58.089764    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/897e3e04-a7d6-4670-baf2-d8f91d40cabf-kube-api-access-swfzz" (OuterVolumeSpecName: "kube-api-access-swfzz") pod "897e3e04-a7d6-4670-baf2-d8f91d40cabf" (UID: "897e3e04-a7d6-4670-baf2-d8f91d40cabf"). InnerVolumeSpecName "kube-api-access-swfzz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:59:58.089927 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:59:58.089912    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/897e3e04-a7d6-4670-baf2-d8f91d40cabf-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "897e3e04-a7d6-4670-baf2-d8f91d40cabf" (UID: "897e3e04-a7d6-4670-baf2-d8f91d40cabf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:59:58.090050 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:59:58.090037    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/897e3e04-a7d6-4670-baf2-d8f91d40cabf-dshm" (OuterVolumeSpecName: "dshm") pod "897e3e04-a7d6-4670-baf2-d8f91d40cabf" (UID: "897e3e04-a7d6-4670-baf2-d8f91d40cabf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:59:58.100718 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:59:58.100685    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/897e3e04-a7d6-4670-baf2-d8f91d40cabf-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "897e3e04-a7d6-4670-baf2-d8f91d40cabf" (UID: "897e3e04-a7d6-4670-baf2-d8f91d40cabf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:59:58.103652 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:59:58.103631    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/897e3e04-a7d6-4670-baf2-d8f91d40cabf-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "897e3e04-a7d6-4670-baf2-d8f91d40cabf" (UID: "897e3e04-a7d6-4670-baf2-d8f91d40cabf"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:59:58.189129 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:59:58.189104    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swfzz\" (UniqueName: \"kubernetes.io/projected/897e3e04-a7d6-4670-baf2-d8f91d40cabf-kube-api-access-swfzz\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:59:58.189129 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:59:58.189126    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/897e3e04-a7d6-4670-baf2-d8f91d40cabf-tls-certs\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:59:58.189288 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:59:58.189136    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/897e3e04-a7d6-4670-baf2-d8f91d40cabf-tmp-dir\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:59:58.189288 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:59:58.189144    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/897e3e04-a7d6-4670-baf2-d8f91d40cabf-home\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:59:58.189288 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:59:58.189151    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/897e3e04-a7d6-4670-baf2-d8f91d40cabf-dshm\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:59:58.189288 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:59:58.189158    2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/897e3e04-a7d6-4670-baf2-d8f91d40cabf-cabundle-cert\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:59:58.189288 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:59:58.189166    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/897e3e04-a7d6-4670-baf2-d8f91d40cabf-kserve-provision-location\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 21:59:58.472256 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:59:58.472223    2576 generic.go:356] "Generic (PLEG): container finished" podID="897e3e04-a7d6-4670-baf2-d8f91d40cabf" containerID="38e892034b712611a00c8ccf683813bbbd6cd43e183de4cef41c3ad036eb00c2" exitCode=0
Sep 09 21:59:58.472256 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:59:58.472247    2576 generic.go:356] "Generic (PLEG): container finished" podID="897e3e04-a7d6-4670-baf2-d8f91d40cabf" containerID="0899c4f53f535398749e6d2d8974994461531f3ccceebd3c36638b2028517604" exitCode=0
Sep 09 21:59:58.472528 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:59:58.472297    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-684c688995-bqsvh"
Sep 09 21:59:58.472528 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:59:58.472302    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-684c688995-bqsvh" event={"ID":"897e3e04-a7d6-4670-baf2-d8f91d40cabf","Type":"ContainerDied","Data":"38e892034b712611a00c8ccf683813bbbd6cd43e183de4cef41c3ad036eb00c2"}
Sep 09 21:59:58.472528 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:59:58.472353    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-684c688995-bqsvh" event={"ID":"897e3e04-a7d6-4670-baf2-d8f91d40cabf","Type":"ContainerDied","Data":"0899c4f53f535398749e6d2d8974994461531f3ccceebd3c36638b2028517604"}
Sep 09 21:59:58.472528 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:59:58.472382    2576 scope.go:117] "RemoveContainer" containerID="38e892034b712611a00c8ccf683813bbbd6cd43e183de4cef41c3ad036eb00c2"
Sep 09 21:59:58.472528 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:59:58.472394    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-684c688995-bqsvh" event={"ID":"897e3e04-a7d6-4670-baf2-d8f91d40cabf","Type":"ContainerDied","Data":"f623a50b747ec14fff6427f513aadafec053ab4c7aa3373bffe8b176a7a3e6e5"}
Sep 09 21:59:58.474128 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:59:58.474103    2576 generic.go:356] "Generic (PLEG): container finished" podID="ef59f263-d3c3-4201-a6af-0f0992e3fd25" containerID="3b7a6aaddcd001a29ef1ec31bdaadf3350cc6ddeffab54d762f33127dacd9f9c" exitCode=0
Sep 09 21:59:58.474238 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:59:58.474150    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-65c8845bdf-t6625"
Sep 09 21:59:58.474238 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:59:58.474221    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-65c8845bdf-t6625" event={"ID":"ef59f263-d3c3-4201-a6af-0f0992e3fd25","Type":"ContainerDied","Data":"3b7a6aaddcd001a29ef1ec31bdaadf3350cc6ddeffab54d762f33127dacd9f9c"}
Sep 09 21:59:58.474317 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:59:58.474276    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-65c8845bdf-t6625" event={"ID":"ef59f263-d3c3-4201-a6af-0f0992e3fd25","Type":"ContainerDied","Data":"ba89cd99eae9bf056c855ae1c910a57bd39fb1fed0045f61df881d6c3df22e7a"}
Sep 09 21:59:58.482107 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:59:58.482086    2576 scope.go:117] "RemoveContainer" containerID="c9168e7f67f68723344d8eb289d963a8594825b9caf60a6e2415f5b04af0b60a"
Sep 09 21:59:58.490605 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:59:58.490591    2576 scope.go:117] "RemoveContainer" containerID="0899c4f53f535398749e6d2d8974994461531f3ccceebd3c36638b2028517604"
Sep 09 21:59:58.499080 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:59:58.499059    2576 scope.go:117] "RemoveContainer" containerID="38e892034b712611a00c8ccf683813bbbd6cd43e183de4cef41c3ad036eb00c2"
Sep 09 21:59:58.499475 ip-10-0-134-72 kubenswrapper[2576]: E0909 21:59:58.499445    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38e892034b712611a00c8ccf683813bbbd6cd43e183de4cef41c3ad036eb00c2\": container with ID starting with 38e892034b712611a00c8ccf683813bbbd6cd43e183de4cef41c3ad036eb00c2 not found: ID does not exist" containerID="38e892034b712611a00c8ccf683813bbbd6cd43e183de4cef41c3ad036eb00c2"
Sep 09 21:59:58.499557 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:59:58.499489    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38e892034b712611a00c8ccf683813bbbd6cd43e183de4cef41c3ad036eb00c2"} err="failed to get container status \"38e892034b712611a00c8ccf683813bbbd6cd43e183de4cef41c3ad036eb00c2\": rpc error: code = NotFound desc = could not find container \"38e892034b712611a00c8ccf683813bbbd6cd43e183de4cef41c3ad036eb00c2\": container with ID starting with 38e892034b712611a00c8ccf683813bbbd6cd43e183de4cef41c3ad036eb00c2 not found: ID does not exist"
Sep 09 21:59:58.499557 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:59:58.499512    2576 scope.go:117] "RemoveContainer" containerID="c9168e7f67f68723344d8eb289d963a8594825b9caf60a6e2415f5b04af0b60a"
Sep 09 21:59:58.499815 ip-10-0-134-72 kubenswrapper[2576]: E0909 21:59:58.499797    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9168e7f67f68723344d8eb289d963a8594825b9caf60a6e2415f5b04af0b60a\": container with ID starting with c9168e7f67f68723344d8eb289d963a8594825b9caf60a6e2415f5b04af0b60a not found: ID does not exist" containerID="c9168e7f67f68723344d8eb289d963a8594825b9caf60a6e2415f5b04af0b60a"
Sep 09 21:59:58.499854 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:59:58.499821    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9168e7f67f68723344d8eb289d963a8594825b9caf60a6e2415f5b04af0b60a"} err="failed to get container status \"c9168e7f67f68723344d8eb289d963a8594825b9caf60a6e2415f5b04af0b60a\": rpc error: code = NotFound desc = could not find container \"c9168e7f67f68723344d8eb289d963a8594825b9caf60a6e2415f5b04af0b60a\": container with ID starting with c9168e7f67f68723344d8eb289d963a8594825b9caf60a6e2415f5b04af0b60a not found: ID does not exist"
Sep 09 21:59:58.499854 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:59:58.499835    2576 scope.go:117] "RemoveContainer" containerID="0899c4f53f535398749e6d2d8974994461531f3ccceebd3c36638b2028517604"
Sep 09 21:59:58.499928 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:59:58.499856    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-684c688995-bqsvh"]
Sep 09 21:59:58.500070 ip-10-0-134-72 kubenswrapper[2576]: E0909 21:59:58.500053    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0899c4f53f535398749e6d2d8974994461531f3ccceebd3c36638b2028517604\": container with ID starting with 0899c4f53f535398749e6d2d8974994461531f3ccceebd3c36638b2028517604 not found: ID does not exist" containerID="0899c4f53f535398749e6d2d8974994461531f3ccceebd3c36638b2028517604"
Sep 09 21:59:58.500119 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:59:58.500075    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0899c4f53f535398749e6d2d8974994461531f3ccceebd3c36638b2028517604"} err="failed to get container status \"0899c4f53f535398749e6d2d8974994461531f3ccceebd3c36638b2028517604\": rpc error: code = NotFound desc = could not find container \"0899c4f53f535398749e6d2d8974994461531f3ccceebd3c36638b2028517604\": container with ID starting with 0899c4f53f535398749e6d2d8974994461531f3ccceebd3c36638b2028517604 not found: ID does not exist"
Sep 09 21:59:58.500119 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:59:58.500088    2576 scope.go:117] "RemoveContainer" containerID="38e892034b712611a00c8ccf683813bbbd6cd43e183de4cef41c3ad036eb00c2"
Sep 09 21:59:58.500334 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:59:58.500316    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38e892034b712611a00c8ccf683813bbbd6cd43e183de4cef41c3ad036eb00c2"} err="failed to get container status \"38e892034b712611a00c8ccf683813bbbd6cd43e183de4cef41c3ad036eb00c2\": rpc error: code = NotFound desc = could not find container \"38e892034b712611a00c8ccf683813bbbd6cd43e183de4cef41c3ad036eb00c2\": container with ID starting with 38e892034b712611a00c8ccf683813bbbd6cd43e183de4cef41c3ad036eb00c2 not found: ID does not exist"
Sep 09 21:59:58.500395 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:59:58.500336    2576 scope.go:117] "RemoveContainer" containerID="c9168e7f67f68723344d8eb289d963a8594825b9caf60a6e2415f5b04af0b60a"
Sep 09 21:59:58.500571 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:59:58.500553    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9168e7f67f68723344d8eb289d963a8594825b9caf60a6e2415f5b04af0b60a"} err="failed to get container status \"c9168e7f67f68723344d8eb289d963a8594825b9caf60a6e2415f5b04af0b60a\": rpc error: code = NotFound desc = could not find container \"c9168e7f67f68723344d8eb289d963a8594825b9caf60a6e2415f5b04af0b60a\": container with ID starting with c9168e7f67f68723344d8eb289d963a8594825b9caf60a6e2415f5b04af0b60a not found: ID does not exist"
Sep 09 21:59:58.500614 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:59:58.500572    2576 scope.go:117] "RemoveContainer" containerID="0899c4f53f535398749e6d2d8974994461531f3ccceebd3c36638b2028517604"
Sep 09 21:59:58.500781 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:59:58.500760    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0899c4f53f535398749e6d2d8974994461531f3ccceebd3c36638b2028517604"} err="failed to get container status \"0899c4f53f535398749e6d2d8974994461531f3ccceebd3c36638b2028517604\": rpc error: code = NotFound desc = could not find container \"0899c4f53f535398749e6d2d8974994461531f3ccceebd3c36638b2028517604\": container with ID starting with 0899c4f53f535398749e6d2d8974994461531f3ccceebd3c36638b2028517604 not found: ID does not exist"
Sep 09 21:59:58.500815 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:59:58.500783    2576 scope.go:117] "RemoveContainer" containerID="3b7a6aaddcd001a29ef1ec31bdaadf3350cc6ddeffab54d762f33127dacd9f9c"
Sep 09 21:59:58.505412 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:59:58.505347    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-684c688995-bqsvh"]
Sep 09 21:59:58.508981 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:59:58.508966    2576 scope.go:117] "RemoveContainer" containerID="154bd45ee58259db58eb761af585283901533a4d4cf77d38359a070d53562137"
Sep 09 21:59:58.517047 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:59:58.517029    2576 scope.go:117] "RemoveContainer" containerID="3b7a6aaddcd001a29ef1ec31bdaadf3350cc6ddeffab54d762f33127dacd9f9c"
Sep 09 21:59:58.517328 ip-10-0-134-72 kubenswrapper[2576]: E0909 21:59:58.517313    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b7a6aaddcd001a29ef1ec31bdaadf3350cc6ddeffab54d762f33127dacd9f9c\": container with ID starting with 3b7a6aaddcd001a29ef1ec31bdaadf3350cc6ddeffab54d762f33127dacd9f9c not found: ID does not exist" containerID="3b7a6aaddcd001a29ef1ec31bdaadf3350cc6ddeffab54d762f33127dacd9f9c"
Sep 09 21:59:58.517399 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:59:58.517332    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b7a6aaddcd001a29ef1ec31bdaadf3350cc6ddeffab54d762f33127dacd9f9c"} err="failed to get container status \"3b7a6aaddcd001a29ef1ec31bdaadf3350cc6ddeffab54d762f33127dacd9f9c\": rpc error: code = NotFound desc = could not find container \"3b7a6aaddcd001a29ef1ec31bdaadf3350cc6ddeffab54d762f33127dacd9f9c\": container with ID starting with 3b7a6aaddcd001a29ef1ec31bdaadf3350cc6ddeffab54d762f33127dacd9f9c not found: ID does not exist"
Sep 09 21:59:58.517399 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:59:58.517349    2576 scope.go:117] "RemoveContainer" containerID="154bd45ee58259db58eb761af585283901533a4d4cf77d38359a070d53562137"
Sep 09 21:59:58.517642 ip-10-0-134-72 kubenswrapper[2576]: E0909 21:59:58.517626    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"154bd45ee58259db58eb761af585283901533a4d4cf77d38359a070d53562137\": container with ID starting with 154bd45ee58259db58eb761af585283901533a4d4cf77d38359a070d53562137 not found: ID does not exist" containerID="154bd45ee58259db58eb761af585283901533a4d4cf77d38359a070d53562137"
Sep 09 21:59:58.517695 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:59:58.517644    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"154bd45ee58259db58eb761af585283901533a4d4cf77d38359a070d53562137"} err="failed to get container status \"154bd45ee58259db58eb761af585283901533a4d4cf77d38359a070d53562137\": rpc error: code = NotFound desc = could not find container \"154bd45ee58259db58eb761af585283901533a4d4cf77d38359a070d53562137\": container with ID starting with 154bd45ee58259db58eb761af585283901533a4d4cf77d38359a070d53562137 not found: ID does not exist"
Sep 09 21:59:58.518188 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:59:58.518173    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-65c8845bdf-t6625"]
Sep 09 21:59:58.523914 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:59:58.523875    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-65c8845bdf-t6625"]
Sep 09 21:59:59.408961 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:59:59.408918    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="897e3e04-a7d6-4670-baf2-d8f91d40cabf" path="/var/lib/kubelet/pods/897e3e04-a7d6-4670-baf2-d8f91d40cabf/volumes"
Sep 09 21:59:59.409614 ip-10-0-134-72 kubenswrapper[2576]: I0909 21:59:59.409594    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ef59f263-d3c3-4201-a6af-0f0992e3fd25" path="/var/lib/kubelet/pods/ef59f263-d3c3-4201-a6af-0f0992e3fd25/volumes"
Sep 09 22:00:28.136236 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:00:28.136202    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 22:00:28.200204 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:00:28.200167    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 22:00:31.896006 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:00:31.895968    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 22:00:31.973310 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:00:31.973267    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 22:00:32.194291 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:00:32.194262    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-qm75c"]
Sep 09 22:00:32.194610 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:00:32.194597    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef59f263-d3c3-4201-a6af-0f0992e3fd25" containerName="main"
Sep 09 22:00:32.194657 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:00:32.194611    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef59f263-d3c3-4201-a6af-0f0992e3fd25" containerName="main"
Sep 09 22:00:32.194657 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:00:32.194627    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="897e3e04-a7d6-4670-baf2-d8f91d40cabf" containerName="llm-d-routing-sidecar"
Sep 09 22:00:32.194657 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:00:32.194632    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="897e3e04-a7d6-4670-baf2-d8f91d40cabf" containerName="llm-d-routing-sidecar"
Sep 09 22:00:32.194657 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:00:32.194641    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="897e3e04-a7d6-4670-baf2-d8f91d40cabf" containerName="storage-initializer"
Sep 09 22:00:32.194657 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:00:32.194647    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="897e3e04-a7d6-4670-baf2-d8f91d40cabf" containerName="storage-initializer"
Sep 09 22:00:32.194657 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:00:32.194655    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="897e3e04-a7d6-4670-baf2-d8f91d40cabf" containerName="main"
Sep 09 22:00:32.194657 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:00:32.194660    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="897e3e04-a7d6-4670-baf2-d8f91d40cabf" containerName="main"
Sep 09 22:00:32.194879 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:00:32.194667    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef59f263-d3c3-4201-a6af-0f0992e3fd25" containerName="storage-initializer"
Sep 09 22:00:32.194879 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:00:32.194672    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef59f263-d3c3-4201-a6af-0f0992e3fd25" containerName="storage-initializer"
Sep 09 22:00:32.194879 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:00:32.194723    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef59f263-d3c3-4201-a6af-0f0992e3fd25" containerName="main"
Sep 09 22:00:32.194879 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:00:32.194729    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="897e3e04-a7d6-4670-baf2-d8f91d40cabf" containerName="main"
Sep 09 22:00:32.194879 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:00:32.194738    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="897e3e04-a7d6-4670-baf2-d8f91d40cabf" containerName="llm-d-routing-sidecar"
Sep 09 22:00:32.197411 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:00:32.197394    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-qm75c"
Sep 09 22:00:32.201655 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:00:32.201632    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-gateway-2-openshift-default-dockercfg-hrfv9\""
Sep 09 22:00:32.201983 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:00:32.201967    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"openshift-gw-ca-root-cert\""
Sep 09 22:00:32.202767 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:00:32.202743    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"openshift-service-ca.crt\""
Sep 09 22:00:32.202879 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:00:32.202847    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"kube-root-ca.crt\""
Sep 09 22:00:32.230192 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:00:32.230162    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-qm75c"]
Sep 09 22:00:32.241789 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:00:32.241757    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 22:00:32.281541 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:00:32.281511    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71a5281b-b9f1-4ea5-a489-8f7dc9089465-credential-socket\") pod \"router-gateway-2-openshift-default-5c874f788-qm75c\" (UID: \"71a5281b-b9f1-4ea5-a489-8f7dc9089465\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-qm75c"
Sep 09 22:00:32.281697 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:00:32.281547    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71a5281b-b9f1-4ea5-a489-8f7dc9089465-istio-envoy\") pod \"router-gateway-2-openshift-default-5c874f788-qm75c\" (UID: \"71a5281b-b9f1-4ea5-a489-8f7dc9089465\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-qm75c"
Sep 09 22:00:32.281697 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:00:32.281579    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71a5281b-b9f1-4ea5-a489-8f7dc9089465-istio-data\") pod \"router-gateway-2-openshift-default-5c874f788-qm75c\" (UID: \"71a5281b-b9f1-4ea5-a489-8f7dc9089465\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-qm75c"
Sep 09 22:00:32.281697 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:00:32.281666    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzktl\" (UniqueName: \"kubernetes.io/projected/71a5281b-b9f1-4ea5-a489-8f7dc9089465-kube-api-access-pzktl\") pod \"router-gateway-2-openshift-default-5c874f788-qm75c\" (UID: \"71a5281b-b9f1-4ea5-a489-8f7dc9089465\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-qm75c"
Sep 09 22:00:32.281697 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:00:32.281691    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71a5281b-b9f1-4ea5-a489-8f7dc9089465-workload-certs\") pod \"router-gateway-2-openshift-default-5c874f788-qm75c\" (UID: \"71a5281b-b9f1-4ea5-a489-8f7dc9089465\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-qm75c"
Sep 09 22:00:32.281838 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:00:32.281734    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71a5281b-b9f1-4ea5-a489-8f7dc9089465-workload-socket\") pod \"router-gateway-2-openshift-default-5c874f788-qm75c\" (UID: \"71a5281b-b9f1-4ea5-a489-8f7dc9089465\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-qm75c"
Sep 09 22:00:32.281838 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:00:32.281749    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71a5281b-b9f1-4ea5-a489-8f7dc9089465-istio-token\") pod \"router-gateway-2-openshift-default-5c874f788-qm75c\" (UID: \"71a5281b-b9f1-4ea5-a489-8f7dc9089465\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-qm75c"
Sep 09 22:00:32.281838 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:00:32.281766    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71a5281b-b9f1-4ea5-a489-8f7dc9089465-istio-podinfo\") pod \"router-gateway-2-openshift-default-5c874f788-qm75c\" (UID: \"71a5281b-b9f1-4ea5-a489-8f7dc9089465\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-qm75c"
Sep 09 22:00:32.281838 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:00:32.281800    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71a5281b-b9f1-4ea5-a489-8f7dc9089465-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-5c874f788-qm75c\" (UID: \"71a5281b-b9f1-4ea5-a489-8f7dc9089465\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-qm75c"
Sep 09 22:00:32.328739 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:00:32.328699    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 22:00:32.382688 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:00:32.382654    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71a5281b-b9f1-4ea5-a489-8f7dc9089465-credential-socket\") pod \"router-gateway-2-openshift-default-5c874f788-qm75c\" (UID: \"71a5281b-b9f1-4ea5-a489-8f7dc9089465\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-qm75c"
Sep 09 22:00:32.382865 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:00:32.382697    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71a5281b-b9f1-4ea5-a489-8f7dc9089465-istio-envoy\") pod \"router-gateway-2-openshift-default-5c874f788-qm75c\" (UID: \"71a5281b-b9f1-4ea5-a489-8f7dc9089465\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-qm75c"
Sep 09 22:00:32.382865 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:00:32.382739    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71a5281b-b9f1-4ea5-a489-8f7dc9089465-istio-data\") pod \"router-gateway-2-openshift-default-5c874f788-qm75c\" (UID: \"71a5281b-b9f1-4ea5-a489-8f7dc9089465\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-qm75c"
Sep 09 22:00:32.382865 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:00:32.382795    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzktl\" (UniqueName: \"kubernetes.io/projected/71a5281b-b9f1-4ea5-a489-8f7dc9089465-kube-api-access-pzktl\") pod \"router-gateway-2-openshift-default-5c874f788-qm75c\" (UID: \"71a5281b-b9f1-4ea5-a489-8f7dc9089465\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-qm75c"
Sep 09 22:00:32.382865 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:00:32.382829    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71a5281b-b9f1-4ea5-a489-8f7dc9089465-workload-certs\") pod \"router-gateway-2-openshift-default-5c874f788-qm75c\" (UID: \"71a5281b-b9f1-4ea5-a489-8f7dc9089465\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-qm75c"
Sep 09 22:00:32.382865 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:00:32.382862    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71a5281b-b9f1-4ea5-a489-8f7dc9089465-workload-socket\") pod \"router-gateway-2-openshift-default-5c874f788-qm75c\" (UID: \"71a5281b-b9f1-4ea5-a489-8f7dc9089465\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-qm75c"
Sep 09 22:00:32.383120 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:00:32.382886    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71a5281b-b9f1-4ea5-a489-8f7dc9089465-istio-token\") pod \"router-gateway-2-openshift-default-5c874f788-qm75c\" (UID: \"71a5281b-b9f1-4ea5-a489-8f7dc9089465\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-qm75c"
Sep 09 22:00:32.383120 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:00:32.382905    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71a5281b-b9f1-4ea5-a489-8f7dc9089465-istio-podinfo\") pod \"router-gateway-2-openshift-default-5c874f788-qm75c\" (UID: \"71a5281b-b9f1-4ea5-a489-8f7dc9089465\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-qm75c"
Sep 09 22:00:32.383120 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:00:32.382927    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71a5281b-b9f1-4ea5-a489-8f7dc9089465-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-5c874f788-qm75c\" (UID: \"71a5281b-b9f1-4ea5-a489-8f7dc9089465\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-qm75c"
Sep 09 22:00:32.383269 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:00:32.383117    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71a5281b-b9f1-4ea5-a489-8f7dc9089465-credential-socket\") pod \"router-gateway-2-openshift-default-5c874f788-qm75c\" (UID: \"71a5281b-b9f1-4ea5-a489-8f7dc9089465\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-qm75c"
Sep 09 22:00:32.383269 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:00:32.383144    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71a5281b-b9f1-4ea5-a489-8f7dc9089465-istio-data\") pod \"router-gateway-2-openshift-default-5c874f788-qm75c\" (UID: \"71a5281b-b9f1-4ea5-a489-8f7dc9089465\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-qm75c"
Sep 09 22:00:32.383269 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:00:32.383241    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71a5281b-b9f1-4ea5-a489-8f7dc9089465-workload-socket\") pod \"router-gateway-2-openshift-default-5c874f788-qm75c\" (UID: \"71a5281b-b9f1-4ea5-a489-8f7dc9089465\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-qm75c"
Sep 09 22:00:32.383433 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:00:32.383319    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71a5281b-b9f1-4ea5-a489-8f7dc9089465-workload-certs\") pod \"router-gateway-2-openshift-default-5c874f788-qm75c\" (UID: \"71a5281b-b9f1-4ea5-a489-8f7dc9089465\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-qm75c"
Sep 09 22:00:32.383637 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:00:32.383619    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71a5281b-b9f1-4ea5-a489-8f7dc9089465-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-5c874f788-qm75c\" (UID: \"71a5281b-b9f1-4ea5-a489-8f7dc9089465\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-qm75c"
Sep 09 22:00:32.385496 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:00:32.385469    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71a5281b-b9f1-4ea5-a489-8f7dc9089465-istio-envoy\") pod \"router-gateway-2-openshift-default-5c874f788-qm75c\" (UID: \"71a5281b-b9f1-4ea5-a489-8f7dc9089465\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-qm75c"
Sep 09 22:00:32.385806 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:00:32.385787    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71a5281b-b9f1-4ea5-a489-8f7dc9089465-istio-podinfo\") pod \"router-gateway-2-openshift-default-5c874f788-qm75c\" (UID: \"71a5281b-b9f1-4ea5-a489-8f7dc9089465\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-qm75c"
Sep 09 22:00:32.393683 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:00:32.393661    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71a5281b-b9f1-4ea5-a489-8f7dc9089465-istio-token\") pod \"router-gateway-2-openshift-default-5c874f788-qm75c\" (UID: \"71a5281b-b9f1-4ea5-a489-8f7dc9089465\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-qm75c"
Sep 09 22:00:32.393887 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:00:32.393868    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzktl\" (UniqueName: \"kubernetes.io/projected/71a5281b-b9f1-4ea5-a489-8f7dc9089465-kube-api-access-pzktl\") pod \"router-gateway-2-openshift-default-5c874f788-qm75c\" (UID: \"71a5281b-b9f1-4ea5-a489-8f7dc9089465\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-qm75c"
Sep 09 22:00:32.507345 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:00:32.507252    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-qm75c"
Sep 09 22:00:32.795834 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:00:32.795811    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-qm75c"]
Sep 09 22:00:32.797816 ip-10-0-134-72 kubenswrapper[2576]: W0909 22:00:32.797791    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71a5281b_b9f1_4ea5_a489_8f7dc9089465.slice/crio-81892fd4f4a38187c8b5d627878a494c71f81bf07d2cd2886619f95ae4d20d85 WatchSource:0}: Error finding container 81892fd4f4a38187c8b5d627878a494c71f81bf07d2cd2886619f95ae4d20d85: Status 404 returned error can't find the container with id 81892fd4f4a38187c8b5d627878a494c71f81bf07d2cd2886619f95ae4d20d85
Sep 09 22:00:32.799864 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:00:32.799832    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 09 22:00:32.799954 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:00:32.799908    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 09 22:00:32.799954 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:00:32.799935    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 09 22:00:33.648359 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:00:33.648316    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-qm75c" event={"ID":"71a5281b-b9f1-4ea5-a489-8f7dc9089465","Type":"ContainerStarted","Data":"fd524abeae7a414bcb33dfea6a45c103fd5836cbb72ab71426ba2193eb84f110"}
Sep 09 22:00:33.648813 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:00:33.648392    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-qm75c" event={"ID":"71a5281b-b9f1-4ea5-a489-8f7dc9089465","Type":"ContainerStarted","Data":"81892fd4f4a38187c8b5d627878a494c71f81bf07d2cd2886619f95ae4d20d85"}
Sep 09 22:00:33.672122 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:00:33.672071    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-qm75c" podStartSLOduration=1.672055268 podStartE2EDuration="1.672055268s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 22:00:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 22:00:33.670598769 +0000 UTC m=+4076.784284639" watchObservedRunningTime="2026-09-09 22:00:33.672055268 +0000 UTC m=+4076.785741127"
Sep 09 22:00:34.507455 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:00:34.507423    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-qm75c"
Sep 09 22:00:34.512233 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:00:34.512200    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-qm75c"
Sep 09 22:00:34.653273 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:00:34.653237    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-qm75c"
Sep 09 22:00:34.654117 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:00:34.654096    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-qm75c"
Sep 09 22:00:36.340879 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:00:36.340839    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 22:00:38.731575 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:00:38.731534    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 22:00:44.712850 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:00:44.712768    2576 generic.go:356] "Generic (PLEG): container finished" podID="71a5281b-b9f1-4ea5-a489-8f7dc9089465" containerID="fd524abeae7a414bcb33dfea6a45c103fd5836cbb72ab71426ba2193eb84f110" exitCode=0
Sep 09 22:00:44.713258 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:00:44.712846    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-qm75c" event={"ID":"71a5281b-b9f1-4ea5-a489-8f7dc9089465","Type":"ContainerDied","Data":"fd524abeae7a414bcb33dfea6a45c103fd5836cbb72ab71426ba2193eb84f110"}
Sep 09 22:00:44.713315 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:00:44.713301    2576 scope.go:117] "RemoveContainer" containerID="fd524abeae7a414bcb33dfea6a45c103fd5836cbb72ab71426ba2193eb84f110"
Sep 09 22:00:44.714303 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:00:44.714277    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 09 22:00:44.714392 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:00:44.714342    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 09 22:00:44.714441 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:00:44.714390    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 09 22:00:44.781747 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:00:44.781712    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 22:00:44.883330 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:00:44.883294    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 22:00:45.031389 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:00:45.031281    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5566488c4c-qbxf5"]
Sep 09 22:00:45.033853 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:00:45.033829    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5566488c4c-qbxf5"
Sep 09 22:00:45.036161 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:00:45.036135    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"seaweedfs-s3-creds\""
Sep 09 22:00:45.037238 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:00:45.037039    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-kserve-self-signed-certs\""
Sep 09 22:00:45.037238 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:00:45.037050    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-kserve-dockercfg-qbbpx\""
Sep 09 22:00:45.037461 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:00:45.037289    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"odh-kserve-custom-ca-bundle\""
Sep 09 22:00:45.037729 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:00:45.037707    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"default-dockercfg-6xgvz\""
Sep 09 22:00:45.052546 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:00:45.052518    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5566488c4c-qbxf5"]
Sep 09 22:00:45.105928 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:00:45.105893    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/462f4d7b-9ada-4c9e-97b2-4b3a9372bcc3-dshm\") pod \"router-with-refs-pd-test-kserve-5566488c4c-qbxf5\" (UID: \"462f4d7b-9ada-4c9e-97b2-4b3a9372bcc3\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5566488c4c-qbxf5"
Sep 09 22:00:45.106110 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:00:45.105956    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/462f4d7b-9ada-4c9e-97b2-4b3a9372bcc3-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-5566488c4c-qbxf5\" (UID: \"462f4d7b-9ada-4c9e-97b2-4b3a9372bcc3\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5566488c4c-qbxf5"
Sep 09 22:00:45.106110 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:00:45.106044    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/462f4d7b-9ada-4c9e-97b2-4b3a9372bcc3-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-5566488c4c-qbxf5\" (UID: \"462f4d7b-9ada-4c9e-97b2-4b3a9372bcc3\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5566488c4c-qbxf5"
Sep 09 22:00:45.106110 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:00:45.106095    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/462f4d7b-9ada-4c9e-97b2-4b3a9372bcc3-model-cache\") pod \"router-with-refs-pd-test-kserve-5566488c4c-qbxf5\" (UID: \"462f4d7b-9ada-4c9e-97b2-4b3a9372bcc3\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5566488c4c-qbxf5"
Sep 09 22:00:45.106243 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:00:45.106133    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/462f4d7b-9ada-4c9e-97b2-4b3a9372bcc3-tmp-dir\") pod \"router-with-refs-pd-test-kserve-5566488c4c-qbxf5\" (UID: \"462f4d7b-9ada-4c9e-97b2-4b3a9372bcc3\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5566488c4c-qbxf5"
Sep 09 22:00:45.106243 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:00:45.106179    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/462f4d7b-9ada-4c9e-97b2-4b3a9372bcc3-tls-certs\") pod \"router-with-refs-pd-test-kserve-5566488c4c-qbxf5\" (UID: \"462f4d7b-9ada-4c9e-97b2-4b3a9372bcc3\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5566488c4c-qbxf5"
Sep 09 22:00:45.106243 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:00:45.106216    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/462f4d7b-9ada-4c9e-97b2-4b3a9372bcc3-home\") pod \"router-with-refs-pd-test-kserve-5566488c4c-qbxf5\" (UID: \"462f4d7b-9ada-4c9e-97b2-4b3a9372bcc3\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5566488c4c-qbxf5"
Sep 09 22:00:45.106389 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:00:45.106245    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mh8bb\" (UniqueName: \"kubernetes.io/projected/462f4d7b-9ada-4c9e-97b2-4b3a9372bcc3-kube-api-access-mh8bb\") pod \"router-with-refs-pd-test-kserve-5566488c4c-qbxf5\" (UID: \"462f4d7b-9ada-4c9e-97b2-4b3a9372bcc3\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5566488c4c-qbxf5"
Sep 09 22:00:45.207450 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:00:45.207411    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/462f4d7b-9ada-4c9e-97b2-4b3a9372bcc3-dshm\") pod \"router-with-refs-pd-test-kserve-5566488c4c-qbxf5\" (UID: \"462f4d7b-9ada-4c9e-97b2-4b3a9372bcc3\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5566488c4c-qbxf5"
Sep 09 22:00:45.207643 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:00:45.207484    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/462f4d7b-9ada-4c9e-97b2-4b3a9372bcc3-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-5566488c4c-qbxf5\" (UID: \"462f4d7b-9ada-4c9e-97b2-4b3a9372bcc3\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5566488c4c-qbxf5"
Sep 09 22:00:45.207643 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:00:45.207540    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/462f4d7b-9ada-4c9e-97b2-4b3a9372bcc3-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-5566488c4c-qbxf5\" (UID: \"462f4d7b-9ada-4c9e-97b2-4b3a9372bcc3\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5566488c4c-qbxf5"
Sep 09 22:00:45.207643 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:00:45.207576    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/462f4d7b-9ada-4c9e-97b2-4b3a9372bcc3-model-cache\") pod \"router-with-refs-pd-test-kserve-5566488c4c-qbxf5\" (UID: \"462f4d7b-9ada-4c9e-97b2-4b3a9372bcc3\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5566488c4c-qbxf5"
Sep 09 22:00:45.207643 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:00:45.207606    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/462f4d7b-9ada-4c9e-97b2-4b3a9372bcc3-tmp-dir\") pod \"router-with-refs-pd-test-kserve-5566488c4c-qbxf5\" (UID: \"462f4d7b-9ada-4c9e-97b2-4b3a9372bcc3\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5566488c4c-qbxf5"
Sep 09 22:00:45.207899 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:00:45.207647    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/462f4d7b-9ada-4c9e-97b2-4b3a9372bcc3-tls-certs\") pod \"router-with-refs-pd-test-kserve-5566488c4c-qbxf5\" (UID: \"462f4d7b-9ada-4c9e-97b2-4b3a9372bcc3\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5566488c4c-qbxf5"
Sep 09 22:00:45.207899 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:00:45.207674    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/462f4d7b-9ada-4c9e-97b2-4b3a9372bcc3-home\") pod \"router-with-refs-pd-test-kserve-5566488c4c-qbxf5\" (UID: \"462f4d7b-9ada-4c9e-97b2-4b3a9372bcc3\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5566488c4c-qbxf5"
Sep 09 22:00:45.207899 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:00:45.207704    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mh8bb\" (UniqueName: \"kubernetes.io/projected/462f4d7b-9ada-4c9e-97b2-4b3a9372bcc3-kube-api-access-mh8bb\") pod \"router-with-refs-pd-test-kserve-5566488c4c-qbxf5\" (UID: \"462f4d7b-9ada-4c9e-97b2-4b3a9372bcc3\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5566488c4c-qbxf5"
Sep 09 22:00:45.208062 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:00:45.207971    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/462f4d7b-9ada-4c9e-97b2-4b3a9372bcc3-tmp-dir\") pod \"router-with-refs-pd-test-kserve-5566488c4c-qbxf5\" (UID: \"462f4d7b-9ada-4c9e-97b2-4b3a9372bcc3\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5566488c4c-qbxf5"
Sep 09 22:00:45.208313 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:00:45.208236    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/462f4d7b-9ada-4c9e-97b2-4b3a9372bcc3-home\") pod \"router-with-refs-pd-test-kserve-5566488c4c-qbxf5\" (UID: \"462f4d7b-9ada-4c9e-97b2-4b3a9372bcc3\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5566488c4c-qbxf5"
Sep 09 22:00:45.208313 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:00:45.208259    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/462f4d7b-9ada-4c9e-97b2-4b3a9372bcc3-model-cache\") pod \"router-with-refs-pd-test-kserve-5566488c4c-qbxf5\" (UID: \"462f4d7b-9ada-4c9e-97b2-4b3a9372bcc3\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5566488c4c-qbxf5"
Sep 09 22:00:45.208313 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:00:45.208295    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/462f4d7b-9ada-4c9e-97b2-4b3a9372bcc3-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-5566488c4c-qbxf5\" (UID: \"462f4d7b-9ada-4c9e-97b2-4b3a9372bcc3\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5566488c4c-qbxf5"
Sep 09 22:00:45.208509 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:00:45.208312    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/462f4d7b-9ada-4c9e-97b2-4b3a9372bcc3-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-5566488c4c-qbxf5\" (UID: \"462f4d7b-9ada-4c9e-97b2-4b3a9372bcc3\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5566488c4c-qbxf5"
Sep 09 22:00:45.209964 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:00:45.209942    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/462f4d7b-9ada-4c9e-97b2-4b3a9372bcc3-dshm\") pod \"router-with-refs-pd-test-kserve-5566488c4c-qbxf5\" (UID: \"462f4d7b-9ada-4c9e-97b2-4b3a9372bcc3\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5566488c4c-qbxf5"
Sep 09 22:00:45.210263 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:00:45.210244    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/462f4d7b-9ada-4c9e-97b2-4b3a9372bcc3-tls-certs\") pod \"router-with-refs-pd-test-kserve-5566488c4c-qbxf5\" (UID: \"462f4d7b-9ada-4c9e-97b2-4b3a9372bcc3\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5566488c4c-qbxf5"
Sep 09 22:00:45.216959 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:00:45.216936    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mh8bb\" (UniqueName: \"kubernetes.io/projected/462f4d7b-9ada-4c9e-97b2-4b3a9372bcc3-kube-api-access-mh8bb\") pod \"router-with-refs-pd-test-kserve-5566488c4c-qbxf5\" (UID: \"462f4d7b-9ada-4c9e-97b2-4b3a9372bcc3\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5566488c4c-qbxf5"
Sep 09 22:00:45.345291 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:00:45.345260    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5566488c4c-qbxf5"
Sep 09 22:00:45.631546 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:00:45.631510    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5566488c4c-qbxf5"]
Sep 09 22:00:45.633454 ip-10-0-134-72 kubenswrapper[2576]: W0909 22:00:45.633424    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod462f4d7b_9ada_4c9e_97b2_4b3a9372bcc3.slice/crio-22e9755a8b0961edad3c49f44f4c56e0859ddd214c8a738b5c37221328a1de58 WatchSource:0}: Error finding container 22e9755a8b0961edad3c49f44f4c56e0859ddd214c8a738b5c37221328a1de58: Status 404 returned error can't find the container with id 22e9755a8b0961edad3c49f44f4c56e0859ddd214c8a738b5c37221328a1de58
Sep 09 22:00:45.720306 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:00:45.720271    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-qm75c" event={"ID":"71a5281b-b9f1-4ea5-a489-8f7dc9089465","Type":"ContainerStarted","Data":"97521b6d7255a9f825371908bb6a93bf87cb9620ef30ea57480733d867c06379"}
Sep 09 22:00:45.722071 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:00:45.722044    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5566488c4c-qbxf5" event={"ID":"462f4d7b-9ada-4c9e-97b2-4b3a9372bcc3","Type":"ContainerStarted","Data":"16ee3c8429cfbe17346349a019ef35801ad2207eb2cfa2f1756da4e57cb4df57"}
Sep 09 22:00:45.722171 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:00:45.722100    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5566488c4c-qbxf5"
Sep 09 22:00:45.722171 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:00:45.722122    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5566488c4c-qbxf5" event={"ID":"462f4d7b-9ada-4c9e-97b2-4b3a9372bcc3","Type":"ContainerStarted","Data":"22e9755a8b0961edad3c49f44f4c56e0859ddd214c8a738b5c37221328a1de58"}
Sep 09 22:00:45.869353 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:00:45.869316    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 22:00:45.967548 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:00:45.967433    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 22:00:46.507718 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:00:46.507674    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-qm75c"
Sep 09 22:00:46.508980 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:00:46.508947    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-qm75c" podUID="71a5281b-b9f1-4ea5-a489-8f7dc9089465" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Sep 09 22:00:46.729779 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:00:46.729738    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5566488c4c-qbxf5" event={"ID":"462f4d7b-9ada-4c9e-97b2-4b3a9372bcc3","Type":"ContainerStarted","Data":"3c7d33c31ca34fbe88b4efd84fb911d81715f64413e4df9a46be138075f1de0b"}
Sep 09 22:00:47.507913 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:00:47.507872    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-qm75c" podUID="71a5281b-b9f1-4ea5-a489-8f7dc9089465" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Sep 09 22:00:48.508044 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:00:48.508004    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-qm75c" podUID="71a5281b-b9f1-4ea5-a489-8f7dc9089465" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Sep 09 22:00:48.741813 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:00:48.741715    2576 generic.go:356] "Generic (PLEG): container finished" podID="462f4d7b-9ada-4c9e-97b2-4b3a9372bcc3" containerID="3c7d33c31ca34fbe88b4efd84fb911d81715f64413e4df9a46be138075f1de0b" exitCode=0
Sep 09 22:00:48.741813 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:00:48.741784    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5566488c4c-qbxf5" event={"ID":"462f4d7b-9ada-4c9e-97b2-4b3a9372bcc3","Type":"ContainerDied","Data":"3c7d33c31ca34fbe88b4efd84fb911d81715f64413e4df9a46be138075f1de0b"}
Sep 09 22:00:49.508246 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:00:49.508199    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-qm75c" podUID="71a5281b-b9f1-4ea5-a489-8f7dc9089465" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Sep 09 22:00:49.653576 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:00:49.653537    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-qm75c"
Sep 09 22:00:49.749211 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:00:49.749171    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5566488c4c-qbxf5" event={"ID":"462f4d7b-9ada-4c9e-97b2-4b3a9372bcc3","Type":"ContainerStarted","Data":"bf009df330c9e367deac4f6d21d45ef2b9b11b20f1633ba93f12e50f808c8f41"}
Sep 09 22:00:49.777403 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:00:49.777251    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5566488c4c-qbxf5" podStartSLOduration=2.215475229 podStartE2EDuration="4.777233094s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.561757865s" isStatefulPod=false podCreationTimestamp="2026-09-09 22:00:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 22:00:49.772081563 +0000 UTC m=+4092.885767423" watchObservedRunningTime="2026-09-09 22:00:49.777233094 +0000 UTC m=+4092.890918953"
Sep 09 22:00:50.507714 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:00:50.507676    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-qm75c" podUID="71a5281b-b9f1-4ea5-a489-8f7dc9089465" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Sep 09 22:00:51.513263 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:00:51.513225    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-qm75c"
Sep 09 22:00:51.514299 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:00:51.514271    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-qm75c"
Sep 09 22:00:55.346352 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:00:55.346315    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5566488c4c-qbxf5"
Sep 09 22:00:55.346855 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:00:55.346412    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5566488c4c-qbxf5"
Sep 09 22:00:55.347801 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:00:55.347761    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5566488c4c-qbxf5" podUID="462f4d7b-9ada-4c9e-97b2-4b3a9372bcc3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8001/health\": dial tcp 10.132.0.87:8001: connect: connection refused"
Sep 09 22:00:55.359601 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:00:55.359577    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5566488c4c-qbxf5"
Sep 09 22:01:05.346119 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:01:05.346079    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5566488c4c-qbxf5" podUID="462f4d7b-9ada-4c9e-97b2-4b3a9372bcc3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8001/health\": dial tcp 10.132.0.87:8001: connect: connection refused"
Sep 09 22:01:15.346611 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:01:15.346555    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5566488c4c-qbxf5" podUID="462f4d7b-9ada-4c9e-97b2-4b3a9372bcc3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8001/health\": dial tcp 10.132.0.87:8001: connect: connection refused"
Sep 09 22:01:18.252299 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:01:18.252252    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 22:01:18.354207 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:01:18.354164    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 22:01:22.787929 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:01:22.787869    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 22:01:24.449875 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:01:24.449836    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 22:01:25.346435 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:01:25.346386    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5566488c4c-qbxf5" podUID="462f4d7b-9ada-4c9e-97b2-4b3a9372bcc3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8001/health\": dial tcp 10.132.0.87:8001: connect: connection refused"
Sep 09 22:01:25.715582 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:01:25.715491    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 22:01:25.816650 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:01:25.816618    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 22:01:30.370169 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:01:30.370073    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f95b98795-qj5tp"]
Sep 09 22:01:30.373641 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:01:30.373617    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f95b98795-qj5tp"
Sep 09 22:01:30.375925 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:01:30.375907    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"seaweedfs-s3-creds\""
Sep 09 22:01:30.376297 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:01:30.376278    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"kube-root-ca.crt\""
Sep 09 22:01:30.376416 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:01:30.376337    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"odh-kserve-custom-ca-bundle\""
Sep 09 22:01:30.376416 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:01:30.376344    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"prestop-hook-test-kserve-self-signed-certs\""
Sep 09 22:01:30.376416 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:01:30.376409    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"default-dockercfg-nhv94\""
Sep 09 22:01:30.376836 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:01:30.376817    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"openshift-service-ca.crt\""
Sep 09 22:01:30.386138 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:01:30.386108    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f95b98795-qj5tp"]
Sep 09 22:01:30.399307 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:01:30.399282    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 22:01:30.424428 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:01:30.424391    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34982e1c-46a7-4b8f-add2-3ae056a2794f-home\") pod \"prestop-hook-test-kserve-7f95b98795-qj5tp\" (UID: \"34982e1c-46a7-4b8f-add2-3ae056a2794f\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f95b98795-qj5tp"
Sep 09 22:01:30.424589 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:01:30.424444    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34982e1c-46a7-4b8f-add2-3ae056a2794f-tmp-dir\") pod \"prestop-hook-test-kserve-7f95b98795-qj5tp\" (UID: \"34982e1c-46a7-4b8f-add2-3ae056a2794f\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f95b98795-qj5tp"
Sep 09 22:01:30.424589 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:01:30.424510    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34982e1c-46a7-4b8f-add2-3ae056a2794f-dshm\") pod \"prestop-hook-test-kserve-7f95b98795-qj5tp\" (UID: \"34982e1c-46a7-4b8f-add2-3ae056a2794f\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f95b98795-qj5tp"
Sep 09 22:01:30.424589 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:01:30.424535    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34982e1c-46a7-4b8f-add2-3ae056a2794f-tls-certs\") pod \"prestop-hook-test-kserve-7f95b98795-qj5tp\" (UID: \"34982e1c-46a7-4b8f-add2-3ae056a2794f\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f95b98795-qj5tp"
Sep 09 22:01:30.424589 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:01:30.424568    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27fpv\" (UniqueName: \"kubernetes.io/projected/34982e1c-46a7-4b8f-add2-3ae056a2794f-kube-api-access-27fpv\") pod \"prestop-hook-test-kserve-7f95b98795-qj5tp\" (UID: \"34982e1c-46a7-4b8f-add2-3ae056a2794f\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f95b98795-qj5tp"
Sep 09 22:01:30.424793 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:01:30.424612    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/34982e1c-46a7-4b8f-add2-3ae056a2794f-kserve-provision-location\") pod \"prestop-hook-test-kserve-7f95b98795-qj5tp\" (UID: \"34982e1c-46a7-4b8f-add2-3ae056a2794f\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f95b98795-qj5tp"
Sep 09 22:01:30.424793 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:01:30.424638    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34982e1c-46a7-4b8f-add2-3ae056a2794f-model-cache\") pod \"prestop-hook-test-kserve-7f95b98795-qj5tp\" (UID: \"34982e1c-46a7-4b8f-add2-3ae056a2794f\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f95b98795-qj5tp"
Sep 09 22:01:30.424793 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:01:30.424656    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/34982e1c-46a7-4b8f-add2-3ae056a2794f-cabundle-cert\") pod \"prestop-hook-test-kserve-7f95b98795-qj5tp\" (UID: \"34982e1c-46a7-4b8f-add2-3ae056a2794f\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f95b98795-qj5tp"
Sep 09 22:01:30.521023 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:01:30.520982    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 22:01:30.525931 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:01:30.525909    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34982e1c-46a7-4b8f-add2-3ae056a2794f-dshm\") pod \"prestop-hook-test-kserve-7f95b98795-qj5tp\" (UID: \"34982e1c-46a7-4b8f-add2-3ae056a2794f\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f95b98795-qj5tp"
Sep 09 22:01:30.526060 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:01:30.525956    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34982e1c-46a7-4b8f-add2-3ae056a2794f-tls-certs\") pod \"prestop-hook-test-kserve-7f95b98795-qj5tp\" (UID: \"34982e1c-46a7-4b8f-add2-3ae056a2794f\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f95b98795-qj5tp"
Sep 09 22:01:30.526060 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:01:30.525982    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27fpv\" (UniqueName: \"kubernetes.io/projected/34982e1c-46a7-4b8f-add2-3ae056a2794f-kube-api-access-27fpv\") pod \"prestop-hook-test-kserve-7f95b98795-qj5tp\" (UID: \"34982e1c-46a7-4b8f-add2-3ae056a2794f\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f95b98795-qj5tp"
Sep 09 22:01:30.526060 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:01:30.526026    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/34982e1c-46a7-4b8f-add2-3ae056a2794f-kserve-provision-location\") pod \"prestop-hook-test-kserve-7f95b98795-qj5tp\" (UID: \"34982e1c-46a7-4b8f-add2-3ae056a2794f\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f95b98795-qj5tp"
Sep 09 22:01:30.526230 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:01:30.526060    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34982e1c-46a7-4b8f-add2-3ae056a2794f-model-cache\") pod \"prestop-hook-test-kserve-7f95b98795-qj5tp\" (UID: \"34982e1c-46a7-4b8f-add2-3ae056a2794f\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f95b98795-qj5tp"
Sep 09 22:01:30.526230 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:01:30.526087    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/34982e1c-46a7-4b8f-add2-3ae056a2794f-cabundle-cert\") pod \"prestop-hook-test-kserve-7f95b98795-qj5tp\" (UID: \"34982e1c-46a7-4b8f-add2-3ae056a2794f\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f95b98795-qj5tp"
Sep 09 22:01:30.526230 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:01:30.526125    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34982e1c-46a7-4b8f-add2-3ae056a2794f-home\") pod \"prestop-hook-test-kserve-7f95b98795-qj5tp\" (UID: \"34982e1c-46a7-4b8f-add2-3ae056a2794f\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f95b98795-qj5tp"
Sep 09 22:01:30.526230 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:01:30.526178    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34982e1c-46a7-4b8f-add2-3ae056a2794f-tmp-dir\") pod \"prestop-hook-test-kserve-7f95b98795-qj5tp\" (UID: \"34982e1c-46a7-4b8f-add2-3ae056a2794f\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f95b98795-qj5tp"
Sep 09 22:01:30.526554 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:01:30.526528    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34982e1c-46a7-4b8f-add2-3ae056a2794f-model-cache\") pod \"prestop-hook-test-kserve-7f95b98795-qj5tp\" (UID: \"34982e1c-46a7-4b8f-add2-3ae056a2794f\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f95b98795-qj5tp"
Sep 09 22:01:30.526947 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:01:30.526583    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/34982e1c-46a7-4b8f-add2-3ae056a2794f-kserve-provision-location\") pod \"prestop-hook-test-kserve-7f95b98795-qj5tp\" (UID: \"34982e1c-46a7-4b8f-add2-3ae056a2794f\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f95b98795-qj5tp"
Sep 09 22:01:30.527047 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:01:30.526618    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34982e1c-46a7-4b8f-add2-3ae056a2794f-home\") pod \"prestop-hook-test-kserve-7f95b98795-qj5tp\" (UID: \"34982e1c-46a7-4b8f-add2-3ae056a2794f\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f95b98795-qj5tp"
Sep 09 22:01:30.527047 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:01:30.526678    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34982e1c-46a7-4b8f-add2-3ae056a2794f-tmp-dir\") pod \"prestop-hook-test-kserve-7f95b98795-qj5tp\" (UID: \"34982e1c-46a7-4b8f-add2-3ae056a2794f\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f95b98795-qj5tp"
Sep 09 22:01:30.527386 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:01:30.527334    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/34982e1c-46a7-4b8f-add2-3ae056a2794f-cabundle-cert\") pod \"prestop-hook-test-kserve-7f95b98795-qj5tp\" (UID: \"34982e1c-46a7-4b8f-add2-3ae056a2794f\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f95b98795-qj5tp"
Sep 09 22:01:30.528429 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:01:30.528401    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34982e1c-46a7-4b8f-add2-3ae056a2794f-dshm\") pod \"prestop-hook-test-kserve-7f95b98795-qj5tp\" (UID: \"34982e1c-46a7-4b8f-add2-3ae056a2794f\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f95b98795-qj5tp"
Sep 09 22:01:30.528814 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:01:30.528794    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34982e1c-46a7-4b8f-add2-3ae056a2794f-tls-certs\") pod \"prestop-hook-test-kserve-7f95b98795-qj5tp\" (UID: \"34982e1c-46a7-4b8f-add2-3ae056a2794f\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f95b98795-qj5tp"
Sep 09 22:01:30.534175 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:01:30.534147    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-27fpv\" (UniqueName: \"kubernetes.io/projected/34982e1c-46a7-4b8f-add2-3ae056a2794f-kube-api-access-27fpv\") pod \"prestop-hook-test-kserve-7f95b98795-qj5tp\" (UID: \"34982e1c-46a7-4b8f-add2-3ae056a2794f\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f95b98795-qj5tp"
Sep 09 22:01:30.584749 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:01:30.584703    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 22:01:30.670028 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:01:30.669933    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 22:01:30.685078 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:01:30.685043    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f95b98795-qj5tp"
Sep 09 22:01:30.964597 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:01:30.964569    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f95b98795-qj5tp"]
Sep 09 22:01:30.976313 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:01:30.976285    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f95b98795-qj5tp" event={"ID":"34982e1c-46a7-4b8f-add2-3ae056a2794f","Type":"ContainerStarted","Data":"7738d591303234e66ca31ece7d95a97900e65e834e0b24ba371f8311fad6abac"}
Sep 09 22:01:31.983469 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:01:31.983413    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f95b98795-qj5tp" event={"ID":"34982e1c-46a7-4b8f-add2-3ae056a2794f","Type":"ContainerStarted","Data":"b68fbe64cf35226496415268604c277719656cd45d51ce7531042b5736e551c3"}
Sep 09 22:01:33.995813 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:01:33.995777    2576 generic.go:356] "Generic (PLEG): container finished" podID="34982e1c-46a7-4b8f-add2-3ae056a2794f" containerID="b68fbe64cf35226496415268604c277719656cd45d51ce7531042b5736e551c3" exitCode=0
Sep 09 22:01:33.996204 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:01:33.995852    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f95b98795-qj5tp" event={"ID":"34982e1c-46a7-4b8f-add2-3ae056a2794f","Type":"ContainerDied","Data":"b68fbe64cf35226496415268604c277719656cd45d51ce7531042b5736e551c3"}
Sep 09 22:01:35.003812 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:01:35.003776    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f95b98795-qj5tp" event={"ID":"34982e1c-46a7-4b8f-add2-3ae056a2794f","Type":"ContainerStarted","Data":"f5c3fdddd3e84bf9763de303313bf490506f51a1cbc5d62b72f53bfa6a61e88f"}
Sep 09 22:01:35.030324 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:01:35.030263    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f95b98795-qj5tp" podStartSLOduration=2.721652197 podStartE2EDuration="5.030246676s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.308594479s" isStatefulPod=false podCreationTimestamp="2026-09-09 22:01:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 22:01:35.028407793 +0000 UTC m=+4138.142093652" watchObservedRunningTime="2026-09-09 22:01:35.030246676 +0000 UTC m=+4138.143932535"
Sep 09 22:01:35.345878 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:01:35.345830    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5566488c4c-qbxf5" podUID="462f4d7b-9ada-4c9e-97b2-4b3a9372bcc3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8001/health\": dial tcp 10.132.0.87:8001: connect: connection refused"
Sep 09 22:01:35.685498 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:01:35.685396    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f95b98795-qj5tp"
Sep 09 22:01:40.685759 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:01:40.685719    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f95b98795-qj5tp"
Sep 09 22:01:40.687789 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:01:40.687763    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f95b98795-qj5tp" podUID="34982e1c-46a7-4b8f-add2-3ae056a2794f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/health\": dial tcp 10.132.0.88:8000: connect: connection refused"
Sep 09 22:01:42.045312 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:01:42.045276    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 22:01:45.345969 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:01:45.345920    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5566488c4c-qbxf5" podUID="462f4d7b-9ada-4c9e-97b2-4b3a9372bcc3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8001/health\": dial tcp 10.132.0.87:8001: connect: connection refused"
Sep 09 22:01:50.686416 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:01:50.686339    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f95b98795-qj5tp" podUID="34982e1c-46a7-4b8f-add2-3ae056a2794f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/health\": dial tcp 10.132.0.88:8000: connect: connection refused"
Sep 09 22:01:55.345805 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:01:55.345754    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5566488c4c-qbxf5" podUID="462f4d7b-9ada-4c9e-97b2-4b3a9372bcc3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8001/health\": dial tcp 10.132.0.87:8001: connect: connection refused"
Sep 09 22:02:00.685718 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:02:00.685665    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f95b98795-qj5tp" podUID="34982e1c-46a7-4b8f-add2-3ae056a2794f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/health\": dial tcp 10.132.0.88:8000: connect: connection refused"
Sep 09 22:02:05.345829 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:02:05.345777    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5566488c4c-qbxf5" podUID="462f4d7b-9ada-4c9e-97b2-4b3a9372bcc3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8001/health\": dial tcp 10.132.0.87:8001: connect: connection refused"
Sep 09 22:02:10.686043 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:02:10.685989    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f95b98795-qj5tp" podUID="34982e1c-46a7-4b8f-add2-3ae056a2794f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/health\": dial tcp 10.132.0.88:8000: connect: connection refused"
Sep 09 22:02:15.346130 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:02:15.346088    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5566488c4c-qbxf5" podUID="462f4d7b-9ada-4c9e-97b2-4b3a9372bcc3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8001/health\": dial tcp 10.132.0.87:8001: connect: connection refused"
Sep 09 22:02:20.686425 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:02:20.686379    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f95b98795-qj5tp" podUID="34982e1c-46a7-4b8f-add2-3ae056a2794f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/health\": dial tcp 10.132.0.88:8000: connect: connection refused"
Sep 09 22:02:25.346035 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:02:25.345990    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5566488c4c-qbxf5" podUID="462f4d7b-9ada-4c9e-97b2-4b3a9372bcc3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8001/health\": dial tcp 10.132.0.87:8001: connect: connection refused"
Sep 09 22:02:30.686099 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:02:30.686045    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f95b98795-qj5tp" podUID="34982e1c-46a7-4b8f-add2-3ae056a2794f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/health\": dial tcp 10.132.0.88:8000: connect: connection refused"
Sep 09 22:02:35.345971 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:02:35.345915    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5566488c4c-qbxf5" podUID="462f4d7b-9ada-4c9e-97b2-4b3a9372bcc3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8001/health\": dial tcp 10.132.0.87:8001: connect: connection refused"
Sep 09 22:02:40.686305 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:02:40.686257    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f95b98795-qj5tp" podUID="34982e1c-46a7-4b8f-add2-3ae056a2794f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/health\": dial tcp 10.132.0.88:8000: connect: connection refused"
Sep 09 22:02:45.356072 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:02:45.356042    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5566488c4c-qbxf5"
Sep 09 22:02:45.368670 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:02:45.368646    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5566488c4c-qbxf5"
Sep 09 22:02:50.686135 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:02:50.686090    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f95b98795-qj5tp" podUID="34982e1c-46a7-4b8f-add2-3ae056a2794f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/health\": dial tcp 10.132.0.88:8000: connect: connection refused"
Sep 09 22:02:57.911877 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:02:57.911814    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 22:02:57.943077 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:02:57.943044    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5566488c4c-qbxf5"]
Sep 09 22:02:59.661170 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:02:59.661070    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 22:03:00.686285 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:03:00.686241    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f95b98795-qj5tp" podUID="34982e1c-46a7-4b8f-add2-3ae056a2794f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/health\": dial tcp 10.132.0.88:8000: connect: connection refused"
Sep 09 22:03:10.685605 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:03:10.685563    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f95b98795-qj5tp" podUID="34982e1c-46a7-4b8f-add2-3ae056a2794f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/health\": dial tcp 10.132.0.88:8000: connect: connection refused"
Sep 09 22:03:12.966470 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:03:12.966391    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5566488c4c-qbxf5" podUID="462f4d7b-9ada-4c9e-97b2-4b3a9372bcc3" containerName="main" containerID="cri-o://bf009df330c9e367deac4f6d21d45ef2b9b11b20f1633ba93f12e50f808c8f41" gracePeriod=45
Sep 09 22:03:13.788030 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:03:13.787988    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5566488c4c-qbxf5" podUID="462f4d7b-9ada-4c9e-97b2-4b3a9372bcc3" containerName="llm-d-routing-sidecar" containerID="cri-o://16ee3c8429cfbe17346349a019ef35801ad2207eb2cfa2f1756da4e57cb4df57" gracePeriod=45
Sep 09 22:03:14.034173 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:03:14.034148    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5566488c4c-qbxf5"
Sep 09 22:03:14.102505 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:03:14.102468    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/462f4d7b-9ada-4c9e-97b2-4b3a9372bcc3-cabundle-cert\") pod \"462f4d7b-9ada-4c9e-97b2-4b3a9372bcc3\" (UID: \"462f4d7b-9ada-4c9e-97b2-4b3a9372bcc3\") "
Sep 09 22:03:14.102661 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:03:14.102544    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/462f4d7b-9ada-4c9e-97b2-4b3a9372bcc3-model-cache\") pod \"462f4d7b-9ada-4c9e-97b2-4b3a9372bcc3\" (UID: \"462f4d7b-9ada-4c9e-97b2-4b3a9372bcc3\") "
Sep 09 22:03:14.102661 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:03:14.102566    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/462f4d7b-9ada-4c9e-97b2-4b3a9372bcc3-tmp-dir\") pod \"462f4d7b-9ada-4c9e-97b2-4b3a9372bcc3\" (UID: \"462f4d7b-9ada-4c9e-97b2-4b3a9372bcc3\") "
Sep 09 22:03:14.102661 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:03:14.102592    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/462f4d7b-9ada-4c9e-97b2-4b3a9372bcc3-kserve-provision-location\") pod \"462f4d7b-9ada-4c9e-97b2-4b3a9372bcc3\" (UID: \"462f4d7b-9ada-4c9e-97b2-4b3a9372bcc3\") "
Sep 09 22:03:14.102661 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:03:14.102608    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/462f4d7b-9ada-4c9e-97b2-4b3a9372bcc3-tls-certs\") pod \"462f4d7b-9ada-4c9e-97b2-4b3a9372bcc3\" (UID: \"462f4d7b-9ada-4c9e-97b2-4b3a9372bcc3\") "
Sep 09 22:03:14.102661 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:03:14.102626    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/462f4d7b-9ada-4c9e-97b2-4b3a9372bcc3-dshm\") pod \"462f4d7b-9ada-4c9e-97b2-4b3a9372bcc3\" (UID: \"462f4d7b-9ada-4c9e-97b2-4b3a9372bcc3\") "
Sep 09 22:03:14.102919 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:03:14.102689    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mh8bb\" (UniqueName: \"kubernetes.io/projected/462f4d7b-9ada-4c9e-97b2-4b3a9372bcc3-kube-api-access-mh8bb\") pod \"462f4d7b-9ada-4c9e-97b2-4b3a9372bcc3\" (UID: \"462f4d7b-9ada-4c9e-97b2-4b3a9372bcc3\") "
Sep 09 22:03:14.102919 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:03:14.102732    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/462f4d7b-9ada-4c9e-97b2-4b3a9372bcc3-home\") pod \"462f4d7b-9ada-4c9e-97b2-4b3a9372bcc3\" (UID: \"462f4d7b-9ada-4c9e-97b2-4b3a9372bcc3\") "
Sep 09 22:03:14.103027 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:03:14.102933    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/462f4d7b-9ada-4c9e-97b2-4b3a9372bcc3-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "462f4d7b-9ada-4c9e-97b2-4b3a9372bcc3" (UID: "462f4d7b-9ada-4c9e-97b2-4b3a9372bcc3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 22:03:14.103084 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:03:14.103068    2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/462f4d7b-9ada-4c9e-97b2-4b3a9372bcc3-cabundle-cert\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 22:03:14.103222 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:03:14.103197    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/462f4d7b-9ada-4c9e-97b2-4b3a9372bcc3-model-cache" (OuterVolumeSpecName: "model-cache") pod "462f4d7b-9ada-4c9e-97b2-4b3a9372bcc3" (UID: "462f4d7b-9ada-4c9e-97b2-4b3a9372bcc3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:03:14.104149 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:03:14.104119    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/462f4d7b-9ada-4c9e-97b2-4b3a9372bcc3-home" (OuterVolumeSpecName: "home") pod "462f4d7b-9ada-4c9e-97b2-4b3a9372bcc3" (UID: "462f4d7b-9ada-4c9e-97b2-4b3a9372bcc3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:03:14.105230 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:03:14.105204    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/462f4d7b-9ada-4c9e-97b2-4b3a9372bcc3-dshm" (OuterVolumeSpecName: "dshm") pod "462f4d7b-9ada-4c9e-97b2-4b3a9372bcc3" (UID: "462f4d7b-9ada-4c9e-97b2-4b3a9372bcc3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:03:14.105450 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:03:14.105426    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/462f4d7b-9ada-4c9e-97b2-4b3a9372bcc3-kube-api-access-mh8bb" (OuterVolumeSpecName: "kube-api-access-mh8bb") pod "462f4d7b-9ada-4c9e-97b2-4b3a9372bcc3" (UID: "462f4d7b-9ada-4c9e-97b2-4b3a9372bcc3"). InnerVolumeSpecName "kube-api-access-mh8bb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 22:03:14.105526 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:03:14.105457    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/462f4d7b-9ada-4c9e-97b2-4b3a9372bcc3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "462f4d7b-9ada-4c9e-97b2-4b3a9372bcc3" (UID: "462f4d7b-9ada-4c9e-97b2-4b3a9372bcc3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 22:03:14.120949 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:03:14.120921    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/462f4d7b-9ada-4c9e-97b2-4b3a9372bcc3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "462f4d7b-9ada-4c9e-97b2-4b3a9372bcc3" (UID: "462f4d7b-9ada-4c9e-97b2-4b3a9372bcc3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:03:14.128606 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:03:14.128585    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/462f4d7b-9ada-4c9e-97b2-4b3a9372bcc3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "462f4d7b-9ada-4c9e-97b2-4b3a9372bcc3" (UID: "462f4d7b-9ada-4c9e-97b2-4b3a9372bcc3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:03:14.204193 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:03:14.204166    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/462f4d7b-9ada-4c9e-97b2-4b3a9372bcc3-kserve-provision-location\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 22:03:14.204193 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:03:14.204192    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/462f4d7b-9ada-4c9e-97b2-4b3a9372bcc3-tls-certs\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 22:03:14.204355 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:03:14.204202    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/462f4d7b-9ada-4c9e-97b2-4b3a9372bcc3-dshm\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 22:03:14.204355 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:03:14.204210    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mh8bb\" (UniqueName: \"kubernetes.io/projected/462f4d7b-9ada-4c9e-97b2-4b3a9372bcc3-kube-api-access-mh8bb\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 22:03:14.204355 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:03:14.204220    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/462f4d7b-9ada-4c9e-97b2-4b3a9372bcc3-home\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 22:03:14.204355 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:03:14.204227    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/462f4d7b-9ada-4c9e-97b2-4b3a9372bcc3-model-cache\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 22:03:14.204355 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:03:14.204235    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/462f4d7b-9ada-4c9e-97b2-4b3a9372bcc3-tmp-dir\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 22:03:14.615785 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:03:14.615750    2576 generic.go:356] "Generic (PLEG): container finished" podID="462f4d7b-9ada-4c9e-97b2-4b3a9372bcc3" containerID="bf009df330c9e367deac4f6d21d45ef2b9b11b20f1633ba93f12e50f808c8f41" exitCode=0
Sep 09 22:03:14.615785 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:03:14.615777    2576 generic.go:356] "Generic (PLEG): container finished" podID="462f4d7b-9ada-4c9e-97b2-4b3a9372bcc3" containerID="16ee3c8429cfbe17346349a019ef35801ad2207eb2cfa2f1756da4e57cb4df57" exitCode=0
Sep 09 22:03:14.616041 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:03:14.615819    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5566488c4c-qbxf5"
Sep 09 22:03:14.616041 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:03:14.615830    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5566488c4c-qbxf5" event={"ID":"462f4d7b-9ada-4c9e-97b2-4b3a9372bcc3","Type":"ContainerDied","Data":"bf009df330c9e367deac4f6d21d45ef2b9b11b20f1633ba93f12e50f808c8f41"}
Sep 09 22:03:14.616041 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:03:14.615899    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5566488c4c-qbxf5" event={"ID":"462f4d7b-9ada-4c9e-97b2-4b3a9372bcc3","Type":"ContainerDied","Data":"16ee3c8429cfbe17346349a019ef35801ad2207eb2cfa2f1756da4e57cb4df57"}
Sep 09 22:03:14.616041 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:03:14.615924    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5566488c4c-qbxf5" event={"ID":"462f4d7b-9ada-4c9e-97b2-4b3a9372bcc3","Type":"ContainerDied","Data":"22e9755a8b0961edad3c49f44f4c56e0859ddd214c8a738b5c37221328a1de58"}
Sep 09 22:03:14.616041 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:03:14.615952    2576 scope.go:117] "RemoveContainer" containerID="bf009df330c9e367deac4f6d21d45ef2b9b11b20f1633ba93f12e50f808c8f41"
Sep 09 22:03:14.625212 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:03:14.625193    2576 scope.go:117] "RemoveContainer" containerID="3c7d33c31ca34fbe88b4efd84fb911d81715f64413e4df9a46be138075f1de0b"
Sep 09 22:03:14.633637 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:03:14.633619    2576 scope.go:117] "RemoveContainer" containerID="16ee3c8429cfbe17346349a019ef35801ad2207eb2cfa2f1756da4e57cb4df57"
Sep 09 22:03:14.641560 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:03:14.641534    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5566488c4c-qbxf5"]
Sep 09 22:03:14.642187 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:03:14.642165    2576 scope.go:117] "RemoveContainer" containerID="bf009df330c9e367deac4f6d21d45ef2b9b11b20f1633ba93f12e50f808c8f41"
Sep 09 22:03:14.642581 ip-10-0-134-72 kubenswrapper[2576]: E0909 22:03:14.642503    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf009df330c9e367deac4f6d21d45ef2b9b11b20f1633ba93f12e50f808c8f41\": container with ID starting with bf009df330c9e367deac4f6d21d45ef2b9b11b20f1633ba93f12e50f808c8f41 not found: ID does not exist" containerID="bf009df330c9e367deac4f6d21d45ef2b9b11b20f1633ba93f12e50f808c8f41"
Sep 09 22:03:14.642581 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:03:14.642538    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf009df330c9e367deac4f6d21d45ef2b9b11b20f1633ba93f12e50f808c8f41"} err="failed to get container status \"bf009df330c9e367deac4f6d21d45ef2b9b11b20f1633ba93f12e50f808c8f41\": rpc error: code = NotFound desc = could not find container \"bf009df330c9e367deac4f6d21d45ef2b9b11b20f1633ba93f12e50f808c8f41\": container with ID starting with bf009df330c9e367deac4f6d21d45ef2b9b11b20f1633ba93f12e50f808c8f41 not found: ID does not exist"
Sep 09 22:03:14.642581 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:03:14.642561    2576 scope.go:117] "RemoveContainer" containerID="3c7d33c31ca34fbe88b4efd84fb911d81715f64413e4df9a46be138075f1de0b"
Sep 09 22:03:14.643027 ip-10-0-134-72 kubenswrapper[2576]: E0909 22:03:14.643001    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c7d33c31ca34fbe88b4efd84fb911d81715f64413e4df9a46be138075f1de0b\": container with ID starting with 3c7d33c31ca34fbe88b4efd84fb911d81715f64413e4df9a46be138075f1de0b not found: ID does not exist" containerID="3c7d33c31ca34fbe88b4efd84fb911d81715f64413e4df9a46be138075f1de0b"
Sep 09 22:03:14.643129 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:03:14.643036    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c7d33c31ca34fbe88b4efd84fb911d81715f64413e4df9a46be138075f1de0b"} err="failed to get container status \"3c7d33c31ca34fbe88b4efd84fb911d81715f64413e4df9a46be138075f1de0b\": rpc error: code = NotFound desc = could not find container \"3c7d33c31ca34fbe88b4efd84fb911d81715f64413e4df9a46be138075f1de0b\": container with ID starting with 3c7d33c31ca34fbe88b4efd84fb911d81715f64413e4df9a46be138075f1de0b not found: ID does not exist"
Sep 09 22:03:14.643129 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:03:14.643058    2576 scope.go:117] "RemoveContainer" containerID="16ee3c8429cfbe17346349a019ef35801ad2207eb2cfa2f1756da4e57cb4df57"
Sep 09 22:03:14.643443 ip-10-0-134-72 kubenswrapper[2576]: E0909 22:03:14.643346    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16ee3c8429cfbe17346349a019ef35801ad2207eb2cfa2f1756da4e57cb4df57\": container with ID starting with 16ee3c8429cfbe17346349a019ef35801ad2207eb2cfa2f1756da4e57cb4df57 not found: ID does not exist" containerID="16ee3c8429cfbe17346349a019ef35801ad2207eb2cfa2f1756da4e57cb4df57"
Sep 09 22:03:14.643443 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:03:14.643424    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16ee3c8429cfbe17346349a019ef35801ad2207eb2cfa2f1756da4e57cb4df57"} err="failed to get container status \"16ee3c8429cfbe17346349a019ef35801ad2207eb2cfa2f1756da4e57cb4df57\": rpc error: code = NotFound desc = could not find container \"16ee3c8429cfbe17346349a019ef35801ad2207eb2cfa2f1756da4e57cb4df57\": container with ID starting with 16ee3c8429cfbe17346349a019ef35801ad2207eb2cfa2f1756da4e57cb4df57 not found: ID does not exist"
Sep 09 22:03:14.643595 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:03:14.643448    2576 scope.go:117] "RemoveContainer" containerID="bf009df330c9e367deac4f6d21d45ef2b9b11b20f1633ba93f12e50f808c8f41"
Sep 09 22:03:14.643991 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:03:14.643972    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf009df330c9e367deac4f6d21d45ef2b9b11b20f1633ba93f12e50f808c8f41"} err="failed to get container status \"bf009df330c9e367deac4f6d21d45ef2b9b11b20f1633ba93f12e50f808c8f41\": rpc error: code = NotFound desc = could not find container \"bf009df330c9e367deac4f6d21d45ef2b9b11b20f1633ba93f12e50f808c8f41\": container with ID starting with bf009df330c9e367deac4f6d21d45ef2b9b11b20f1633ba93f12e50f808c8f41 not found: ID does not exist"
Sep 09 22:03:14.644054 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:03:14.643992    2576 scope.go:117] "RemoveContainer" containerID="3c7d33c31ca34fbe88b4efd84fb911d81715f64413e4df9a46be138075f1de0b"
Sep 09 22:03:14.644266 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:03:14.644249    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c7d33c31ca34fbe88b4efd84fb911d81715f64413e4df9a46be138075f1de0b"} err="failed to get container status \"3c7d33c31ca34fbe88b4efd84fb911d81715f64413e4df9a46be138075f1de0b\": rpc error: code = NotFound desc = could not find container \"3c7d33c31ca34fbe88b4efd84fb911d81715f64413e4df9a46be138075f1de0b\": container with ID starting with 3c7d33c31ca34fbe88b4efd84fb911d81715f64413e4df9a46be138075f1de0b not found: ID does not exist"
Sep 09 22:03:14.644341 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:03:14.644266    2576 scope.go:117] "RemoveContainer" containerID="16ee3c8429cfbe17346349a019ef35801ad2207eb2cfa2f1756da4e57cb4df57"
Sep 09 22:03:14.644559 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:03:14.644538    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16ee3c8429cfbe17346349a019ef35801ad2207eb2cfa2f1756da4e57cb4df57"} err="failed to get container status \"16ee3c8429cfbe17346349a019ef35801ad2207eb2cfa2f1756da4e57cb4df57\": rpc error: code = NotFound desc = could not find container \"16ee3c8429cfbe17346349a019ef35801ad2207eb2cfa2f1756da4e57cb4df57\": container with ID starting with 16ee3c8429cfbe17346349a019ef35801ad2207eb2cfa2f1756da4e57cb4df57 not found: ID does not exist"
Sep 09 22:03:14.645588 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:03:14.645571    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5566488c4c-qbxf5"]
Sep 09 22:03:15.410579 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:03:15.410536    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="462f4d7b-9ada-4c9e-97b2-4b3a9372bcc3" path="/var/lib/kubelet/pods/462f4d7b-9ada-4c9e-97b2-4b3a9372bcc3/volumes"
Sep 09 22:03:20.685551 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:03:20.685504    2576 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f95b98795-qj5tp" podUID="34982e1c-46a7-4b8f-add2-3ae056a2794f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/health\": dial tcp 10.132.0.88:8000: connect: connection refused"
Sep 09 22:03:30.695867 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:03:30.695829    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f95b98795-qj5tp"
Sep 09 22:03:30.703579 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:03:30.703557    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f95b98795-qj5tp"
Sep 09 22:03:34.057410 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:03:34.057375    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f95b98795-qj5tp"]
Sep 09 22:03:48.081707 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:03:48.081667    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 22:03:48.169548 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:03:48.169516    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 22:03:49.080125 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:03:49.080058    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f95b98795-qj5tp" podUID="34982e1c-46a7-4b8f-add2-3ae056a2794f" containerName="main" containerID="cri-o://f5c3fdddd3e84bf9763de303313bf490506f51a1cbc5d62b72f53bfa6a61e88f" gracePeriod=45
Sep 09 22:03:49.696446 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:03:49.696402    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f95b98795-qj5tp" podUID="34982e1c-46a7-4b8f-add2-3ae056a2794f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/health\": dial tcp 10.132.0.88:8000: connect: connection refused"
Sep 09 22:03:49.811706 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:03:49.811678    2576 generic.go:356] "Generic (PLEG): container finished" podID="34982e1c-46a7-4b8f-add2-3ae056a2794f" containerID="f5c3fdddd3e84bf9763de303313bf490506f51a1cbc5d62b72f53bfa6a61e88f" exitCode=0
Sep 09 22:03:49.811825 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:03:49.811738    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f95b98795-qj5tp" event={"ID":"34982e1c-46a7-4b8f-add2-3ae056a2794f","Type":"ContainerDied","Data":"f5c3fdddd3e84bf9763de303313bf490506f51a1cbc5d62b72f53bfa6a61e88f"}
Sep 09 22:03:49.825416 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:03:49.825390    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f95b98795-qj5tp"
Sep 09 22:03:50.015159 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:03:50.015074    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34982e1c-46a7-4b8f-add2-3ae056a2794f-model-cache\") pod \"34982e1c-46a7-4b8f-add2-3ae056a2794f\" (UID: \"34982e1c-46a7-4b8f-add2-3ae056a2794f\") "
Sep 09 22:03:50.015159 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:03:50.015147    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34982e1c-46a7-4b8f-add2-3ae056a2794f-home\") pod \"34982e1c-46a7-4b8f-add2-3ae056a2794f\" (UID: \"34982e1c-46a7-4b8f-add2-3ae056a2794f\") "
Sep 09 22:03:50.015414 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:03:50.015172    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34982e1c-46a7-4b8f-add2-3ae056a2794f-tls-certs\") pod \"34982e1c-46a7-4b8f-add2-3ae056a2794f\" (UID: \"34982e1c-46a7-4b8f-add2-3ae056a2794f\") "
Sep 09 22:03:50.015414 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:03:50.015199    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/34982e1c-46a7-4b8f-add2-3ae056a2794f-kserve-provision-location\") pod \"34982e1c-46a7-4b8f-add2-3ae056a2794f\" (UID: \"34982e1c-46a7-4b8f-add2-3ae056a2794f\") "
Sep 09 22:03:50.015414 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:03:50.015218    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-27fpv\" (UniqueName: \"kubernetes.io/projected/34982e1c-46a7-4b8f-add2-3ae056a2794f-kube-api-access-27fpv\") pod \"34982e1c-46a7-4b8f-add2-3ae056a2794f\" (UID: \"34982e1c-46a7-4b8f-add2-3ae056a2794f\") "
Sep 09 22:03:50.015414 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:03:50.015256    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34982e1c-46a7-4b8f-add2-3ae056a2794f-dshm\") pod \"34982e1c-46a7-4b8f-add2-3ae056a2794f\" (UID: \"34982e1c-46a7-4b8f-add2-3ae056a2794f\") "
Sep 09 22:03:50.015414 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:03:50.015293    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34982e1c-46a7-4b8f-add2-3ae056a2794f-tmp-dir\") pod \"34982e1c-46a7-4b8f-add2-3ae056a2794f\" (UID: \"34982e1c-46a7-4b8f-add2-3ae056a2794f\") "
Sep 09 22:03:50.015414 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:03:50.015315    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/34982e1c-46a7-4b8f-add2-3ae056a2794f-cabundle-cert\") pod \"34982e1c-46a7-4b8f-add2-3ae056a2794f\" (UID: \"34982e1c-46a7-4b8f-add2-3ae056a2794f\") "
Sep 09 22:03:50.015752 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:03:50.015412    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34982e1c-46a7-4b8f-add2-3ae056a2794f-model-cache" (OuterVolumeSpecName: "model-cache") pod "34982e1c-46a7-4b8f-add2-3ae056a2794f" (UID: "34982e1c-46a7-4b8f-add2-3ae056a2794f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:03:50.015827 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:03:50.015808    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34982e1c-46a7-4b8f-add2-3ae056a2794f-model-cache\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 22:03:50.015947 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:03:50.015919    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34982e1c-46a7-4b8f-add2-3ae056a2794f-home" (OuterVolumeSpecName: "home") pod "34982e1c-46a7-4b8f-add2-3ae056a2794f" (UID: "34982e1c-46a7-4b8f-add2-3ae056a2794f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:03:50.016005 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:03:50.015920    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34982e1c-46a7-4b8f-add2-3ae056a2794f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "34982e1c-46a7-4b8f-add2-3ae056a2794f" (UID: "34982e1c-46a7-4b8f-add2-3ae056a2794f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 22:03:50.017941 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:03:50.017912    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34982e1c-46a7-4b8f-add2-3ae056a2794f-kube-api-access-27fpv" (OuterVolumeSpecName: "kube-api-access-27fpv") pod "34982e1c-46a7-4b8f-add2-3ae056a2794f" (UID: "34982e1c-46a7-4b8f-add2-3ae056a2794f"). InnerVolumeSpecName "kube-api-access-27fpv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 22:03:50.018113 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:03:50.017943    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34982e1c-46a7-4b8f-add2-3ae056a2794f-dshm" (OuterVolumeSpecName: "dshm") pod "34982e1c-46a7-4b8f-add2-3ae056a2794f" (UID: "34982e1c-46a7-4b8f-add2-3ae056a2794f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:03:50.018113 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:03:50.017948    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34982e1c-46a7-4b8f-add2-3ae056a2794f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "34982e1c-46a7-4b8f-add2-3ae056a2794f" (UID: "34982e1c-46a7-4b8f-add2-3ae056a2794f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 22:03:50.029790 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:03:50.029765    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34982e1c-46a7-4b8f-add2-3ae056a2794f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "34982e1c-46a7-4b8f-add2-3ae056a2794f" (UID: "34982e1c-46a7-4b8f-add2-3ae056a2794f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:03:50.038433 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:03:50.038410    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34982e1c-46a7-4b8f-add2-3ae056a2794f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "34982e1c-46a7-4b8f-add2-3ae056a2794f" (UID: "34982e1c-46a7-4b8f-add2-3ae056a2794f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:03:50.117283 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:03:50.117248    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34982e1c-46a7-4b8f-add2-3ae056a2794f-home\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 22:03:50.117283 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:03:50.117281    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34982e1c-46a7-4b8f-add2-3ae056a2794f-tls-certs\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 22:03:50.117514 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:03:50.117292    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/34982e1c-46a7-4b8f-add2-3ae056a2794f-kserve-provision-location\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 22:03:50.117514 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:03:50.117301    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27fpv\" (UniqueName: \"kubernetes.io/projected/34982e1c-46a7-4b8f-add2-3ae056a2794f-kube-api-access-27fpv\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 22:03:50.117514 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:03:50.117312    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34982e1c-46a7-4b8f-add2-3ae056a2794f-dshm\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 22:03:50.117514 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:03:50.117320    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34982e1c-46a7-4b8f-add2-3ae056a2794f-tmp-dir\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 22:03:50.117514 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:03:50.117327    2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/34982e1c-46a7-4b8f-add2-3ae056a2794f-cabundle-cert\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 22:03:50.818984 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:03:50.818947    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f95b98795-qj5tp" event={"ID":"34982e1c-46a7-4b8f-add2-3ae056a2794f","Type":"ContainerDied","Data":"7738d591303234e66ca31ece7d95a97900e65e834e0b24ba371f8311fad6abac"}
Sep 09 22:03:50.819430 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:03:50.819013    2576 scope.go:117] "RemoveContainer" containerID="f5c3fdddd3e84bf9763de303313bf490506f51a1cbc5d62b72f53bfa6a61e88f"
Sep 09 22:03:50.819430 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:03:50.818963    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f95b98795-qj5tp"
Sep 09 22:03:50.828698 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:03:50.828680    2576 scope.go:117] "RemoveContainer" containerID="b68fbe64cf35226496415268604c277719656cd45d51ce7531042b5736e551c3"
Sep 09 22:03:50.842962 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:03:50.842937    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f95b98795-qj5tp"]
Sep 09 22:03:50.847679 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:03:50.847654    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7f95b98795-qj5tp"]
Sep 09 22:03:51.409570 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:03:51.409534    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="34982e1c-46a7-4b8f-add2-3ae056a2794f" path="/var/lib/kubelet/pods/34982e1c-46a7-4b8f-add2-3ae056a2794f/volumes"
Sep 09 22:03:53.040861 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:03:53.040814    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-qm75c"]
Sep 09 22:03:53.041328 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:03:53.041113    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-qm75c" podUID="71a5281b-b9f1-4ea5-a489-8f7dc9089465" containerName="istio-proxy" containerID="cri-o://97521b6d7255a9f825371908bb6a93bf87cb9620ef30ea57480733d867c06379" gracePeriod=30
Sep 09 22:03:53.142824 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:03:53.142781    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 22:03:58.293152 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:03:58.293124    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-qm75c"
Sep 09 22:03:58.377207 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:03:58.377176    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71a5281b-b9f1-4ea5-a489-8f7dc9089465-istio-data\") pod \"71a5281b-b9f1-4ea5-a489-8f7dc9089465\" (UID: \"71a5281b-b9f1-4ea5-a489-8f7dc9089465\") "
Sep 09 22:03:58.377337 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:03:58.377225    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71a5281b-b9f1-4ea5-a489-8f7dc9089465-workload-socket\") pod \"71a5281b-b9f1-4ea5-a489-8f7dc9089465\" (UID: \"71a5281b-b9f1-4ea5-a489-8f7dc9089465\") "
Sep 09 22:03:58.377337 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:03:58.377280    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pzktl\" (UniqueName: \"kubernetes.io/projected/71a5281b-b9f1-4ea5-a489-8f7dc9089465-kube-api-access-pzktl\") pod \"71a5281b-b9f1-4ea5-a489-8f7dc9089465\" (UID: \"71a5281b-b9f1-4ea5-a489-8f7dc9089465\") "
Sep 09 22:03:58.377337 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:03:58.377305    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71a5281b-b9f1-4ea5-a489-8f7dc9089465-istio-token\") pod \"71a5281b-b9f1-4ea5-a489-8f7dc9089465\" (UID: \"71a5281b-b9f1-4ea5-a489-8f7dc9089465\") "
Sep 09 22:03:58.377532 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:03:58.377339    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71a5281b-b9f1-4ea5-a489-8f7dc9089465-credential-socket\") pod \"71a5281b-b9f1-4ea5-a489-8f7dc9089465\" (UID: \"71a5281b-b9f1-4ea5-a489-8f7dc9089465\") "
Sep 09 22:03:58.377532 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:03:58.377430    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71a5281b-b9f1-4ea5-a489-8f7dc9089465-workload-certs\") pod \"71a5281b-b9f1-4ea5-a489-8f7dc9089465\" (UID: \"71a5281b-b9f1-4ea5-a489-8f7dc9089465\") "
Sep 09 22:03:58.377532 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:03:58.377468    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71a5281b-b9f1-4ea5-a489-8f7dc9089465-istiod-ca-cert\") pod \"71a5281b-b9f1-4ea5-a489-8f7dc9089465\" (UID: \"71a5281b-b9f1-4ea5-a489-8f7dc9089465\") "
Sep 09 22:03:58.377532 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:03:58.377511    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71a5281b-b9f1-4ea5-a489-8f7dc9089465-istio-envoy\") pod \"71a5281b-b9f1-4ea5-a489-8f7dc9089465\" (UID: \"71a5281b-b9f1-4ea5-a489-8f7dc9089465\") "
Sep 09 22:03:58.377736 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:03:58.377561    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71a5281b-b9f1-4ea5-a489-8f7dc9089465-istio-podinfo\") pod \"71a5281b-b9f1-4ea5-a489-8f7dc9089465\" (UID: \"71a5281b-b9f1-4ea5-a489-8f7dc9089465\") "
Sep 09 22:03:58.377736 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:03:58.377568    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71a5281b-b9f1-4ea5-a489-8f7dc9089465-istio-data" (OuterVolumeSpecName: "istio-data") pod "71a5281b-b9f1-4ea5-a489-8f7dc9089465" (UID: "71a5281b-b9f1-4ea5-a489-8f7dc9089465"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:03:58.377736 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:03:58.377581    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71a5281b-b9f1-4ea5-a489-8f7dc9089465-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "71a5281b-b9f1-4ea5-a489-8f7dc9089465" (UID: "71a5281b-b9f1-4ea5-a489-8f7dc9089465"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:03:58.377916 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:03:58.377769    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71a5281b-b9f1-4ea5-a489-8f7dc9089465-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "71a5281b-b9f1-4ea5-a489-8f7dc9089465" (UID: "71a5281b-b9f1-4ea5-a489-8f7dc9089465"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:03:58.377916 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:03:58.377796    2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71a5281b-b9f1-4ea5-a489-8f7dc9089465-istio-data\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 22:03:58.377916 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:03:58.377814    2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71a5281b-b9f1-4ea5-a489-8f7dc9089465-workload-socket\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 22:03:58.377916 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:03:58.377893    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71a5281b-b9f1-4ea5-a489-8f7dc9089465-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "71a5281b-b9f1-4ea5-a489-8f7dc9089465" (UID: "71a5281b-b9f1-4ea5-a489-8f7dc9089465"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 22:03:58.378178 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:03:58.377972    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71a5281b-b9f1-4ea5-a489-8f7dc9089465-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "71a5281b-b9f1-4ea5-a489-8f7dc9089465" (UID: "71a5281b-b9f1-4ea5-a489-8f7dc9089465"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:03:58.379792 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:03:58.379759    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71a5281b-b9f1-4ea5-a489-8f7dc9089465-kube-api-access-pzktl" (OuterVolumeSpecName: "kube-api-access-pzktl") pod "71a5281b-b9f1-4ea5-a489-8f7dc9089465" (UID: "71a5281b-b9f1-4ea5-a489-8f7dc9089465"). InnerVolumeSpecName "kube-api-access-pzktl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 22:03:58.379958 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:03:58.379935    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/71a5281b-b9f1-4ea5-a489-8f7dc9089465-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "71a5281b-b9f1-4ea5-a489-8f7dc9089465" (UID: "71a5281b-b9f1-4ea5-a489-8f7dc9089465"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 22:03:58.380160 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:03:58.380135    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71a5281b-b9f1-4ea5-a489-8f7dc9089465-istio-token" (OuterVolumeSpecName: "istio-token") pod "71a5281b-b9f1-4ea5-a489-8f7dc9089465" (UID: "71a5281b-b9f1-4ea5-a489-8f7dc9089465"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 22:03:58.380261 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:03:58.380153    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71a5281b-b9f1-4ea5-a489-8f7dc9089465-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "71a5281b-b9f1-4ea5-a489-8f7dc9089465" (UID: "71a5281b-b9f1-4ea5-a489-8f7dc9089465"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:03:58.478791 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:03:58.478763    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzktl\" (UniqueName: \"kubernetes.io/projected/71a5281b-b9f1-4ea5-a489-8f7dc9089465-kube-api-access-pzktl\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 22:03:58.478791 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:03:58.478785    2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71a5281b-b9f1-4ea5-a489-8f7dc9089465-istio-token\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 22:03:58.478791 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:03:58.478795    2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71a5281b-b9f1-4ea5-a489-8f7dc9089465-credential-socket\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 22:03:58.478999 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:03:58.478804    2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71a5281b-b9f1-4ea5-a489-8f7dc9089465-workload-certs\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 22:03:58.478999 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:03:58.478813    2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71a5281b-b9f1-4ea5-a489-8f7dc9089465-istiod-ca-cert\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 22:03:58.478999 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:03:58.478820    2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71a5281b-b9f1-4ea5-a489-8f7dc9089465-istio-envoy\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 22:03:58.478999 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:03:58.478828    2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71a5281b-b9f1-4ea5-a489-8f7dc9089465-istio-podinfo\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 22:03:58.863262 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:03:58.863230    2576 generic.go:356] "Generic (PLEG): container finished" podID="71a5281b-b9f1-4ea5-a489-8f7dc9089465" containerID="97521b6d7255a9f825371908bb6a93bf87cb9620ef30ea57480733d867c06379" exitCode=0
Sep 09 22:03:58.863444 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:03:58.863280    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-qm75c" event={"ID":"71a5281b-b9f1-4ea5-a489-8f7dc9089465","Type":"ContainerDied","Data":"97521b6d7255a9f825371908bb6a93bf87cb9620ef30ea57480733d867c06379"}
Sep 09 22:03:58.863444 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:03:58.863321    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-qm75c" event={"ID":"71a5281b-b9f1-4ea5-a489-8f7dc9089465","Type":"ContainerDied","Data":"81892fd4f4a38187c8b5d627878a494c71f81bf07d2cd2886619f95ae4d20d85"}
Sep 09 22:03:58.863444 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:03:58.863341    2576 scope.go:117] "RemoveContainer" containerID="97521b6d7255a9f825371908bb6a93bf87cb9620ef30ea57480733d867c06379"
Sep 09 22:03:58.863444 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:03:58.863282    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-qm75c"
Sep 09 22:03:58.873302 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:03:58.873214    2576 scope.go:117] "RemoveContainer" containerID="fd524abeae7a414bcb33dfea6a45c103fd5836cbb72ab71426ba2193eb84f110"
Sep 09 22:03:58.882015 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:03:58.881993    2576 scope.go:117] "RemoveContainer" containerID="97521b6d7255a9f825371908bb6a93bf87cb9620ef30ea57480733d867c06379"
Sep 09 22:03:58.882289 ip-10-0-134-72 kubenswrapper[2576]: E0909 22:03:58.882273    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97521b6d7255a9f825371908bb6a93bf87cb9620ef30ea57480733d867c06379\": container with ID starting with 97521b6d7255a9f825371908bb6a93bf87cb9620ef30ea57480733d867c06379 not found: ID does not exist" containerID="97521b6d7255a9f825371908bb6a93bf87cb9620ef30ea57480733d867c06379"
Sep 09 22:03:58.882347 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:03:58.882296    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97521b6d7255a9f825371908bb6a93bf87cb9620ef30ea57480733d867c06379"} err="failed to get container status \"97521b6d7255a9f825371908bb6a93bf87cb9620ef30ea57480733d867c06379\": rpc error: code = NotFound desc = could not find container \"97521b6d7255a9f825371908bb6a93bf87cb9620ef30ea57480733d867c06379\": container with ID starting with 97521b6d7255a9f825371908bb6a93bf87cb9620ef30ea57480733d867c06379 not found: ID does not exist"
Sep 09 22:03:58.882347 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:03:58.882314    2576 scope.go:117] "RemoveContainer" containerID="fd524abeae7a414bcb33dfea6a45c103fd5836cbb72ab71426ba2193eb84f110"
Sep 09 22:03:58.882576 ip-10-0-134-72 kubenswrapper[2576]: E0909 22:03:58.882558    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd524abeae7a414bcb33dfea6a45c103fd5836cbb72ab71426ba2193eb84f110\": container with ID starting with fd524abeae7a414bcb33dfea6a45c103fd5836cbb72ab71426ba2193eb84f110 not found: ID does not exist" containerID="fd524abeae7a414bcb33dfea6a45c103fd5836cbb72ab71426ba2193eb84f110"
Sep 09 22:03:58.882634 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:03:58.882586    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd524abeae7a414bcb33dfea6a45c103fd5836cbb72ab71426ba2193eb84f110"} err="failed to get container status \"fd524abeae7a414bcb33dfea6a45c103fd5836cbb72ab71426ba2193eb84f110\": rpc error: code = NotFound desc = could not find container \"fd524abeae7a414bcb33dfea6a45c103fd5836cbb72ab71426ba2193eb84f110\": container with ID starting with fd524abeae7a414bcb33dfea6a45c103fd5836cbb72ab71426ba2193eb84f110 not found: ID does not exist"
Sep 09 22:03:58.886340 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:03:58.886317    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-qm75c"]
Sep 09 22:03:58.890874 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:03:58.890855    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-qm75c"]
Sep 09 22:03:59.411656 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:03:59.411610    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="71a5281b-b9f1-4ea5-a489-8f7dc9089465" path="/var/lib/kubelet/pods/71a5281b-b9f1-4ea5-a489-8f7dc9089465/volumes"
Sep 09 22:04:15.761695 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:04:15.761652    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qqbc4"]
Sep 09 22:04:15.762278 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:04:15.762165    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="462f4d7b-9ada-4c9e-97b2-4b3a9372bcc3" containerName="main"
Sep 09 22:04:15.762278 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:04:15.762185    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="462f4d7b-9ada-4c9e-97b2-4b3a9372bcc3" containerName="main"
Sep 09 22:04:15.762278 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:04:15.762202    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="462f4d7b-9ada-4c9e-97b2-4b3a9372bcc3" containerName="llm-d-routing-sidecar"
Sep 09 22:04:15.762278 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:04:15.762212    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="462f4d7b-9ada-4c9e-97b2-4b3a9372bcc3" containerName="llm-d-routing-sidecar"
Sep 09 22:04:15.762278 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:04:15.762231    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34982e1c-46a7-4b8f-add2-3ae056a2794f" containerName="storage-initializer"
Sep 09 22:04:15.762278 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:04:15.762241    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="34982e1c-46a7-4b8f-add2-3ae056a2794f" containerName="storage-initializer"
Sep 09 22:04:15.762278 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:04:15.762252    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="462f4d7b-9ada-4c9e-97b2-4b3a9372bcc3" containerName="storage-initializer"
Sep 09 22:04:15.762278 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:04:15.762260    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="462f4d7b-9ada-4c9e-97b2-4b3a9372bcc3" containerName="storage-initializer"
Sep 09 22:04:15.762278 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:04:15.762276    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34982e1c-46a7-4b8f-add2-3ae056a2794f" containerName="main"
Sep 09 22:04:15.762770 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:04:15.762284    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="34982e1c-46a7-4b8f-add2-3ae056a2794f" containerName="main"
Sep 09 22:04:15.762770 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:04:15.762296    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71a5281b-b9f1-4ea5-a489-8f7dc9089465" containerName="istio-proxy"
Sep 09 22:04:15.762770 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:04:15.762303    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="71a5281b-b9f1-4ea5-a489-8f7dc9089465" containerName="istio-proxy"
Sep 09 22:04:15.762770 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:04:15.762311    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71a5281b-b9f1-4ea5-a489-8f7dc9089465" containerName="istio-proxy"
Sep 09 22:04:15.762770 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:04:15.762320    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="71a5281b-b9f1-4ea5-a489-8f7dc9089465" containerName="istio-proxy"
Sep 09 22:04:15.762770 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:04:15.762428    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="462f4d7b-9ada-4c9e-97b2-4b3a9372bcc3" containerName="llm-d-routing-sidecar"
Sep 09 22:04:15.762770 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:04:15.762440    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="34982e1c-46a7-4b8f-add2-3ae056a2794f" containerName="main"
Sep 09 22:04:15.762770 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:04:15.762451    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="462f4d7b-9ada-4c9e-97b2-4b3a9372bcc3" containerName="main"
Sep 09 22:04:15.762770 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:04:15.762459    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="71a5281b-b9f1-4ea5-a489-8f7dc9089465" containerName="istio-proxy"
Sep 09 22:04:15.762770 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:04:15.762466    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="71a5281b-b9f1-4ea5-a489-8f7dc9089465" containerName="istio-proxy"
Sep 09 22:04:15.765300 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:04:15.765279    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qqbc4"
Sep 09 22:04:15.767851 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:04:15.767830    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"openshift-service-ca.crt\""
Sep 09 22:04:15.767976 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:04:15.767915    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"kube-root-ca.crt\""
Sep 09 22:04:15.767976 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:04:15.767959    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"default-dockercfg-nh5dm\""
Sep 09 22:04:15.767976 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:04:15.767968    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"rolling-upgrade-test-kserve-self-signed-certs\""
Sep 09 22:04:15.778053 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:04:15.778033    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qqbc4"]
Sep 09 22:04:15.817302 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:04:15.817274    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5cd9703-6ff3-417c-b8f6-f453fc3916c6-tmp-dir\") pod \"rolling-upgrade-test-kserve-74b79d7f77-qqbc4\" (UID: \"e5cd9703-6ff3-417c-b8f6-f453fc3916c6\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qqbc4"
Sep 09 22:04:15.817439 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:04:15.817310    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5cd9703-6ff3-417c-b8f6-f453fc3916c6-tls-certs\") pod \"rolling-upgrade-test-kserve-74b79d7f77-qqbc4\" (UID: \"e5cd9703-6ff3-417c-b8f6-f453fc3916c6\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qqbc4"
Sep 09 22:04:15.817439 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:04:15.817341    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99k47\" (UniqueName: \"kubernetes.io/projected/e5cd9703-6ff3-417c-b8f6-f453fc3916c6-kube-api-access-99k47\") pod \"rolling-upgrade-test-kserve-74b79d7f77-qqbc4\" (UID: \"e5cd9703-6ff3-417c-b8f6-f453fc3916c6\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qqbc4"
Sep 09 22:04:15.817439 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:04:15.817417    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5cd9703-6ff3-417c-b8f6-f453fc3916c6-model-cache\") pod \"rolling-upgrade-test-kserve-74b79d7f77-qqbc4\" (UID: \"e5cd9703-6ff3-417c-b8f6-f453fc3916c6\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qqbc4"
Sep 09 22:04:15.817600 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:04:15.817446    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5cd9703-6ff3-417c-b8f6-f453fc3916c6-home\") pod \"rolling-upgrade-test-kserve-74b79d7f77-qqbc4\" (UID: \"e5cd9703-6ff3-417c-b8f6-f453fc3916c6\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qqbc4"
Sep 09 22:04:15.817600 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:04:15.817487    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5cd9703-6ff3-417c-b8f6-f453fc3916c6-dshm\") pod \"rolling-upgrade-test-kserve-74b79d7f77-qqbc4\" (UID: \"e5cd9703-6ff3-417c-b8f6-f453fc3916c6\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qqbc4"
Sep 09 22:04:15.918616 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:04:15.918590    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5cd9703-6ff3-417c-b8f6-f453fc3916c6-model-cache\") pod \"rolling-upgrade-test-kserve-74b79d7f77-qqbc4\" (UID: \"e5cd9703-6ff3-417c-b8f6-f453fc3916c6\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qqbc4"
Sep 09 22:04:15.918731 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:04:15.918628    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5cd9703-6ff3-417c-b8f6-f453fc3916c6-home\") pod \"rolling-upgrade-test-kserve-74b79d7f77-qqbc4\" (UID: \"e5cd9703-6ff3-417c-b8f6-f453fc3916c6\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qqbc4"
Sep 09 22:04:15.918731 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:04:15.918646    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5cd9703-6ff3-417c-b8f6-f453fc3916c6-dshm\") pod \"rolling-upgrade-test-kserve-74b79d7f77-qqbc4\" (UID: \"e5cd9703-6ff3-417c-b8f6-f453fc3916c6\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qqbc4"
Sep 09 22:04:15.918846 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:04:15.918825    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5cd9703-6ff3-417c-b8f6-f453fc3916c6-tmp-dir\") pod \"rolling-upgrade-test-kserve-74b79d7f77-qqbc4\" (UID: \"e5cd9703-6ff3-417c-b8f6-f453fc3916c6\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qqbc4"
Sep 09 22:04:15.918884 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:04:15.918869    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5cd9703-6ff3-417c-b8f6-f453fc3916c6-tls-certs\") pod \"rolling-upgrade-test-kserve-74b79d7f77-qqbc4\" (UID: \"e5cd9703-6ff3-417c-b8f6-f453fc3916c6\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qqbc4"
Sep 09 22:04:15.918990 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:04:15.918971    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99k47\" (UniqueName: \"kubernetes.io/projected/e5cd9703-6ff3-417c-b8f6-f453fc3916c6-kube-api-access-99k47\") pod \"rolling-upgrade-test-kserve-74b79d7f77-qqbc4\" (UID: \"e5cd9703-6ff3-417c-b8f6-f453fc3916c6\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qqbc4"
Sep 09 22:04:15.919091 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:04:15.919069    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5cd9703-6ff3-417c-b8f6-f453fc3916c6-home\") pod \"rolling-upgrade-test-kserve-74b79d7f77-qqbc4\" (UID: \"e5cd9703-6ff3-417c-b8f6-f453fc3916c6\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qqbc4"
Sep 09 22:04:15.919154 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:04:15.919087    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5cd9703-6ff3-417c-b8f6-f453fc3916c6-model-cache\") pod \"rolling-upgrade-test-kserve-74b79d7f77-qqbc4\" (UID: \"e5cd9703-6ff3-417c-b8f6-f453fc3916c6\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qqbc4"
Sep 09 22:04:15.919154 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:04:15.919114    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5cd9703-6ff3-417c-b8f6-f453fc3916c6-tmp-dir\") pod \"rolling-upgrade-test-kserve-74b79d7f77-qqbc4\" (UID: \"e5cd9703-6ff3-417c-b8f6-f453fc3916c6\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qqbc4"
Sep 09 22:04:15.921128 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:04:15.921103    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5cd9703-6ff3-417c-b8f6-f453fc3916c6-dshm\") pod \"rolling-upgrade-test-kserve-74b79d7f77-qqbc4\" (UID: \"e5cd9703-6ff3-417c-b8f6-f453fc3916c6\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qqbc4"
Sep 09 22:04:15.921307 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:04:15.921289    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5cd9703-6ff3-417c-b8f6-f453fc3916c6-tls-certs\") pod \"rolling-upgrade-test-kserve-74b79d7f77-qqbc4\" (UID: \"e5cd9703-6ff3-417c-b8f6-f453fc3916c6\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qqbc4"
Sep 09 22:04:15.929057 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:04:15.929027    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-99k47\" (UniqueName: \"kubernetes.io/projected/e5cd9703-6ff3-417c-b8f6-f453fc3916c6-kube-api-access-99k47\") pod \"rolling-upgrade-test-kserve-74b79d7f77-qqbc4\" (UID: \"e5cd9703-6ff3-417c-b8f6-f453fc3916c6\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qqbc4"
Sep 09 22:04:15.969432 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:04:15.969399    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 22:04:16.075755 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:04:16.075671    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qqbc4"
Sep 09 22:04:16.348582 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:04:16.348554    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qqbc4"]
Sep 09 22:04:16.350668 ip-10-0-134-72 kubenswrapper[2576]: W0909 22:04:16.350642    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5cd9703_6ff3_417c_b8f6_f453fc3916c6.slice/crio-e9a1e1018b4951f410703d574486a7c70d9909f800ed7fd9f6ddd667b15d9849 WatchSource:0}: Error finding container e9a1e1018b4951f410703d574486a7c70d9909f800ed7fd9f6ddd667b15d9849: Status 404 returned error can't find the container with id e9a1e1018b4951f410703d574486a7c70d9909f800ed7fd9f6ddd667b15d9849
Sep 09 22:04:16.368289 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:04:16.368262    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 22:04:16.753073 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:04:16.753037    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 22:04:16.959409 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:04:16.959350    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qqbc4" event={"ID":"e5cd9703-6ff3-417c-b8f6-f453fc3916c6","Type":"ContainerStarted","Data":"d1b18f401006214cdfb02335280f21cea5f9eda8f86ad15dfdc6172fcf65ffd8"}
Sep 09 22:04:16.959779 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:04:16.959423    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qqbc4" event={"ID":"e5cd9703-6ff3-417c-b8f6-f453fc3916c6","Type":"ContainerStarted","Data":"e9a1e1018b4951f410703d574486a7c70d9909f800ed7fd9f6ddd667b15d9849"}
Sep 09 22:04:16.979437 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:04:16.979324    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qqbc4" podStartSLOduration=1.979309706 podStartE2EDuration="1.979309706s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 22:04:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 22:04:16.977549879 +0000 UTC m=+4300.091235738" watchObservedRunningTime="2026-09-09 22:04:16.979309706 +0000 UTC m=+4300.092995607"
Sep 09 22:04:17.076148 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:04:17.076113    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qqbc4"
Sep 09 22:04:17.083565 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:04:17.083536    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 22:04:21.986844 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:04:21.986799    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 22:04:22.081692 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:04:22.081658    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 22:04:23.163675 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:04:23.163640    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 22:04:25.423712 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:04:25.423674    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 22:04:26.076265 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:04:26.076231    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qqbc4"
Sep 09 22:04:26.088879 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:04:26.088845    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qqbc4"
Sep 09 22:04:27.024116 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:04:27.024085    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qqbc4"
Sep 09 22:04:32.739180 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:04:32.739134    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 22:04:38.076813 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:04:38.076772    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 22:04:38.160135 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:04:38.160092    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 22:04:51.346679 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:04:51.346648    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-zzfcp"]
Sep 09 22:04:51.351178 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:04:51.351159    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-zzfcp"
Sep 09 22:04:51.365781 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:04:51.365757    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-zzfcp"]
Sep 09 22:04:51.424820 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:04:51.424795    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc40079b-c0f0-4119-94b9-fbe3e3b7cb13-home\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-zzfcp\" (UID: \"cc40079b-c0f0-4119-94b9-fbe3e3b7cb13\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-zzfcp"
Sep 09 22:04:51.424978 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:04:51.424840    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc40079b-c0f0-4119-94b9-fbe3e3b7cb13-dshm\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-zzfcp\" (UID: \"cc40079b-c0f0-4119-94b9-fbe3e3b7cb13\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-zzfcp"
Sep 09 22:04:51.424978 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:04:51.424864    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc40079b-c0f0-4119-94b9-fbe3e3b7cb13-model-cache\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-zzfcp\" (UID: \"cc40079b-c0f0-4119-94b9-fbe3e3b7cb13\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-zzfcp"
Sep 09 22:04:51.424978 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:04:51.424888    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcnkw\" (UniqueName: \"kubernetes.io/projected/cc40079b-c0f0-4119-94b9-fbe3e3b7cb13-kube-api-access-kcnkw\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-zzfcp\" (UID: \"cc40079b-c0f0-4119-94b9-fbe3e3b7cb13\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-zzfcp"
Sep 09 22:04:51.424978 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:04:51.424915    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc40079b-c0f0-4119-94b9-fbe3e3b7cb13-tmp-dir\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-zzfcp\" (UID: \"cc40079b-c0f0-4119-94b9-fbe3e3b7cb13\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-zzfcp"
Sep 09 22:04:51.424978 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:04:51.424948    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc40079b-c0f0-4119-94b9-fbe3e3b7cb13-tls-certs\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-zzfcp\" (UID: \"cc40079b-c0f0-4119-94b9-fbe3e3b7cb13\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-zzfcp"
Sep 09 22:04:51.526062 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:04:51.526033    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc40079b-c0f0-4119-94b9-fbe3e3b7cb13-home\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-zzfcp\" (UID: \"cc40079b-c0f0-4119-94b9-fbe3e3b7cb13\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-zzfcp"
Sep 09 22:04:51.526178 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:04:51.526074    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc40079b-c0f0-4119-94b9-fbe3e3b7cb13-dshm\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-zzfcp\" (UID: \"cc40079b-c0f0-4119-94b9-fbe3e3b7cb13\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-zzfcp"
Sep 09 22:04:51.526178 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:04:51.526098    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc40079b-c0f0-4119-94b9-fbe3e3b7cb13-model-cache\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-zzfcp\" (UID: \"cc40079b-c0f0-4119-94b9-fbe3e3b7cb13\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-zzfcp"
Sep 09 22:04:51.526298 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:04:51.526218    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcnkw\" (UniqueName: \"kubernetes.io/projected/cc40079b-c0f0-4119-94b9-fbe3e3b7cb13-kube-api-access-kcnkw\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-zzfcp\" (UID: \"cc40079b-c0f0-4119-94b9-fbe3e3b7cb13\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-zzfcp"
Sep 09 22:04:51.526298 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:04:51.526264    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc40079b-c0f0-4119-94b9-fbe3e3b7cb13-tmp-dir\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-zzfcp\" (UID: \"cc40079b-c0f0-4119-94b9-fbe3e3b7cb13\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-zzfcp"
Sep 09 22:04:51.526439 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:04:51.526308    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc40079b-c0f0-4119-94b9-fbe3e3b7cb13-tls-certs\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-zzfcp\" (UID: \"cc40079b-c0f0-4119-94b9-fbe3e3b7cb13\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-zzfcp"
Sep 09 22:04:51.526503 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:04:51.526487    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc40079b-c0f0-4119-94b9-fbe3e3b7cb13-home\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-zzfcp\" (UID: \"cc40079b-c0f0-4119-94b9-fbe3e3b7cb13\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-zzfcp"
Sep 09 22:04:51.526560 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:04:51.526522    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc40079b-c0f0-4119-94b9-fbe3e3b7cb13-model-cache\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-zzfcp\" (UID: \"cc40079b-c0f0-4119-94b9-fbe3e3b7cb13\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-zzfcp"
Sep 09 22:04:51.526658 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:04:51.526637    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc40079b-c0f0-4119-94b9-fbe3e3b7cb13-tmp-dir\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-zzfcp\" (UID: \"cc40079b-c0f0-4119-94b9-fbe3e3b7cb13\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-zzfcp"
Sep 09 22:04:51.528519 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:04:51.528500    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc40079b-c0f0-4119-94b9-fbe3e3b7cb13-dshm\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-zzfcp\" (UID: \"cc40079b-c0f0-4119-94b9-fbe3e3b7cb13\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-zzfcp"
Sep 09 22:04:51.528831 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:04:51.528812    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc40079b-c0f0-4119-94b9-fbe3e3b7cb13-tls-certs\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-zzfcp\" (UID: \"cc40079b-c0f0-4119-94b9-fbe3e3b7cb13\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-zzfcp"
Sep 09 22:04:51.535101 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:04:51.535081    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcnkw\" (UniqueName: \"kubernetes.io/projected/cc40079b-c0f0-4119-94b9-fbe3e3b7cb13-kube-api-access-kcnkw\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-zzfcp\" (UID: \"cc40079b-c0f0-4119-94b9-fbe3e3b7cb13\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-zzfcp"
Sep 09 22:04:51.661180 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:04:51.661088    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-zzfcp"
Sep 09 22:04:51.935290 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:04:51.935253    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-zzfcp"]
Sep 09 22:04:51.940295 ip-10-0-134-72 kubenswrapper[2576]: W0909 22:04:51.940262    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc40079b_c0f0_4119_94b9_fbe3e3b7cb13.slice/crio-60bca34867fc66b4b7f40eba7553482150c9cfa2a09b95f1d6e0f286a9a21eec WatchSource:0}: Error finding container 60bca34867fc66b4b7f40eba7553482150c9cfa2a09b95f1d6e0f286a9a21eec: Status 404 returned error can't find the container with id 60bca34867fc66b4b7f40eba7553482150c9cfa2a09b95f1d6e0f286a9a21eec
Sep 09 22:04:52.137988 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:04:52.137944    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-zzfcp" event={"ID":"cc40079b-c0f0-4119-94b9-fbe3e3b7cb13","Type":"ContainerStarted","Data":"71f6ba079f0aceba5a2a13401a2734a40f78f45b0a35437904e0b5716a37e7f2"}
Sep 09 22:04:52.138161 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:04:52.138007    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-zzfcp" event={"ID":"cc40079b-c0f0-4119-94b9-fbe3e3b7cb13","Type":"ContainerStarted","Data":"60bca34867fc66b4b7f40eba7553482150c9cfa2a09b95f1d6e0f286a9a21eec"}
Sep 09 22:04:52.159930 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:04:52.159883    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-zzfcp" podStartSLOduration=1.159866458 podStartE2EDuration="1.159866458s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 22:04:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 22:04:52.158640781 +0000 UTC m=+4335.272326664" watchObservedRunningTime="2026-09-09 22:04:52.159866458 +0000 UTC m=+4335.273552360"
Sep 09 22:04:52.661839 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:04:52.661806    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-zzfcp"
Sep 09 22:05:01.661909 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:05:01.661860    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-zzfcp"
Sep 09 22:05:01.674682 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:05:01.674652    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-zzfcp"
Sep 09 22:05:02.198760 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:05:02.198720    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-zzfcp"
Sep 09 22:05:02.250474 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:05:02.250433    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qqbc4"]
Sep 09 22:05:02.260032 ip-10-0-134-72 kubenswrapper[2576]: E0909 22:05:02.259988    2576 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 09 22:05:02.260032 ip-10-0-134-72 kubenswrapper[2576]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 22:05:02.260032 ip-10-0-134-72 kubenswrapper[2576]:         , stdout: , stderr: , exit code -1
Sep 09 22:05:02.260032 ip-10-0-134-72 kubenswrapper[2576]:  > containerID="d1b18f401006214cdfb02335280f21cea5f9eda8f86ad15dfdc6172fcf65ffd8" cmd=["/bin/sleep","15"]
Sep 09 22:05:02.260313 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:05:02.260049    2576 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qqbc4" message="" err=<
Sep 09 22:05:02.260313 ip-10-0-134-72 kubenswrapper[2576]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 22:05:02.260313 ip-10-0-134-72 kubenswrapper[2576]:         , stdout: , stderr: , exit code -1
Sep 09 22:05:02.260313 ip-10-0-134-72 kubenswrapper[2576]:  >
Sep 09 22:05:02.260313 ip-10-0-134-72 kubenswrapper[2576]: E0909 22:05:02.260063    2576 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 09 22:05:02.260313 ip-10-0-134-72 kubenswrapper[2576]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 22:05:02.260313 ip-10-0-134-72 kubenswrapper[2576]:         , stdout: , stderr: , exit code -1
Sep 09 22:05:02.260313 ip-10-0-134-72 kubenswrapper[2576]:  > pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qqbc4" podUID="e5cd9703-6ff3-417c-b8f6-f453fc3916c6" containerName="main" containerID="cri-o://d1b18f401006214cdfb02335280f21cea5f9eda8f86ad15dfdc6172fcf65ffd8"
Sep 09 22:05:02.260313 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:05:02.260101    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qqbc4" podUID="e5cd9703-6ff3-417c-b8f6-f453fc3916c6" containerName="main" containerID="cri-o://d1b18f401006214cdfb02335280f21cea5f9eda8f86ad15dfdc6172fcf65ffd8" gracePeriod=60
Sep 09 22:05:02.607892 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:05:02.607859    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qqbc4"
Sep 09 22:05:02.717788 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:05:02.717740    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5cd9703-6ff3-417c-b8f6-f453fc3916c6-tmp-dir\") pod \"e5cd9703-6ff3-417c-b8f6-f453fc3916c6\" (UID: \"e5cd9703-6ff3-417c-b8f6-f453fc3916c6\") "
Sep 09 22:05:02.717788 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:05:02.717791    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5cd9703-6ff3-417c-b8f6-f453fc3916c6-dshm\") pod \"e5cd9703-6ff3-417c-b8f6-f453fc3916c6\" (UID: \"e5cd9703-6ff3-417c-b8f6-f453fc3916c6\") "
Sep 09 22:05:02.718389 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:05:02.717828    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5cd9703-6ff3-417c-b8f6-f453fc3916c6-tls-certs\") pod \"e5cd9703-6ff3-417c-b8f6-f453fc3916c6\" (UID: \"e5cd9703-6ff3-417c-b8f6-f453fc3916c6\") "
Sep 09 22:05:02.718389 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:05:02.717858    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5cd9703-6ff3-417c-b8f6-f453fc3916c6-model-cache\") pod \"e5cd9703-6ff3-417c-b8f6-f453fc3916c6\" (UID: \"e5cd9703-6ff3-417c-b8f6-f453fc3916c6\") "
Sep 09 22:05:02.718389 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:05:02.717904    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5cd9703-6ff3-417c-b8f6-f453fc3916c6-home\") pod \"e5cd9703-6ff3-417c-b8f6-f453fc3916c6\" (UID: \"e5cd9703-6ff3-417c-b8f6-f453fc3916c6\") "
Sep 09 22:05:02.718389 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:05:02.717946    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-99k47\" (UniqueName: \"kubernetes.io/projected/e5cd9703-6ff3-417c-b8f6-f453fc3916c6-kube-api-access-99k47\") pod \"e5cd9703-6ff3-417c-b8f6-f453fc3916c6\" (UID: \"e5cd9703-6ff3-417c-b8f6-f453fc3916c6\") "
Sep 09 22:05:02.718389 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:05:02.718102    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5cd9703-6ff3-417c-b8f6-f453fc3916c6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e5cd9703-6ff3-417c-b8f6-f453fc3916c6" (UID: "e5cd9703-6ff3-417c-b8f6-f453fc3916c6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:05:02.718389 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:05:02.718152    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5cd9703-6ff3-417c-b8f6-f453fc3916c6-model-cache" (OuterVolumeSpecName: "model-cache") pod "e5cd9703-6ff3-417c-b8f6-f453fc3916c6" (UID: "e5cd9703-6ff3-417c-b8f6-f453fc3916c6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:05:02.718389 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:05:02.718188    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5cd9703-6ff3-417c-b8f6-f453fc3916c6-home" (OuterVolumeSpecName: "home") pod "e5cd9703-6ff3-417c-b8f6-f453fc3916c6" (UID: "e5cd9703-6ff3-417c-b8f6-f453fc3916c6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:05:02.718389 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:05:02.718286    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5cd9703-6ff3-417c-b8f6-f453fc3916c6-tmp-dir\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 22:05:02.718389 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:05:02.718302    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5cd9703-6ff3-417c-b8f6-f453fc3916c6-model-cache\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 22:05:02.718389 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:05:02.718316    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5cd9703-6ff3-417c-b8f6-f453fc3916c6-home\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 22:05:02.720421 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:05:02.720393    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5cd9703-6ff3-417c-b8f6-f453fc3916c6-dshm" (OuterVolumeSpecName: "dshm") pod "e5cd9703-6ff3-417c-b8f6-f453fc3916c6" (UID: "e5cd9703-6ff3-417c-b8f6-f453fc3916c6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:05:02.720608 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:05:02.720588    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5cd9703-6ff3-417c-b8f6-f453fc3916c6-kube-api-access-99k47" (OuterVolumeSpecName: "kube-api-access-99k47") pod "e5cd9703-6ff3-417c-b8f6-f453fc3916c6" (UID: "e5cd9703-6ff3-417c-b8f6-f453fc3916c6"). InnerVolumeSpecName "kube-api-access-99k47". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 22:05:02.720680 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:05:02.720591    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5cd9703-6ff3-417c-b8f6-f453fc3916c6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e5cd9703-6ff3-417c-b8f6-f453fc3916c6" (UID: "e5cd9703-6ff3-417c-b8f6-f453fc3916c6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 22:05:02.819652 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:05:02.819558    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5cd9703-6ff3-417c-b8f6-f453fc3916c6-dshm\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 22:05:02.819652 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:05:02.819592    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5cd9703-6ff3-417c-b8f6-f453fc3916c6-tls-certs\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 22:05:02.819652 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:05:02.819604    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99k47\" (UniqueName: \"kubernetes.io/projected/e5cd9703-6ff3-417c-b8f6-f453fc3916c6-kube-api-access-99k47\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 22:05:03.195041 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:05:03.195006    2576 generic.go:356] "Generic (PLEG): container finished" podID="e5cd9703-6ff3-417c-b8f6-f453fc3916c6" containerID="d1b18f401006214cdfb02335280f21cea5f9eda8f86ad15dfdc6172fcf65ffd8" exitCode=0
Sep 09 22:05:03.195217 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:05:03.195072    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qqbc4"
Sep 09 22:05:03.195217 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:05:03.195095    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qqbc4" event={"ID":"e5cd9703-6ff3-417c-b8f6-f453fc3916c6","Type":"ContainerDied","Data":"d1b18f401006214cdfb02335280f21cea5f9eda8f86ad15dfdc6172fcf65ffd8"}
Sep 09 22:05:03.195217 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:05:03.195177    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qqbc4" event={"ID":"e5cd9703-6ff3-417c-b8f6-f453fc3916c6","Type":"ContainerDied","Data":"e9a1e1018b4951f410703d574486a7c70d9909f800ed7fd9f6ddd667b15d9849"}
Sep 09 22:05:03.195217 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:05:03.195207    2576 scope.go:117] "RemoveContainer" containerID="d1b18f401006214cdfb02335280f21cea5f9eda8f86ad15dfdc6172fcf65ffd8"
Sep 09 22:05:03.205331 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:05:03.205308    2576 scope.go:117] "RemoveContainer" containerID="d1b18f401006214cdfb02335280f21cea5f9eda8f86ad15dfdc6172fcf65ffd8"
Sep 09 22:05:03.205652 ip-10-0-134-72 kubenswrapper[2576]: E0909 22:05:03.205631    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1b18f401006214cdfb02335280f21cea5f9eda8f86ad15dfdc6172fcf65ffd8\": container with ID starting with d1b18f401006214cdfb02335280f21cea5f9eda8f86ad15dfdc6172fcf65ffd8 not found: ID does not exist" containerID="d1b18f401006214cdfb02335280f21cea5f9eda8f86ad15dfdc6172fcf65ffd8"
Sep 09 22:05:03.205727 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:05:03.205666    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1b18f401006214cdfb02335280f21cea5f9eda8f86ad15dfdc6172fcf65ffd8"} err="failed to get container status \"d1b18f401006214cdfb02335280f21cea5f9eda8f86ad15dfdc6172fcf65ffd8\": rpc error: code = NotFound desc = could not find container \"d1b18f401006214cdfb02335280f21cea5f9eda8f86ad15dfdc6172fcf65ffd8\": container with ID starting with d1b18f401006214cdfb02335280f21cea5f9eda8f86ad15dfdc6172fcf65ffd8 not found: ID does not exist"
Sep 09 22:05:03.217794 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:05:03.217765    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qqbc4"]
Sep 09 22:05:03.221735 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:05:03.221708    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qqbc4"]
Sep 09 22:05:03.409917 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:05:03.409879    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e5cd9703-6ff3-417c-b8f6-f453fc3916c6" path="/var/lib/kubelet/pods/e5cd9703-6ff3-417c-b8f6-f453fc3916c6/volumes"
Sep 09 22:05:03.809925 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:05:03.809891    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-zzfcp"]
Sep 09 22:05:03.830227 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:05:03.830195    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 22:05:03.922092 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:05:03.922052    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 22:05:04.210694 ip-10-0-134-72 kubenswrapper[2576]: E0909 22:05:04.210651    2576 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 09 22:05:04.210694 ip-10-0-134-72 kubenswrapper[2576]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 22:05:04.210694 ip-10-0-134-72 kubenswrapper[2576]:         , stdout: , stderr: , exit code -1
Sep 09 22:05:04.210694 ip-10-0-134-72 kubenswrapper[2576]:  > containerID="71f6ba079f0aceba5a2a13401a2734a40f78f45b0a35437904e0b5716a37e7f2" cmd=["/bin/sleep","15"]
Sep 09 22:05:04.210955 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:05:04.210702    2576 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-zzfcp" message="" err=<
Sep 09 22:05:04.210955 ip-10-0-134-72 kubenswrapper[2576]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 22:05:04.210955 ip-10-0-134-72 kubenswrapper[2576]:         , stdout: , stderr: , exit code -1
Sep 09 22:05:04.210955 ip-10-0-134-72 kubenswrapper[2576]:  >
Sep 09 22:05:04.210955 ip-10-0-134-72 kubenswrapper[2576]: E0909 22:05:04.210714    2576 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 09 22:05:04.210955 ip-10-0-134-72 kubenswrapper[2576]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 22:05:04.210955 ip-10-0-134-72 kubenswrapper[2576]:         , stdout: , stderr: , exit code -1
Sep 09 22:05:04.210955 ip-10-0-134-72 kubenswrapper[2576]:  > pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-zzfcp" podUID="cc40079b-c0f0-4119-94b9-fbe3e3b7cb13" containerName="main" containerID="cri-o://71f6ba079f0aceba5a2a13401a2734a40f78f45b0a35437904e0b5716a37e7f2"
Sep 09 22:05:04.210955 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:05:04.210759    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-zzfcp" podUID="cc40079b-c0f0-4119-94b9-fbe3e3b7cb13" containerName="main" containerID="cri-o://71f6ba079f0aceba5a2a13401a2734a40f78f45b0a35437904e0b5716a37e7f2" gracePeriod=60
Sep 09 22:05:04.567027 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:05:04.567004    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-zzfcp"
Sep 09 22:05:04.636468 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:05:04.636441    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc40079b-c0f0-4119-94b9-fbe3e3b7cb13-home\") pod \"cc40079b-c0f0-4119-94b9-fbe3e3b7cb13\" (UID: \"cc40079b-c0f0-4119-94b9-fbe3e3b7cb13\") "
Sep 09 22:05:04.636645 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:05:04.636476    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc40079b-c0f0-4119-94b9-fbe3e3b7cb13-tmp-dir\") pod \"cc40079b-c0f0-4119-94b9-fbe3e3b7cb13\" (UID: \"cc40079b-c0f0-4119-94b9-fbe3e3b7cb13\") "
Sep 09 22:05:04.636645 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:05:04.636506    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc40079b-c0f0-4119-94b9-fbe3e3b7cb13-tls-certs\") pod \"cc40079b-c0f0-4119-94b9-fbe3e3b7cb13\" (UID: \"cc40079b-c0f0-4119-94b9-fbe3e3b7cb13\") "
Sep 09 22:05:04.636645 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:05:04.636573    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc40079b-c0f0-4119-94b9-fbe3e3b7cb13-model-cache\") pod \"cc40079b-c0f0-4119-94b9-fbe3e3b7cb13\" (UID: \"cc40079b-c0f0-4119-94b9-fbe3e3b7cb13\") "
Sep 09 22:05:04.636645 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:05:04.636593    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kcnkw\" (UniqueName: \"kubernetes.io/projected/cc40079b-c0f0-4119-94b9-fbe3e3b7cb13-kube-api-access-kcnkw\") pod \"cc40079b-c0f0-4119-94b9-fbe3e3b7cb13\" (UID: \"cc40079b-c0f0-4119-94b9-fbe3e3b7cb13\") "
Sep 09 22:05:04.636645 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:05:04.636617    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc40079b-c0f0-4119-94b9-fbe3e3b7cb13-dshm\") pod \"cc40079b-c0f0-4119-94b9-fbe3e3b7cb13\" (UID: \"cc40079b-c0f0-4119-94b9-fbe3e3b7cb13\") "
Sep 09 22:05:04.636910 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:05:04.636710    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc40079b-c0f0-4119-94b9-fbe3e3b7cb13-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cc40079b-c0f0-4119-94b9-fbe3e3b7cb13" (UID: "cc40079b-c0f0-4119-94b9-fbe3e3b7cb13"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:05:04.636910 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:05:04.636746    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc40079b-c0f0-4119-94b9-fbe3e3b7cb13-home" (OuterVolumeSpecName: "home") pod "cc40079b-c0f0-4119-94b9-fbe3e3b7cb13" (UID: "cc40079b-c0f0-4119-94b9-fbe3e3b7cb13"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:05:04.636910 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:05:04.636862    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc40079b-c0f0-4119-94b9-fbe3e3b7cb13-model-cache" (OuterVolumeSpecName: "model-cache") pod "cc40079b-c0f0-4119-94b9-fbe3e3b7cb13" (UID: "cc40079b-c0f0-4119-94b9-fbe3e3b7cb13"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:05:04.636910 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:05:04.636874    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc40079b-c0f0-4119-94b9-fbe3e3b7cb13-home\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 22:05:04.636910 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:05:04.636893    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc40079b-c0f0-4119-94b9-fbe3e3b7cb13-tmp-dir\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 22:05:04.638839 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:05:04.638815    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc40079b-c0f0-4119-94b9-fbe3e3b7cb13-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cc40079b-c0f0-4119-94b9-fbe3e3b7cb13" (UID: "cc40079b-c0f0-4119-94b9-fbe3e3b7cb13"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 22:05:04.638839 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:05:04.638825    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc40079b-c0f0-4119-94b9-fbe3e3b7cb13-dshm" (OuterVolumeSpecName: "dshm") pod "cc40079b-c0f0-4119-94b9-fbe3e3b7cb13" (UID: "cc40079b-c0f0-4119-94b9-fbe3e3b7cb13"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:05:04.638999 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:05:04.638874    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc40079b-c0f0-4119-94b9-fbe3e3b7cb13-kube-api-access-kcnkw" (OuterVolumeSpecName: "kube-api-access-kcnkw") pod "cc40079b-c0f0-4119-94b9-fbe3e3b7cb13" (UID: "cc40079b-c0f0-4119-94b9-fbe3e3b7cb13"). InnerVolumeSpecName "kube-api-access-kcnkw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 22:05:04.737666 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:05:04.737577    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kcnkw\" (UniqueName: \"kubernetes.io/projected/cc40079b-c0f0-4119-94b9-fbe3e3b7cb13-kube-api-access-kcnkw\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 22:05:04.737666 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:05:04.737609    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc40079b-c0f0-4119-94b9-fbe3e3b7cb13-dshm\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 22:05:04.737666 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:05:04.737620    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc40079b-c0f0-4119-94b9-fbe3e3b7cb13-tls-certs\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 22:05:04.737666 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:05:04.737629    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc40079b-c0f0-4119-94b9-fbe3e3b7cb13-model-cache\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 22:05:05.207802 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:05:05.207765    2576 generic.go:356] "Generic (PLEG): container finished" podID="cc40079b-c0f0-4119-94b9-fbe3e3b7cb13" containerID="71f6ba079f0aceba5a2a13401a2734a40f78f45b0a35437904e0b5716a37e7f2" exitCode=0
Sep 09 22:05:05.208272 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:05:05.207809    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-zzfcp"
Sep 09 22:05:05.208272 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:05:05.207854    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-zzfcp" event={"ID":"cc40079b-c0f0-4119-94b9-fbe3e3b7cb13","Type":"ContainerDied","Data":"71f6ba079f0aceba5a2a13401a2734a40f78f45b0a35437904e0b5716a37e7f2"}
Sep 09 22:05:05.208272 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:05:05.207901    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-zzfcp" event={"ID":"cc40079b-c0f0-4119-94b9-fbe3e3b7cb13","Type":"ContainerDied","Data":"60bca34867fc66b4b7f40eba7553482150c9cfa2a09b95f1d6e0f286a9a21eec"}
Sep 09 22:05:05.208272 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:05:05.207924    2576 scope.go:117] "RemoveContainer" containerID="71f6ba079f0aceba5a2a13401a2734a40f78f45b0a35437904e0b5716a37e7f2"
Sep 09 22:05:05.217399 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:05:05.217176    2576 scope.go:117] "RemoveContainer" containerID="71f6ba079f0aceba5a2a13401a2734a40f78f45b0a35437904e0b5716a37e7f2"
Sep 09 22:05:05.217490 ip-10-0-134-72 kubenswrapper[2576]: E0909 22:05:05.217471    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71f6ba079f0aceba5a2a13401a2734a40f78f45b0a35437904e0b5716a37e7f2\": container with ID starting with 71f6ba079f0aceba5a2a13401a2734a40f78f45b0a35437904e0b5716a37e7f2 not found: ID does not exist" containerID="71f6ba079f0aceba5a2a13401a2734a40f78f45b0a35437904e0b5716a37e7f2"
Sep 09 22:05:05.217536 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:05:05.217498    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71f6ba079f0aceba5a2a13401a2734a40f78f45b0a35437904e0b5716a37e7f2"} err="failed to get container status \"71f6ba079f0aceba5a2a13401a2734a40f78f45b0a35437904e0b5716a37e7f2\": rpc error: code = NotFound desc = could not find container \"71f6ba079f0aceba5a2a13401a2734a40f78f45b0a35437904e0b5716a37e7f2\": container with ID starting with 71f6ba079f0aceba5a2a13401a2734a40f78f45b0a35437904e0b5716a37e7f2 not found: ID does not exist"
Sep 09 22:05:05.229158 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:05:05.229133    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-zzfcp"]
Sep 09 22:05:05.239683 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:05:05.239654    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-zzfcp"]
Sep 09 22:05:05.409816 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:05:05.409778    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cc40079b-c0f0-4119-94b9-fbe3e3b7cb13" path="/var/lib/kubelet/pods/cc40079b-c0f0-4119-94b9-fbe3e3b7cb13/volumes"
Sep 09 22:05:13.765541 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:05:13.765502    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 22:05:27.783761 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:05:27.783729    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 22:05:34.131767 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:05:34.131726    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 22:05:39.981654 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:05:39.981611    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 22:05:40.082859 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:05:40.082821    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 22:05:43.559225 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:05:43.559178    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 22:05:53.610909 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:05:53.610871    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 22:05:57.083496 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:05:57.083454    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-99zsp/must-gather-56946"]
Sep 09 22:05:57.083884 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:05:57.083827    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5cd9703-6ff3-417c-b8f6-f453fc3916c6" containerName="main"
Sep 09 22:05:57.083884 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:05:57.083840    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5cd9703-6ff3-417c-b8f6-f453fc3916c6" containerName="main"
Sep 09 22:05:57.083884 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:05:57.083854    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc40079b-c0f0-4119-94b9-fbe3e3b7cb13" containerName="main"
Sep 09 22:05:57.083884 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:05:57.083860    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc40079b-c0f0-4119-94b9-fbe3e3b7cb13" containerName="main"
Sep 09 22:05:57.084027 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:05:57.083926    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc40079b-c0f0-4119-94b9-fbe3e3b7cb13" containerName="main"
Sep 09 22:05:57.084027 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:05:57.083937    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5cd9703-6ff3-417c-b8f6-f453fc3916c6" containerName="main"
Sep 09 22:05:57.087029 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:05:57.087009    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-99zsp/must-gather-56946"
Sep 09 22:05:57.089891 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:05:57.089869    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-99zsp\"/\"default-dockercfg-f6x8z\""
Sep 09 22:05:57.090022 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:05:57.089939    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-99zsp\"/\"openshift-service-ca.crt\""
Sep 09 22:05:57.090145 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:05:57.090124    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-99zsp\"/\"kube-root-ca.crt\""
Sep 09 22:05:57.097452 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:05:57.097430    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-99zsp/must-gather-56946"]
Sep 09 22:05:57.110176 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:05:57.110143    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 22:05:57.176476 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:05:57.176441    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qx82s\" (UniqueName: \"kubernetes.io/projected/ed1b949a-1c61-4cf5-bdb5-9c94fde6316b-kube-api-access-qx82s\") pod \"must-gather-56946\" (UID: \"ed1b949a-1c61-4cf5-bdb5-9c94fde6316b\") " pod="openshift-must-gather-99zsp/must-gather-56946"
Sep 09 22:05:57.176620 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:05:57.176501    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/ed1b949a-1c61-4cf5-bdb5-9c94fde6316b-must-gather-output\") pod \"must-gather-56946\" (UID: \"ed1b949a-1c61-4cf5-bdb5-9c94fde6316b\") " pod="openshift-must-gather-99zsp/must-gather-56946"
Sep 09 22:05:57.213047 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:05:57.213014    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 22:05:57.276948 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:05:57.276917    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qx82s\" (UniqueName: \"kubernetes.io/projected/ed1b949a-1c61-4cf5-bdb5-9c94fde6316b-kube-api-access-qx82s\") pod \"must-gather-56946\" (UID: \"ed1b949a-1c61-4cf5-bdb5-9c94fde6316b\") " pod="openshift-must-gather-99zsp/must-gather-56946"
Sep 09 22:05:57.277079 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:05:57.276971    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/ed1b949a-1c61-4cf5-bdb5-9c94fde6316b-must-gather-output\") pod \"must-gather-56946\" (UID: \"ed1b949a-1c61-4cf5-bdb5-9c94fde6316b\") " pod="openshift-must-gather-99zsp/must-gather-56946"
Sep 09 22:05:57.277250 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:05:57.277234    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/ed1b949a-1c61-4cf5-bdb5-9c94fde6316b-must-gather-output\") pod \"must-gather-56946\" (UID: \"ed1b949a-1c61-4cf5-bdb5-9c94fde6316b\") " pod="openshift-must-gather-99zsp/must-gather-56946"
Sep 09 22:05:57.287445 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:05:57.287417    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qx82s\" (UniqueName: \"kubernetes.io/projected/ed1b949a-1c61-4cf5-bdb5-9c94fde6316b-kube-api-access-qx82s\") pod \"must-gather-56946\" (UID: \"ed1b949a-1c61-4cf5-bdb5-9c94fde6316b\") " pod="openshift-must-gather-99zsp/must-gather-56946"
Sep 09 22:05:57.397298 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:05:57.397259    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-99zsp/must-gather-56946"
Sep 09 22:05:57.670832 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:05:57.670806    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-99zsp/must-gather-56946"]
Sep 09 22:05:57.672428 ip-10-0-134-72 kubenswrapper[2576]: W0909 22:05:57.672394    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poded1b949a_1c61_4cf5_bdb5_9c94fde6316b.slice/crio-d070b80fe91d4333cf4c37f78c2612b8d15bc3da6948babd7a9222919c36069a WatchSource:0}: Error finding container d070b80fe91d4333cf4c37f78c2612b8d15bc3da6948babd7a9222919c36069a: Status 404 returned error can't find the container with id d070b80fe91d4333cf4c37f78c2612b8d15bc3da6948babd7a9222919c36069a
Sep 09 22:05:57.674245 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:05:57.674218    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 22:05:58.477469 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:05:58.477407    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-99zsp/must-gather-56946" event={"ID":"ed1b949a-1c61-4cf5-bdb5-9c94fde6316b","Type":"ContainerStarted","Data":"d070b80fe91d4333cf4c37f78c2612b8d15bc3da6948babd7a9222919c36069a"}
Sep 09 22:06:00.360743 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:06:00.360703    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 22:06:04.514051 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:06:04.513951    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-99zsp/must-gather-56946" event={"ID":"ed1b949a-1c61-4cf5-bdb5-9c94fde6316b","Type":"ContainerStarted","Data":"96f4673ce530ea3c52075c435b70d1b7c36326c1e380d753d88676ea79077450"}
Sep 09 22:06:04.514051 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:06:04.514005    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-99zsp/must-gather-56946" event={"ID":"ed1b949a-1c61-4cf5-bdb5-9c94fde6316b","Type":"ContainerStarted","Data":"287038f7d82a17c9c7fce1b468635ee19789bf4013e3a2f66aad32b4ecc11c3d"}
Sep 09 22:06:04.533586 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:06:04.533534    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-99zsp/must-gather-56946" podStartSLOduration=1.318521502 podStartE2EDuration="7.533519668s" totalImagesPullingTime="6.214998166s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 22:05:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 22:06:04.531304857 +0000 UTC m=+4407.644990715" watchObservedRunningTime="2026-09-09 22:06:04.533519668 +0000 UTC m=+4407.647205552"
Sep 09 22:06:06.294086 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:06:06.294047    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 22:06:06.394149 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:06:06.394111    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 22:06:24.237382 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:06:24.237337    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-s8t9x_f4780d21-1e4d-447c-82d5-8bd54a4c7ab6/authorino/0.log"
Sep 09 22:06:24.284675 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:06:24.284646    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-xkpgz_91b92fac-cbad-43d1-87f5-8d93369b092c/kuadrant-console-plugin/0.log"
Sep 09 22:06:25.839971 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:06:25.839915    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-s8t9x_f4780d21-1e4d-447c-82d5-8bd54a4c7ab6/authorino/0.log"
Sep 09 22:06:25.908514 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:06:25.908490    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-xkpgz_91b92fac-cbad-43d1-87f5-8d93369b092c/kuadrant-console-plugin/0.log"
Sep 09 22:06:27.444747 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:06:27.444706    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-s8t9x_f4780d21-1e4d-447c-82d5-8bd54a4c7ab6/authorino/0.log"
Sep 09 22:06:27.510489 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:06:27.510460    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-xkpgz_91b92fac-cbad-43d1-87f5-8d93369b092c/kuadrant-console-plugin/0.log"
Sep 09 22:06:29.003533 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:06:29.003486    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-s8t9x_f4780d21-1e4d-447c-82d5-8bd54a4c7ab6/authorino/0.log"
Sep 09 22:06:29.074143 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:06:29.074118    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-xkpgz_91b92fac-cbad-43d1-87f5-8d93369b092c/kuadrant-console-plugin/0.log"
Sep 09 22:06:30.586240 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:06:30.586201    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-s8t9x_f4780d21-1e4d-447c-82d5-8bd54a4c7ab6/authorino/0.log"
Sep 09 22:06:30.647803 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:06:30.647772    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-xkpgz_91b92fac-cbad-43d1-87f5-8d93369b092c/kuadrant-console-plugin/0.log"
Sep 09 22:06:32.149509 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:06:32.149466    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-s8t9x_f4780d21-1e4d-447c-82d5-8bd54a4c7ab6/authorino/0.log"
Sep 09 22:06:32.215115 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:06:32.215084    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-xkpgz_91b92fac-cbad-43d1-87f5-8d93369b092c/kuadrant-console-plugin/0.log"
Sep 09 22:06:33.695868 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:06:33.695827    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-s8t9x_f4780d21-1e4d-447c-82d5-8bd54a4c7ab6/authorino/0.log"
Sep 09 22:06:33.773917 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:06:33.773892    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-xkpgz_91b92fac-cbad-43d1-87f5-8d93369b092c/kuadrant-console-plugin/0.log"
Sep 09 22:06:35.284762 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:06:35.284707    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-s8t9x_f4780d21-1e4d-447c-82d5-8bd54a4c7ab6/authorino/0.log"
Sep 09 22:06:35.344959 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:06:35.344926    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-xkpgz_91b92fac-cbad-43d1-87f5-8d93369b092c/kuadrant-console-plugin/0.log"
Sep 09 22:06:36.879111 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:06:36.879077    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-s8t9x_f4780d21-1e4d-447c-82d5-8bd54a4c7ab6/authorino/0.log"
Sep 09 22:06:36.953238 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:06:36.953210    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-xkpgz_91b92fac-cbad-43d1-87f5-8d93369b092c/kuadrant-console-plugin/0.log"
Sep 09 22:06:38.463220 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:06:38.463171    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-s8t9x_f4780d21-1e4d-447c-82d5-8bd54a4c7ab6/authorino/0.log"
Sep 09 22:06:38.529769 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:06:38.529744    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-xkpgz_91b92fac-cbad-43d1-87f5-8d93369b092c/kuadrant-console-plugin/0.log"
Sep 09 22:06:40.029140 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:06:40.029108    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-s8t9x_f4780d21-1e4d-447c-82d5-8bd54a4c7ab6/authorino/0.log"
Sep 09 22:06:40.084674 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:06:40.084638    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-xkpgz_91b92fac-cbad-43d1-87f5-8d93369b092c/kuadrant-console-plugin/0.log"
Sep 09 22:06:41.565877 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:06:41.565827    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-s8t9x_f4780d21-1e4d-447c-82d5-8bd54a4c7ab6/authorino/0.log"
Sep 09 22:06:41.630842 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:06:41.630817    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-xkpgz_91b92fac-cbad-43d1-87f5-8d93369b092c/kuadrant-console-plugin/0.log"
Sep 09 22:06:43.113879 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:06:43.113836    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-s8t9x_f4780d21-1e4d-447c-82d5-8bd54a4c7ab6/authorino/0.log"
Sep 09 22:06:43.184477 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:06:43.184446    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-xkpgz_91b92fac-cbad-43d1-87f5-8d93369b092c/kuadrant-console-plugin/0.log"
Sep 09 22:06:44.673084 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:06:44.673050    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-s8t9x_f4780d21-1e4d-447c-82d5-8bd54a4c7ab6/authorino/0.log"
Sep 09 22:06:44.750892 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:06:44.750858    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-xkpgz_91b92fac-cbad-43d1-87f5-8d93369b092c/kuadrant-console-plugin/0.log"
Sep 09 22:06:46.225435 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:06:46.225398    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-s8t9x_f4780d21-1e4d-447c-82d5-8bd54a4c7ab6/authorino/0.log"
Sep 09 22:06:46.284438 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:06:46.284408    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-xkpgz_91b92fac-cbad-43d1-87f5-8d93369b092c/kuadrant-console-plugin/0.log"
Sep 09 22:06:47.803346 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:06:47.803305    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-s8t9x_f4780d21-1e4d-447c-82d5-8bd54a4c7ab6/authorino/0.log"
Sep 09 22:06:47.869063 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:06:47.869037    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-xkpgz_91b92fac-cbad-43d1-87f5-8d93369b092c/kuadrant-console-plugin/0.log"
Sep 09 22:06:49.363055 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:06:49.363021    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-s8t9x_f4780d21-1e4d-447c-82d5-8bd54a4c7ab6/authorino/0.log"
Sep 09 22:06:49.430217 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:06:49.430185    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-xkpgz_91b92fac-cbad-43d1-87f5-8d93369b092c/kuadrant-console-plugin/0.log"
Sep 09 22:06:50.903437 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:06:50.903391    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-s8t9x_f4780d21-1e4d-447c-82d5-8bd54a4c7ab6/authorino/0.log"
Sep 09 22:06:50.971907 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:06:50.971880    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-xkpgz_91b92fac-cbad-43d1-87f5-8d93369b092c/kuadrant-console-plugin/0.log"
Sep 09 22:06:52.455101 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:06:52.455070    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-s8t9x_f4780d21-1e4d-447c-82d5-8bd54a4c7ab6/authorino/0.log"
Sep 09 22:06:52.524856 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:06:52.524826    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-xkpgz_91b92fac-cbad-43d1-87f5-8d93369b092c/kuadrant-console-plugin/0.log"
Sep 09 22:06:53.987706 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:06:53.987675    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-s8t9x_f4780d21-1e4d-447c-82d5-8bd54a4c7ab6/authorino/0.log"
Sep 09 22:06:54.059688 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:06:54.059662    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-xkpgz_91b92fac-cbad-43d1-87f5-8d93369b092c/kuadrant-console-plugin/0.log"
Sep 09 22:06:55.559417 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:06:55.559337    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-s8t9x_f4780d21-1e4d-447c-82d5-8bd54a4c7ab6/authorino/0.log"
Sep 09 22:06:55.603159 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:06:55.603131    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-xkpgz_91b92fac-cbad-43d1-87f5-8d93369b092c/kuadrant-console-plugin/0.log"
Sep 09 22:06:57.139936 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:06:57.139890    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-s8t9x_f4780d21-1e4d-447c-82d5-8bd54a4c7ab6/authorino/0.log"
Sep 09 22:06:57.194904 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:06:57.194877    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-xkpgz_91b92fac-cbad-43d1-87f5-8d93369b092c/kuadrant-console-plugin/0.log"
Sep 09 22:06:58.700786 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:06:58.700744    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-s8t9x_f4780d21-1e4d-447c-82d5-8bd54a4c7ab6/authorino/0.log"
Sep 09 22:06:58.776045 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:06:58.776020    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-xkpgz_91b92fac-cbad-43d1-87f5-8d93369b092c/kuadrant-console-plugin/0.log"
Sep 09 22:07:00.219388 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:07:00.219330    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-s8t9x_f4780d21-1e4d-447c-82d5-8bd54a4c7ab6/authorino/0.log"
Sep 09 22:07:00.290666 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:07:00.290639    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-xkpgz_91b92fac-cbad-43d1-87f5-8d93369b092c/kuadrant-console-plugin/0.log"
Sep 09 22:07:01.793552 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:07:01.793516    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-s8t9x_f4780d21-1e4d-447c-82d5-8bd54a4c7ab6/authorino/0.log"
Sep 09 22:07:01.870122 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:07:01.870087    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-xkpgz_91b92fac-cbad-43d1-87f5-8d93369b092c/kuadrant-console-plugin/0.log"
Sep 09 22:07:02.869929 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:07:02.869886    2576 generic.go:356] "Generic (PLEG): container finished" podID="ed1b949a-1c61-4cf5-bdb5-9c94fde6316b" containerID="287038f7d82a17c9c7fce1b468635ee19789bf4013e3a2f66aad32b4ecc11c3d" exitCode=0
Sep 09 22:07:02.870469 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:07:02.869963    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-99zsp/must-gather-56946" event={"ID":"ed1b949a-1c61-4cf5-bdb5-9c94fde6316b","Type":"ContainerDied","Data":"287038f7d82a17c9c7fce1b468635ee19789bf4013e3a2f66aad32b4ecc11c3d"}
Sep 09 22:07:02.870469 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:07:02.870294    2576 scope.go:117] "RemoveContainer" containerID="287038f7d82a17c9c7fce1b468635ee19789bf4013e3a2f66aad32b4ecc11c3d"
Sep 09 22:07:03.539384 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:07:03.539347    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-99zsp_must-gather-56946_ed1b949a-1c61-4cf5-bdb5-9c94fde6316b/gather/0.log"
Sep 09 22:07:04.060500 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:07:04.060464    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 22:07:04.151964 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:07:04.151928    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-tw9dj/must-gather-6srn7"]
Sep 09 22:07:04.156594 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:07:04.156574    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tw9dj/must-gather-6srn7"
Sep 09 22:07:04.158968 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:07:04.158946    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-tw9dj\"/\"default-dockercfg-st6zb\""
Sep 09 22:07:04.158968 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:07:04.158946    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-tw9dj\"/\"openshift-service-ca.crt\""
Sep 09 22:07:04.159199 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:07:04.159184    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-tw9dj\"/\"kube-root-ca.crt\""
Sep 09 22:07:04.162888 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:07:04.162864    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tw9dj/must-gather-6srn7"]
Sep 09 22:07:04.172572 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:07:04.172541    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 22:07:04.275840 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:07:04.275804    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 22:07:04.323866 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:07:04.323778    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvvwn\" (UniqueName: \"kubernetes.io/projected/6b758275-1148-4bba-bad5-7ba1035a3073-kube-api-access-mvvwn\") pod \"must-gather-6srn7\" (UID: \"6b758275-1148-4bba-bad5-7ba1035a3073\") " pod="openshift-must-gather-tw9dj/must-gather-6srn7"
Sep 09 22:07:04.323997 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:07:04.323911    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/6b758275-1148-4bba-bad5-7ba1035a3073-must-gather-output\") pod \"must-gather-6srn7\" (UID: \"6b758275-1148-4bba-bad5-7ba1035a3073\") " pod="openshift-must-gather-tw9dj/must-gather-6srn7"
Sep 09 22:07:04.425200 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:07:04.425158    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/6b758275-1148-4bba-bad5-7ba1035a3073-must-gather-output\") pod \"must-gather-6srn7\" (UID: \"6b758275-1148-4bba-bad5-7ba1035a3073\") " pod="openshift-must-gather-tw9dj/must-gather-6srn7"
Sep 09 22:07:04.425424 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:07:04.425221    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvvwn\" (UniqueName: \"kubernetes.io/projected/6b758275-1148-4bba-bad5-7ba1035a3073-kube-api-access-mvvwn\") pod \"must-gather-6srn7\" (UID: \"6b758275-1148-4bba-bad5-7ba1035a3073\") " pod="openshift-must-gather-tw9dj/must-gather-6srn7"
Sep 09 22:07:04.425558 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:07:04.425535    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/6b758275-1148-4bba-bad5-7ba1035a3073-must-gather-output\") pod \"must-gather-6srn7\" (UID: \"6b758275-1148-4bba-bad5-7ba1035a3073\") " pod="openshift-must-gather-tw9dj/must-gather-6srn7"
Sep 09 22:07:04.434060 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:07:04.434029    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvvwn\" (UniqueName: \"kubernetes.io/projected/6b758275-1148-4bba-bad5-7ba1035a3073-kube-api-access-mvvwn\") pod \"must-gather-6srn7\" (UID: \"6b758275-1148-4bba-bad5-7ba1035a3073\") " pod="openshift-must-gather-tw9dj/must-gather-6srn7"
Sep 09 22:07:04.467131 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:07:04.467091    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tw9dj/must-gather-6srn7"
Sep 09 22:07:04.740156 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:07:04.740126    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tw9dj/must-gather-6srn7"]
Sep 09 22:07:04.741816 ip-10-0-134-72 kubenswrapper[2576]: W0909 22:07:04.741785    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6b758275_1148_4bba_bad5_7ba1035a3073.slice/crio-36dc0da268e30765d82f44776c551f75da204b6fcc326dd651ac9f22f440fc85 WatchSource:0}: Error finding container 36dc0da268e30765d82f44776c551f75da204b6fcc326dd651ac9f22f440fc85: Status 404 returned error can't find the container with id 36dc0da268e30765d82f44776c551f75da204b6fcc326dd651ac9f22f440fc85
Sep 09 22:07:04.882431 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:07:04.882393    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tw9dj/must-gather-6srn7" event={"ID":"6b758275-1148-4bba-bad5-7ba1035a3073","Type":"ContainerStarted","Data":"36dc0da268e30765d82f44776c551f75da204b6fcc326dd651ac9f22f440fc85"}
Sep 09 22:07:06.893737 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:07:06.893705    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tw9dj/must-gather-6srn7" event={"ID":"6b758275-1148-4bba-bad5-7ba1035a3073","Type":"ContainerStarted","Data":"c0952e482f52b90235f2f569b3273334023bf33bfbfdf1931438ab64f7fa3c9a"}
Sep 09 22:07:06.894085 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:07:06.893750    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tw9dj/must-gather-6srn7" event={"ID":"6b758275-1148-4bba-bad5-7ba1035a3073","Type":"ContainerStarted","Data":"a012bca224b8d139e5ca22b1a21ffc1a4a905c5afa79b4fa09f11a8b6d844f06"}
Sep 09 22:07:06.911289 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:07:06.911195    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-tw9dj/must-gather-6srn7" podStartSLOduration=1.003813671 podStartE2EDuration="2.911178411s" totalImagesPullingTime="1.90736474s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 22:07:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 22:07:06.909254306 +0000 UTC m=+4470.022940168" watchObservedRunningTime="2026-09-09 22:07:06.911178411 +0000 UTC m=+4470.024864267"
Sep 09 22:07:08.456023 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:07:08.455991    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-2c9kq_b906e5b3-61d3-41da-9776-795376a73811/global-pull-secret-syncer/0.log"
Sep 09 22:07:08.584605 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:07:08.584570    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log"
Sep 09 22:07:08.688531 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:07:08.688496    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-134-72.ec2.internal_2d40b80525651e1d9758e04c441ca328/haproxy/0.log"
Sep 09 22:07:08.998358 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:07:08.998324    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-99zsp/must-gather-56946"]
Sep 09 22:07:08.999776 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:07:08.999724    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-99zsp/must-gather-56946" podUID="ed1b949a-1c61-4cf5-bdb5-9c94fde6316b" containerName="copy" containerID="cri-o://96f4673ce530ea3c52075c435b70d1b7c36326c1e380d753d88676ea79077450" gracePeriod=2
Sep 09 22:07:09.001709 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:07:09.001687    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-99zsp/must-gather-56946"]
Sep 09 22:07:09.002037 ip-10-0-134-72 kubenswrapper[2576]: E0909 22:07:09.002011    2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-56946\" is forbidden: User \"system:node:ip-10-0-134-72.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-99zsp\": no relationship found between node 'ip-10-0-134-72.ec2.internal' and this object" podUID="ed1b949a-1c61-4cf5-bdb5-9c94fde6316b" pod="openshift-must-gather-99zsp/must-gather-56946"
Sep 09 22:07:09.274681 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:07:09.274641    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-99zsp_must-gather-56946_ed1b949a-1c61-4cf5-bdb5-9c94fde6316b/copy/0.log"
Sep 09 22:07:09.276713 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:07:09.276690    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-99zsp/must-gather-56946"
Sep 09 22:07:09.279184 ip-10-0-134-72 kubenswrapper[2576]: E0909 22:07:09.279145    2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-56946\" is forbidden: User \"system:node:ip-10-0-134-72.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-99zsp\": no relationship found between node 'ip-10-0-134-72.ec2.internal' and this object" podUID="ed1b949a-1c61-4cf5-bdb5-9c94fde6316b" pod="openshift-must-gather-99zsp/must-gather-56946"
Sep 09 22:07:09.370082 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:07:09.370054    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/ed1b949a-1c61-4cf5-bdb5-9c94fde6316b-must-gather-output\") pod \"ed1b949a-1c61-4cf5-bdb5-9c94fde6316b\" (UID: \"ed1b949a-1c61-4cf5-bdb5-9c94fde6316b\") "
Sep 09 22:07:09.370234 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:07:09.370124    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qx82s\" (UniqueName: \"kubernetes.io/projected/ed1b949a-1c61-4cf5-bdb5-9c94fde6316b-kube-api-access-qx82s\") pod \"ed1b949a-1c61-4cf5-bdb5-9c94fde6316b\" (UID: \"ed1b949a-1c61-4cf5-bdb5-9c94fde6316b\") "
Sep 09 22:07:09.373171 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:07:09.373134    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed1b949a-1c61-4cf5-bdb5-9c94fde6316b-kube-api-access-qx82s" (OuterVolumeSpecName: "kube-api-access-qx82s") pod "ed1b949a-1c61-4cf5-bdb5-9c94fde6316b" (UID: "ed1b949a-1c61-4cf5-bdb5-9c94fde6316b"). InnerVolumeSpecName "kube-api-access-qx82s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 22:07:09.376504 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:07:09.376472    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed1b949a-1c61-4cf5-bdb5-9c94fde6316b-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "ed1b949a-1c61-4cf5-bdb5-9c94fde6316b" (UID: "ed1b949a-1c61-4cf5-bdb5-9c94fde6316b"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 22:07:09.409777 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:07:09.409744    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ed1b949a-1c61-4cf5-bdb5-9c94fde6316b" path="/var/lib/kubelet/pods/ed1b949a-1c61-4cf5-bdb5-9c94fde6316b/volumes"
Sep 09 22:07:09.471669 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:07:09.471627    2576 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/ed1b949a-1c61-4cf5-bdb5-9c94fde6316b-must-gather-output\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 22:07:09.471669 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:07:09.471655    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qx82s\" (UniqueName: \"kubernetes.io/projected/ed1b949a-1c61-4cf5-bdb5-9c94fde6316b-kube-api-access-qx82s\") on node \"ip-10-0-134-72.ec2.internal\" DevicePath \"\""
Sep 09 22:07:09.912192 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:07:09.912162    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-99zsp_must-gather-56946_ed1b949a-1c61-4cf5-bdb5-9c94fde6316b/copy/0.log"
Sep 09 22:07:09.912572 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:07:09.912551    2576 generic.go:356] "Generic (PLEG): container finished" podID="ed1b949a-1c61-4cf5-bdb5-9c94fde6316b" containerID="96f4673ce530ea3c52075c435b70d1b7c36326c1e380d753d88676ea79077450" exitCode=143
Sep 09 22:07:09.912643 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:07:09.912596    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-99zsp/must-gather-56946"
Sep 09 22:07:09.912682 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:07:09.912654    2576 scope.go:117] "RemoveContainer" containerID="96f4673ce530ea3c52075c435b70d1b7c36326c1e380d753d88676ea79077450"
Sep 09 22:07:09.921450 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:07:09.921432    2576 scope.go:117] "RemoveContainer" containerID="287038f7d82a17c9c7fce1b468635ee19789bf4013e3a2f66aad32b4ecc11c3d"
Sep 09 22:07:09.937918 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:07:09.937893    2576 scope.go:117] "RemoveContainer" containerID="96f4673ce530ea3c52075c435b70d1b7c36326c1e380d753d88676ea79077450"
Sep 09 22:07:09.938239 ip-10-0-134-72 kubenswrapper[2576]: E0909 22:07:09.938214    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96f4673ce530ea3c52075c435b70d1b7c36326c1e380d753d88676ea79077450\": container with ID starting with 96f4673ce530ea3c52075c435b70d1b7c36326c1e380d753d88676ea79077450 not found: ID does not exist" containerID="96f4673ce530ea3c52075c435b70d1b7c36326c1e380d753d88676ea79077450"
Sep 09 22:07:09.938303 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:07:09.938248    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96f4673ce530ea3c52075c435b70d1b7c36326c1e380d753d88676ea79077450"} err="failed to get container status \"96f4673ce530ea3c52075c435b70d1b7c36326c1e380d753d88676ea79077450\": rpc error: code = NotFound desc = could not find container \"96f4673ce530ea3c52075c435b70d1b7c36326c1e380d753d88676ea79077450\": container with ID starting with 96f4673ce530ea3c52075c435b70d1b7c36326c1e380d753d88676ea79077450 not found: ID does not exist"
Sep 09 22:07:09.938303 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:07:09.938265    2576 scope.go:117] "RemoveContainer" containerID="287038f7d82a17c9c7fce1b468635ee19789bf4013e3a2f66aad32b4ecc11c3d"
Sep 09 22:07:09.938548 ip-10-0-134-72 kubenswrapper[2576]: E0909 22:07:09.938532    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"287038f7d82a17c9c7fce1b468635ee19789bf4013e3a2f66aad32b4ecc11c3d\": container with ID starting with 287038f7d82a17c9c7fce1b468635ee19789bf4013e3a2f66aad32b4ecc11c3d not found: ID does not exist" containerID="287038f7d82a17c9c7fce1b468635ee19789bf4013e3a2f66aad32b4ecc11c3d"
Sep 09 22:07:09.938609 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:07:09.938552    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"287038f7d82a17c9c7fce1b468635ee19789bf4013e3a2f66aad32b4ecc11c3d"} err="failed to get container status \"287038f7d82a17c9c7fce1b468635ee19789bf4013e3a2f66aad32b4ecc11c3d\": rpc error: code = NotFound desc = could not find container \"287038f7d82a17c9c7fce1b468635ee19789bf4013e3a2f66aad32b4ecc11c3d\": container with ID starting with 287038f7d82a17c9c7fce1b468635ee19789bf4013e3a2f66aad32b4ecc11c3d not found: ID does not exist"
Sep 09 22:07:10.350855 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:07:10.350819    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 22:07:10.451321 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:07:10.451288    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 22:07:13.986943 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:07:13.986910    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-s8t9x_f4780d21-1e4d-447c-82d5-8bd54a4c7ab6/authorino/0.log"
Sep 09 22:07:14.080287 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:07:14.080087    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-xkpgz_91b92fac-cbad-43d1-87f5-8d93369b092c/kuadrant-console-plugin/0.log"
Sep 09 22:07:15.367328 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:07:15.367292    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9d8fce6c-74ee-4766-8f8c-c1d4a05337de/alertmanager/0.log"
Sep 09 22:07:15.395565 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:07:15.395541    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9d8fce6c-74ee-4766-8f8c-c1d4a05337de/config-reloader/0.log"
Sep 09 22:07:15.436320 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:07:15.436271    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9d8fce6c-74ee-4766-8f8c-c1d4a05337de/kube-rbac-proxy-web/0.log"
Sep 09 22:07:15.460764 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:07:15.460721    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9d8fce6c-74ee-4766-8f8c-c1d4a05337de/kube-rbac-proxy/0.log"
Sep 09 22:07:15.486339 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:07:15.486304    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9d8fce6c-74ee-4766-8f8c-c1d4a05337de/kube-rbac-proxy-metric/0.log"
Sep 09 22:07:15.526797 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:07:15.526759    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9d8fce6c-74ee-4766-8f8c-c1d4a05337de/prom-label-proxy/0.log"
Sep 09 22:07:15.557122 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:07:15.557084    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9d8fce6c-74ee-4766-8f8c-c1d4a05337de/init-config-reloader/0.log"
Sep 09 22:07:16.046263 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:07:16.046172    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-svqd5_392d4ac7-d590-4225-bef4-c66590b846cb/node-exporter/0.log"
Sep 09 22:07:16.066873 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:07:16.066840    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-svqd5_392d4ac7-d590-4225-bef4-c66590b846cb/kube-rbac-proxy/0.log"
Sep 09 22:07:16.087969 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:07:16.087944    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-svqd5_392d4ac7-d590-4225-bef4-c66590b846cb/init-textfile/0.log"
Sep 09 22:07:16.408908 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:07:16.408874    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-78f599db79-dvfw5_ae5f845e-636d-4484-9c71-bae4cc3c2d2a/prometheus-operator-admission-webhook/0.log"
Sep 09 22:07:16.438570 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:07:16.438518    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6c5555b889-cjqvq_c40f0ae4-b1ca-4c1e-b2b8-c6c122c185dc/telemeter-client/0.log"
Sep 09 22:07:16.462803 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:07:16.462769    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6c5555b889-cjqvq_c40f0ae4-b1ca-4c1e-b2b8-c6c122c185dc/reload/0.log"
Sep 09 22:07:16.496059 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:07:16.495811    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6c5555b889-cjqvq_c40f0ae4-b1ca-4c1e-b2b8-c6c122c185dc/kube-rbac-proxy/0.log"
Sep 09 22:07:18.803244 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:07:18.803213    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-7cb4cf479c-564b2_584b0d1e-fd29-4d8c-bff4-c268361542fe/console/0.log"
Sep 09 22:07:18.858190 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:07:18.858158    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-85d98c998f-wqqtc_ca8ff022-fc16-478e-8dad-d819bf9331f8/download-server/0.log"
Sep 09 22:07:19.140955 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:07:19.140914    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-tw9dj/perf-node-gather-daemonset-rr4rh"]
Sep 09 22:07:19.141273 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:07:19.141258    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed1b949a-1c61-4cf5-bdb5-9c94fde6316b" containerName="gather"
Sep 09 22:07:19.141361 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:07:19.141276    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed1b949a-1c61-4cf5-bdb5-9c94fde6316b" containerName="gather"
Sep 09 22:07:19.141361 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:07:19.141312    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed1b949a-1c61-4cf5-bdb5-9c94fde6316b" containerName="copy"
Sep 09 22:07:19.141361 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:07:19.141321    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed1b949a-1c61-4cf5-bdb5-9c94fde6316b" containerName="copy"
Sep 09 22:07:19.141537 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:07:19.141402    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed1b949a-1c61-4cf5-bdb5-9c94fde6316b" containerName="copy"
Sep 09 22:07:19.141537 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:07:19.141412    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed1b949a-1c61-4cf5-bdb5-9c94fde6316b" containerName="gather"
Sep 09 22:07:19.147043 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:07:19.147019    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tw9dj/perf-node-gather-daemonset-rr4rh"
Sep 09 22:07:19.156663 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:07:19.156638    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tw9dj/perf-node-gather-daemonset-rr4rh"]
Sep 09 22:07:19.170662 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:07:19.170629    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a8db60a0-f7ea-44d4-b016-9452684df836-lib-modules\") pod \"perf-node-gather-daemonset-rr4rh\" (UID: \"a8db60a0-f7ea-44d4-b016-9452684df836\") " pod="openshift-must-gather-tw9dj/perf-node-gather-daemonset-rr4rh"
Sep 09 22:07:19.170915 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:07:19.170896    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqklx\" (UniqueName: \"kubernetes.io/projected/a8db60a0-f7ea-44d4-b016-9452684df836-kube-api-access-nqklx\") pod \"perf-node-gather-daemonset-rr4rh\" (UID: \"a8db60a0-f7ea-44d4-b016-9452684df836\") " pod="openshift-must-gather-tw9dj/perf-node-gather-daemonset-rr4rh"
Sep 09 22:07:19.171056 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:07:19.171043    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a8db60a0-f7ea-44d4-b016-9452684df836-proc\") pod \"perf-node-gather-daemonset-rr4rh\" (UID: \"a8db60a0-f7ea-44d4-b016-9452684df836\") " pod="openshift-must-gather-tw9dj/perf-node-gather-daemonset-rr4rh"
Sep 09 22:07:19.171217 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:07:19.171203    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a8db60a0-f7ea-44d4-b016-9452684df836-sys\") pod \"perf-node-gather-daemonset-rr4rh\" (UID: \"a8db60a0-f7ea-44d4-b016-9452684df836\") " pod="openshift-must-gather-tw9dj/perf-node-gather-daemonset-rr4rh"
Sep 09 22:07:19.171398 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:07:19.171381    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a8db60a0-f7ea-44d4-b016-9452684df836-podres\") pod \"perf-node-gather-daemonset-rr4rh\" (UID: \"a8db60a0-f7ea-44d4-b016-9452684df836\") " pod="openshift-must-gather-tw9dj/perf-node-gather-daemonset-rr4rh"
Sep 09 22:07:19.272051 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:07:19.272016    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a8db60a0-f7ea-44d4-b016-9452684df836-sys\") pod \"perf-node-gather-daemonset-rr4rh\" (UID: \"a8db60a0-f7ea-44d4-b016-9452684df836\") " pod="openshift-must-gather-tw9dj/perf-node-gather-daemonset-rr4rh"
Sep 09 22:07:19.272051 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:07:19.272062    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a8db60a0-f7ea-44d4-b016-9452684df836-podres\") pod \"perf-node-gather-daemonset-rr4rh\" (UID: \"a8db60a0-f7ea-44d4-b016-9452684df836\") " pod="openshift-must-gather-tw9dj/perf-node-gather-daemonset-rr4rh"
Sep 09 22:07:19.272301 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:07:19.272109    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a8db60a0-f7ea-44d4-b016-9452684df836-lib-modules\") pod \"perf-node-gather-daemonset-rr4rh\" (UID: \"a8db60a0-f7ea-44d4-b016-9452684df836\") " pod="openshift-must-gather-tw9dj/perf-node-gather-daemonset-rr4rh"
Sep 09 22:07:19.272301 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:07:19.272158    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqklx\" (UniqueName: \"kubernetes.io/projected/a8db60a0-f7ea-44d4-b016-9452684df836-kube-api-access-nqklx\") pod \"perf-node-gather-daemonset-rr4rh\" (UID: \"a8db60a0-f7ea-44d4-b016-9452684df836\") " pod="openshift-must-gather-tw9dj/perf-node-gather-daemonset-rr4rh"
Sep 09 22:07:19.272301 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:07:19.272189    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a8db60a0-f7ea-44d4-b016-9452684df836-proc\") pod \"perf-node-gather-daemonset-rr4rh\" (UID: \"a8db60a0-f7ea-44d4-b016-9452684df836\") " pod="openshift-must-gather-tw9dj/perf-node-gather-daemonset-rr4rh"
Sep 09 22:07:19.272301 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:07:19.272275    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a8db60a0-f7ea-44d4-b016-9452684df836-lib-modules\") pod \"perf-node-gather-daemonset-rr4rh\" (UID: \"a8db60a0-f7ea-44d4-b016-9452684df836\") " pod="openshift-must-gather-tw9dj/perf-node-gather-daemonset-rr4rh"
Sep 09 22:07:19.272301 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:07:19.272296    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a8db60a0-f7ea-44d4-b016-9452684df836-proc\") pod \"perf-node-gather-daemonset-rr4rh\" (UID: \"a8db60a0-f7ea-44d4-b016-9452684df836\") " pod="openshift-must-gather-tw9dj/perf-node-gather-daemonset-rr4rh"
Sep 09 22:07:19.272581 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:07:19.272272    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a8db60a0-f7ea-44d4-b016-9452684df836-podres\") pod \"perf-node-gather-daemonset-rr4rh\" (UID: \"a8db60a0-f7ea-44d4-b016-9452684df836\") " pod="openshift-must-gather-tw9dj/perf-node-gather-daemonset-rr4rh"
Sep 09 22:07:19.272581 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:07:19.272158    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a8db60a0-f7ea-44d4-b016-9452684df836-sys\") pod \"perf-node-gather-daemonset-rr4rh\" (UID: \"a8db60a0-f7ea-44d4-b016-9452684df836\") " pod="openshift-must-gather-tw9dj/perf-node-gather-daemonset-rr4rh"
Sep 09 22:07:19.285095 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:07:19.285068    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqklx\" (UniqueName: \"kubernetes.io/projected/a8db60a0-f7ea-44d4-b016-9452684df836-kube-api-access-nqklx\") pod \"perf-node-gather-daemonset-rr4rh\" (UID: \"a8db60a0-f7ea-44d4-b016-9452684df836\") " pod="openshift-must-gather-tw9dj/perf-node-gather-daemonset-rr4rh"
Sep 09 22:07:19.459786 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:07:19.459701    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tw9dj/perf-node-gather-daemonset-rr4rh"
Sep 09 22:07:19.752624 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:07:19.752587    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tw9dj/perf-node-gather-daemonset-rr4rh"]
Sep 09 22:07:19.755517 ip-10-0-134-72 kubenswrapper[2576]: W0909 22:07:19.755481    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-poda8db60a0_f7ea_44d4_b016_9452684df836.slice/crio-b07347b17fc3deb8613ef1b5b8f5bd30464cb563211e59d53f3fec3daae4d05a WatchSource:0}: Error finding container b07347b17fc3deb8613ef1b5b8f5bd30464cb563211e59d53f3fec3daae4d05a: Status 404 returned error can't find the container with id b07347b17fc3deb8613ef1b5b8f5bd30464cb563211e59d53f3fec3daae4d05a
Sep 09 22:07:19.992794 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:07:19.992742    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tw9dj/perf-node-gather-daemonset-rr4rh" event={"ID":"a8db60a0-f7ea-44d4-b016-9452684df836","Type":"ContainerStarted","Data":"34a95e33638c12f4757ca0c284bdd6f2d1d501c7d25b882b42556e55fdfc9077"}
Sep 09 22:07:19.993278 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:07:19.992859    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tw9dj/perf-node-gather-daemonset-rr4rh" event={"ID":"a8db60a0-f7ea-44d4-b016-9452684df836","Type":"ContainerStarted","Data":"b07347b17fc3deb8613ef1b5b8f5bd30464cb563211e59d53f3fec3daae4d05a"}
Sep 09 22:07:19.993714 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:07:19.993685    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-tw9dj/perf-node-gather-daemonset-rr4rh"
Sep 09 22:07:20.012148 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:07:20.012062    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-tw9dj/perf-node-gather-daemonset-rr4rh" podStartSLOduration=1.012047721 podStartE2EDuration="1.012047721s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 22:07:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 22:07:20.0113077 +0000 UTC m=+4483.124993560" watchObservedRunningTime="2026-09-09 22:07:20.012047721 +0000 UTC m=+4483.125733580"
Sep 09 22:07:20.549386 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:07:20.549329    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-968c2_f69e7fb4-fa7d-444e-9bfa-d173783106c2/dns/0.log"
Sep 09 22:07:20.570022 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:07:20.569991    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-968c2_f69e7fb4-fa7d-444e-9bfa-d173783106c2/kube-rbac-proxy/0.log"
Sep 09 22:07:20.736474 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:07:20.736445    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-wvw9h_09f40168-2bb2-4eff-8a3f-7ef28a149eed/dns-node-resolver/0.log"
Sep 09 22:07:21.303250 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:07:21.303217    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-dcv48_e706fb03-415d-407d-b105-bd06474f6882/node-ca/0.log"
Sep 09 22:07:23.044181 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:07:23.044151    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-hrrp2_4ff8d077-2ba3-4e24-997c-5f0a263c5bb0/serve-healthcheck-canary/0.log"
Sep 09 22:07:23.719590 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:07:23.719559    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-l852l_c6e98f74-7546-4383-89e2-e8ea2c8197e3/kube-rbac-proxy/0.log"
Sep 09 22:07:23.745711 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:07:23.745682    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-l852l_c6e98f74-7546-4383-89e2-e8ea2c8197e3/exporter/0.log"
Sep 09 22:07:23.771398 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:07:23.771348    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-l852l_c6e98f74-7546-4383-89e2-e8ea2c8197e3/extractor/0.log"
Sep 09 22:07:26.907898 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:07:26.907859    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-85b69767b4-kqss7_60ac6fa1-b9fb-41dc-a64d-09c22503038c/manager/0.log"
Sep 09 22:07:27.017676 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:07:27.017649    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-tw9dj/perf-node-gather-daemonset-rr4rh"
Sep 09 22:07:28.883090 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:07:28.882967    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-dlz7m_b56feb4a-6f9a-40c7-be0d-391097259ccf/s3-init/0.log"
Sep 09 22:07:28.903773 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:07:28.903741    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-dlz7m_b56feb4a-6f9a-40c7-be0d-391097259ccf/download-hf-model/0.log"
Sep 09 22:07:30.378210 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:07:30.378171    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v6vg_f30a6570-2579-4271-ba77-d75fd2fd8390/konnectivity-agent/0.log" limit=1024
Sep 09 22:07:33.966915 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:07:33.966886    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-5567m_b24b3cc1-dff7-4f2a-a5d4-5c7d65e6c8c3/migrator/0.log"
Sep 09 22:07:33.990752 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:07:33.990722    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-5567m_b24b3cc1-dff7-4f2a-a5d4-5c7d65e6c8c3/graceful-termination/0.log"
Sep 09 22:07:34.348999 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:07:34.348960    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-8d7dd4675-l9q9f_8dd00baa-1d00-498a-a56a-bc7f90c53317/kube-storage-version-migrator-operator/1.log"
Sep 09 22:07:34.351691 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:07:34.351644    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-8d7dd4675-l9q9f_8dd00baa-1d00-498a-a56a-bc7f90c53317/kube-storage-version-migrator-operator/0.log"
Sep 09 22:07:35.506359 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:07:35.506331    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-5vzc7_855262cd-a32b-4283-9482-cd25b26b0433/kube-multus/0.log"
Sep 09 22:07:35.529376 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:07:35.529345    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-d8z7g_3ede0a95-3907-4d9c-aa45-acf822104e28/kube-multus-additional-cni-plugins/0.log"
Sep 09 22:07:35.551912 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:07:35.551867    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-d8z7g_3ede0a95-3907-4d9c-aa45-acf822104e28/egress-router-binary-copy/0.log"
Sep 09 22:07:35.572726 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:07:35.572699    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-d8z7g_3ede0a95-3907-4d9c-aa45-acf822104e28/cni-plugins/0.log"
Sep 09 22:07:35.593695 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:07:35.593669    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-d8z7g_3ede0a95-3907-4d9c-aa45-acf822104e28/bond-cni-plugin/0.log"
Sep 09 22:07:35.618350 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:07:35.618320    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-d8z7g_3ede0a95-3907-4d9c-aa45-acf822104e28/routeoverride-cni/0.log"
Sep 09 22:07:35.642472 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:07:35.642440    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-d8z7g_3ede0a95-3907-4d9c-aa45-acf822104e28/whereabouts-cni-bincopy/0.log"
Sep 09 22:07:35.662685 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:07:35.662659    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-d8z7g_3ede0a95-3907-4d9c-aa45-acf822104e28/whereabouts-cni/0.log"
Sep 09 22:07:36.163104 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:07:36.163070    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-v74xn_5b815b2b-983b-4afe-900d-13dd791e4778/network-metrics-daemon/0.log"
Sep 09 22:07:36.179823 ip-10-0-134-72 kubenswrapper[2576]: I0909 22:07:36.179796    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-v74xn_5b815b2b-983b-4afe-900d-13dd791e4778/kube-rbac-proxy/0.log"