Sep 08 13:13:53.917385 ip-10-0-132-254 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 08 13:13:53.917417 ip-10-0-132-254 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 08 13:13:53.917426 ip-10-0-132-254 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 08 13:13:53.917755 ip-10-0-132-254 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 08 13:14:03.177087 ip-10-0-132-254 systemd[1]: kubelet.service: Failed to schedule restart job: Unit openvswitch.service not found.
Sep 08 13:14:03.177101 ip-10-0-132-254 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot b5562624b64c499d89717585f85dcb73 --
Sep 08 13:15:49.628804 ip-10-0-132-254 systemd[1]: Starting Kubernetes Kubelet...
Sep 08 13:15:50.009992 ip-10-0-132-254 kubenswrapper[2573]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 08 13:15:50.009992 ip-10-0-132-254 kubenswrapper[2573]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 08 13:15:50.009992 ip-10-0-132-254 kubenswrapper[2573]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 08 13:15:50.009992 ip-10-0-132-254 kubenswrapper[2573]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 08 13:15:50.009992 ip-10-0-132-254 kubenswrapper[2573]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 08 13:15:50.011054 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.010975    2573 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 08 13:15:50.017834 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.017817    2573 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 13:15:50.017834 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.017833    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 13:15:50.017902 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.017838    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 13:15:50.017902 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.017841    2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 13:15:50.017902 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.017844    2573 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 13:15:50.017902 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.017847    2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 13:15:50.017902 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.017850    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 13:15:50.017902 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.017853    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 13:15:50.017902 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.017856    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 13:15:50.017902 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.017859    2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 13:15:50.017902 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.017862    2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 13:15:50.017902 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.017864    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 13:15:50.017902 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.017874    2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 13:15:50.017902 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.017877    2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 13:15:50.017902 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.017882    2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 13:15:50.017902 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.017887    2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 13:15:50.017902 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.017892    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 13:15:50.017902 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.017895    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 13:15:50.017902 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.017898    2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 13:15:50.017902 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.017901    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 13:15:50.017902 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.017904    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 13:15:50.018439 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.017907    2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 13:15:50.018439 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.017910    2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 13:15:50.018439 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.017914    2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 13:15:50.018439 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.017917    2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 13:15:50.018439 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.017920    2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 13:15:50.018439 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.017923    2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 13:15:50.018439 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.017926    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 13:15:50.018439 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.017929    2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 13:15:50.018439 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.017932    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 13:15:50.018439 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.017934    2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 13:15:50.018439 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.017937    2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 13:15:50.018439 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.017939    2573 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 13:15:50.018439 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.017942    2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 13:15:50.018439 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.017944    2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 13:15:50.018439 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.017947    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 13:15:50.018439 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.017950    2573 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 13:15:50.018439 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.017952    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 13:15:50.018439 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.017955    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 13:15:50.018439 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.017957    2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 13:15:50.018439 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.017960    2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 13:15:50.018952 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.017964    2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 13:15:50.018952 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.017966    2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 13:15:50.018952 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.017969    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 13:15:50.018952 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.017972    2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 13:15:50.018952 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.017974    2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 13:15:50.018952 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.017977    2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 13:15:50.018952 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.017979    2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 13:15:50.018952 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.017982    2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 13:15:50.018952 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.017984    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 13:15:50.018952 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.017987    2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 13:15:50.018952 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.017990    2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 13:15:50.018952 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.017992    2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 13:15:50.018952 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.017995    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 13:15:50.018952 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.017998    2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 13:15:50.018952 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.018001    2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 13:15:50.018952 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.018003    2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 13:15:50.018952 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.018006    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 13:15:50.018952 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.018009    2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 13:15:50.018952 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.018012    2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 13:15:50.019425 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.018015    2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 13:15:50.019425 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.018017    2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 13:15:50.019425 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.018020    2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 13:15:50.019425 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.018022    2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 13:15:50.019425 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.018025    2573 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 13:15:50.019425 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.018029    2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 13:15:50.019425 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.018032    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 13:15:50.019425 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.018035    2573 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 13:15:50.019425 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.018038    2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 13:15:50.019425 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.018041    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 13:15:50.019425 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.018045    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 13:15:50.019425 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.018048    2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 13:15:50.019425 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.018050    2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 13:15:50.019425 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.018053    2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 13:15:50.019425 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.018056    2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 13:15:50.019425 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.018058    2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 13:15:50.019425 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.018061    2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 13:15:50.019425 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.018063    2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 13:15:50.019425 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.018066    2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 13:15:50.019908 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.018069    2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 13:15:50.019908 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.018072    2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 13:15:50.019908 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.018074    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 13:15:50.019908 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.018077    2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 13:15:50.019908 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.018079    2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 13:15:50.019908 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.018082    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 13:15:50.019908 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.018085    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 13:15:50.019908 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.018089    2573 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 13:15:50.019908 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.018091    2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 13:15:50.019908 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.018094    2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 13:15:50.019908 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.018097    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 13:15:50.019908 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.018099    2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 13:15:50.019908 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.018102    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 13:15:50.019908 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.018104    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 13:15:50.019908 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.018107    2573 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 13:15:50.019908 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.018110    2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 13:15:50.019908 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.018112    2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 13:15:50.019908 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.018115    2573 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 13:15:50.019908 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.018117    2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 13:15:50.019908 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.018120    2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 13:15:50.020388 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.018825    2573 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 13:15:50.020388 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.018833    2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 13:15:50.020388 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.018836    2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 13:15:50.020388 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.018840    2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 13:15:50.020388 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.018843    2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 13:15:50.020388 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.018845    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 13:15:50.020388 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.018848    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 13:15:50.020388 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.018851    2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 13:15:50.020388 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.018853    2573 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 13:15:50.020388 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.018856    2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 13:15:50.020388 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.018859    2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 13:15:50.020388 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.018861    2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 13:15:50.020388 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.018864    2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 13:15:50.020388 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.018866    2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 13:15:50.020388 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.018870    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 13:15:50.020388 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.018873    2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 13:15:50.020388 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.018875    2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 13:15:50.020388 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.018878    2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 13:15:50.020388 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.018881    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 13:15:50.020886 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.018886    2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 13:15:50.020886 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.018891    2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 13:15:50.020886 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.018895    2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 13:15:50.020886 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.018898    2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 13:15:50.020886 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.018900    2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 13:15:50.020886 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.018903    2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 13:15:50.020886 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.018905    2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 13:15:50.020886 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.018908    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 13:15:50.020886 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.018911    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 13:15:50.020886 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.018913    2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 13:15:50.020886 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.018916    2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 13:15:50.020886 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.018918    2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 13:15:50.020886 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.018921    2573 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 13:15:50.020886 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.018925    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 13:15:50.020886 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.018928    2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 13:15:50.020886 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.018931    2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 13:15:50.020886 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.018935    2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 13:15:50.020886 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.018939    2573 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 13:15:50.020886 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.018942    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 13:15:50.021376 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.018945    2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 13:15:50.021376 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.018948    2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 13:15:50.021376 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.018951    2573 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 13:15:50.021376 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.018954    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 13:15:50.021376 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.018957    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 13:15:50.021376 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.018959    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 13:15:50.021376 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.018962    2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 13:15:50.021376 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.018965    2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 13:15:50.021376 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.018968    2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 13:15:50.021376 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.018971    2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 13:15:50.021376 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.018973    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 13:15:50.021376 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.018976    2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 13:15:50.021376 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.018979    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 13:15:50.021376 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.018981    2573 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 13:15:50.021376 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.018984    2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 13:15:50.021376 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.018987    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 13:15:50.021376 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.018989    2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 13:15:50.021376 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.018992    2573 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 13:15:50.021376 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.018994    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 13:15:50.021376 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.018997    2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 13:15:50.021886 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.018999    2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 13:15:50.021886 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.019002    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 13:15:50.021886 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.019004    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 13:15:50.021886 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.019007    2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 13:15:50.021886 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.019009    2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 13:15:50.021886 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.019011    2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 13:15:50.021886 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.019014    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 13:15:50.021886 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.019019    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 13:15:50.021886 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.019022    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 13:15:50.021886 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.019024    2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 13:15:50.021886 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.019027    2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 13:15:50.021886 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.019030    2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 13:15:50.021886 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.019033    2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 13:15:50.021886 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.019035    2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 13:15:50.021886 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.019038    2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 13:15:50.021886 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.019041    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 13:15:50.021886 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.019043    2573 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 13:15:50.021886 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.019046    2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 13:15:50.021886 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.019048    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 13:15:50.021886 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.019051    2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 13:15:50.022385 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.019055    2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 13:15:50.022385 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.019065    2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 13:15:50.022385 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.019069    2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 13:15:50.022385 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.019071    2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 13:15:50.022385 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.019074    2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 13:15:50.022385 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.019077    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 13:15:50.022385 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.019079    2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 13:15:50.022385 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.019082    2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 13:15:50.022385 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.019084    2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 13:15:50.022385 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.019087    2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 13:15:50.022385 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.019090    2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 13:15:50.022385 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.019092    2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 13:15:50.022385 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.019095    2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 13:15:50.022385 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.019098    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 13:15:50.022385 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.019100    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 13:15:50.022385 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.019103    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 13:15:50.022385 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.019105    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 13:15:50.022385 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.019108    2573 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 13:15:50.022385 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.019111    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 13:15:50.022385 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.019115    2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 13:15:50.022926 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.019117    2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 13:15:50.022926 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.019527    2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 13:15:50.022926 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.019533    2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 13:15:50.022926 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.019542    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 13:15:50.022926 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.019545    2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 13:15:50.022926 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.019548    2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 13:15:50.022926 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.019551    2573 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 13:15:50.022926 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.019554    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 13:15:50.022926 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.019556    2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 13:15:50.022926 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.019559    2573 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 13:15:50.022926 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.019562    2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 13:15:50.022926 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.019566    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 13:15:50.022926 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.019569    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 13:15:50.022926 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.019572    2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 13:15:50.022926 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.019574    2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 13:15:50.022926 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.019577    2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 13:15:50.022926 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.019579    2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 13:15:50.022926 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.019582    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 13:15:50.022926 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.019585    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 13:15:50.022926 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.019587    2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 13:15:50.023411 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.019590    2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 13:15:50.023411 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.019592    2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 13:15:50.023411 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.019595    2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 13:15:50.023411 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.019597    2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 13:15:50.023411 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.019600    2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 13:15:50.023411 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.019602    2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 13:15:50.023411 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.019605    2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 13:15:50.023411 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.019607    2573 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 13:15:50.023411 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.019610    2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 13:15:50.023411 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.019613    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 13:15:50.023411 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.019615    2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 13:15:50.023411 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.019619    2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 13:15:50.023411 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.019627    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 13:15:50.023411 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.019630    2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 13:15:50.023411 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.019633    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 13:15:50.023411 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.019635    2573 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 13:15:50.023411 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.019643    2573 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 13:15:50.023411 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.019646    2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 13:15:50.023411 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.019648    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 13:15:50.023411 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.019651    2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 13:15:50.023941 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.019653    2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 13:15:50.023941 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.019656    2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 13:15:50.023941 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.019659    2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 13:15:50.023941 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.019663    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 13:15:50.023941 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.019666    2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 13:15:50.023941 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.019668    2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 13:15:50.023941 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.019671    2573 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 13:15:50.023941 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.019673    2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 13:15:50.023941 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.019676    2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 13:15:50.023941 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.019678    2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 13:15:50.023941 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.019681    2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 13:15:50.023941 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.019683    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 13:15:50.023941 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.019686    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 13:15:50.023941 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.019688    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 13:15:50.023941 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.019691    2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 13:15:50.023941 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.019693    2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 13:15:50.023941 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.019696    2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 13:15:50.023941 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.019699    2573 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 13:15:50.023941 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.019701    2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 13:15:50.023941 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.019704    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 13:15:50.024430 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.019706    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 13:15:50.024430 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.019709    2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 13:15:50.024430 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.019711    2573 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 13:15:50.024430 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.019715    2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 13:15:50.024430 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.019718    2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 13:15:50.024430 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.019720    2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 13:15:50.024430 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.019723    2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 13:15:50.024430 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.019725    2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 13:15:50.024430 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.019728    2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 13:15:50.024430 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.019735    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 13:15:50.024430 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.019738    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 13:15:50.024430 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.019740    2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 13:15:50.024430 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.019743    2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 13:15:50.024430 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.019745    2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 13:15:50.024430 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.019748    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 13:15:50.024430 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.019752    2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 13:15:50.024430 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.019754    2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 13:15:50.024430 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.019757    2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 13:15:50.024430 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.019759    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 13:15:50.024430 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.019762    2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 13:15:50.024927 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.019764    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 13:15:50.024927 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.019767    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 13:15:50.024927 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.019770    2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 13:15:50.024927 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.019772    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 13:15:50.024927 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.019774    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 13:15:50.024927 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.019777    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 13:15:50.024927 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.019779    2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 13:15:50.024927 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.019782    2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 13:15:50.024927 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.019787    2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 13:15:50.024927 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.019791    2573 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 13:15:50.024927 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.019794    2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 13:15:50.024927 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.019798    2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 13:15:50.024927 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.019801    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 13:15:50.024927 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.019804    2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 13:15:50.024927 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.019807    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 13:15:50.024927 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.019811    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 13:15:50.024927 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.019813    2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 13:15:50.024927 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.019817    2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 13:15:50.024927 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.019820    2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 13:15:50.025391 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.019823    2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 13:15:50.025391 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.019892    2573 flags.go:64] FLAG: --address="0.0.0.0"
Sep 08 13:15:50.025391 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.019930    2573 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 08 13:15:50.025391 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.019941    2573 flags.go:64] FLAG: --anonymous-auth="true"
Sep 08 13:15:50.025391 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.019945    2573 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 08 13:15:50.025391 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.019949    2573 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 08 13:15:50.025391 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.019952    2573 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 08 13:15:50.025391 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.019957    2573 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 08 13:15:50.025391 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.019963    2573 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 08 13:15:50.025391 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.019966    2573 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 08 13:15:50.025391 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.019969    2573 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 08 13:15:50.025391 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.019973    2573 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 08 13:15:50.025391 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.019976    2573 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 08 13:15:50.025391 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.019980    2573 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 08 13:15:50.025391 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.019983    2573 flags.go:64] FLAG: --cgroup-root=""
Sep 08 13:15:50.025391 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.019986    2573 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 08 13:15:50.025391 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.019989    2573 flags.go:64] FLAG: --client-ca-file=""
Sep 08 13:15:50.025391 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.019992    2573 flags.go:64] FLAG: --cloud-provider="external"
Sep 08 13:15:50.025391 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.019995    2573 flags.go:64] FLAG: --cluster-dns="[]"
Sep 08 13:15:50.025391 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.020001    2573 flags.go:64] FLAG: --cluster-domain=""
Sep 08 13:15:50.025391 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.020004    2573 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 08 13:15:50.025391 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.020007    2573 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 08 13:15:50.025391 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.020010    2573 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 08 13:15:50.025391 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.020014    2573 flags.go:64] FLAG: --container-log-max-files="5"
Sep 08 13:15:50.025984 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.020018    2573 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 08 13:15:50.025984 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.020021    2573 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 08 13:15:50.025984 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.020024    2573 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 08 13:15:50.025984 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.020027    2573 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 08 13:15:50.025984 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.020032    2573 flags.go:64] FLAG: --contention-profiling="false"
Sep 08 13:15:50.025984 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.020035    2573 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 08 13:15:50.025984 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.020038    2573 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 08 13:15:50.025984 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.020041    2573 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 08 13:15:50.025984 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.020044    2573 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 08 13:15:50.025984 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.020047    2573 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 08 13:15:50.025984 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.020051    2573 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 08 13:15:50.025984 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.020054    2573 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 08 13:15:50.025984 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.020058    2573 flags.go:64] FLAG: --enable-load-reader="false"
Sep 08 13:15:50.025984 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.020061    2573 flags.go:64] FLAG: --enable-server="true"
Sep 08 13:15:50.025984 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.020064    2573 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 08 13:15:50.025984 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.020070    2573 flags.go:64] FLAG: --event-burst="100"
Sep 08 13:15:50.025984 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.020074    2573 flags.go:64] FLAG: --event-qps="50"
Sep 08 13:15:50.025984 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.020077    2573 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 08 13:15:50.025984 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.020080    2573 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 08 13:15:50.025984 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.020083    2573 flags.go:64] FLAG: --eviction-hard=""
Sep 08 13:15:50.025984 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.020087    2573 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 08 13:15:50.025984 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.020090    2573 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 08 13:15:50.025984 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.020093    2573 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 08 13:15:50.025984 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.020096    2573 flags.go:64] FLAG: --eviction-soft=""
Sep 08 13:15:50.025984 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.020099    2573 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 08 13:15:50.026623 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.020102    2573 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 08 13:15:50.026623 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.020105    2573 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 08 13:15:50.026623 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.020108    2573 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 08 13:15:50.026623 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.020111    2573 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 08 13:15:50.026623 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.020114    2573 flags.go:64] FLAG: --fail-swap-on="true"
Sep 08 13:15:50.026623 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.020116    2573 flags.go:64] FLAG: --feature-gates=""
Sep 08 13:15:50.026623 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.020120    2573 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 08 13:15:50.026623 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.020123    2573 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 08 13:15:50.026623 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.020126    2573 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 08 13:15:50.026623 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.020129    2573 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 08 13:15:50.026623 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.020132    2573 flags.go:64] FLAG: --healthz-port="10248"
Sep 08 13:15:50.026623 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.020136    2573 flags.go:64] FLAG: --help="false"
Sep 08 13:15:50.026623 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.020139    2573 flags.go:64] FLAG: --hostname-override="ip-10-0-132-254.ec2.internal"
Sep 08 13:15:50.026623 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.020142    2573 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 08 13:15:50.026623 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.020145    2573 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 08 13:15:50.026623 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.020148    2573 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 08 13:15:50.026623 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.020151    2573 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 08 13:15:50.026623 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.020154    2573 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 08 13:15:50.026623 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.020157    2573 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 08 13:15:50.026623 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.020160    2573 flags.go:64] FLAG: --image-service-endpoint=""
Sep 08 13:15:50.026623 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.020164    2573 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 08 13:15:50.026623 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.020166    2573 flags.go:64] FLAG: --kube-api-burst="100"
Sep 08 13:15:50.026623 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.020169    2573 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 08 13:15:50.027221 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.020174    2573 flags.go:64] FLAG: --kube-api-qps="50"
Sep 08 13:15:50.027221 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.020177    2573 flags.go:64] FLAG: --kube-reserved=""
Sep 08 13:15:50.027221 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.020180    2573 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 08 13:15:50.027221 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.020183    2573 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 08 13:15:50.027221 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.020186    2573 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 08 13:15:50.027221 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.020189    2573 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 08 13:15:50.027221 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.020192    2573 flags.go:64] FLAG: --lock-file=""
Sep 08 13:15:50.027221 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.020194    2573 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 08 13:15:50.027221 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.020197    2573 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 08 13:15:50.027221 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.020201    2573 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 08 13:15:50.027221 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.020206    2573 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 08 13:15:50.027221 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.020209    2573 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 08 13:15:50.027221 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.020212    2573 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 08 13:15:50.027221 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.020215    2573 flags.go:64] FLAG: --logging-format="text"
Sep 08 13:15:50.027221 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.020218    2573 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 08 13:15:50.027221 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.020221    2573 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 08 13:15:50.027221 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.020224    2573 flags.go:64] FLAG: --manifest-url=""
Sep 08 13:15:50.027221 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.020226    2573 flags.go:64] FLAG: --manifest-url-header=""
Sep 08 13:15:50.027221 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.020231    2573 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 08 13:15:50.027221 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.020233    2573 flags.go:64] FLAG: --max-open-files="1000000"
Sep 08 13:15:50.027221 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.020239    2573 flags.go:64] FLAG: --max-pods="110"
Sep 08 13:15:50.027221 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.020242    2573 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 08 13:15:50.027221 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.020245    2573 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 08 13:15:50.027221 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.020247    2573 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 08 13:15:50.027221 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.020250    2573 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 08 13:15:50.027885 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.020253    2573 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 08 13:15:50.027885 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.020256    2573 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 08 13:15:50.027885 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.020259    2573 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 08 13:15:50.027885 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.020267    2573 flags.go:64] FLAG: --node-status-max-images="50"
Sep 08 13:15:50.027885 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.020271    2573 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 08 13:15:50.027885 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.020274    2573 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 08 13:15:50.027885 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.020277    2573 flags.go:64] FLAG: --pod-cidr=""
Sep 08 13:15:50.027885 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.020281    2573 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1"
Sep 08 13:15:50.027885 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.020287    2573 flags.go:64] FLAG: --pod-manifest-path=""
Sep 08 13:15:50.027885 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.020290    2573 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 08 13:15:50.027885 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.020293    2573 flags.go:64] FLAG: --pods-per-core="0"
Sep 08 13:15:50.027885 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.020296    2573 flags.go:64] FLAG: --port="10250"
Sep 08 13:15:50.027885 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.020299    2573 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 08 13:15:50.027885 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.020302    2573 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-04edc525c47f33499"
Sep 08 13:15:50.027885 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.020305    2573 flags.go:64] FLAG: --qos-reserved=""
Sep 08 13:15:50.027885 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.020309    2573 flags.go:64] FLAG: --read-only-port="10255"
Sep 08 13:15:50.027885 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.020312    2573 flags.go:64] FLAG: --register-node="true"
Sep 08 13:15:50.027885 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.020314    2573 flags.go:64] FLAG: --register-with-taints=""
Sep 08 13:15:50.027885 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.020318    2573 flags.go:64] FLAG: --registry-burst="10"
Sep 08 13:15:50.027885 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.020321    2573 flags.go:64] FLAG: --registry-qps="5"
Sep 08 13:15:50.027885 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.020324    2573 flags.go:64] FLAG: --reserved-cpus=""
Sep 08 13:15:50.027885 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.020327    2573 flags.go:64] FLAG: --reserved-memory=""
Sep 08 13:15:50.027885 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.020330    2573 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 08 13:15:50.027885 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.020333    2573 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 08 13:15:50.027885 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.020336    2573 flags.go:64] FLAG: --rotate-certificates="false"
Sep 08 13:15:50.028513 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.020339    2573 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 08 13:15:50.028513 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.020342    2573 flags.go:64] FLAG: --runonce="false"
Sep 08 13:15:50.028513 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.020347    2573 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 08 13:15:50.028513 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.020350    2573 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 08 13:15:50.028513 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.020353    2573 flags.go:64] FLAG: --seccomp-default="false"
Sep 08 13:15:50.028513 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.020355    2573 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 08 13:15:50.028513 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.020358    2573 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 08 13:15:50.028513 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.020361    2573 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 08 13:15:50.028513 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.020364    2573 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 08 13:15:50.028513 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.020367    2573 flags.go:64] FLAG: --storage-driver-password="root"
Sep 08 13:15:50.028513 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.020370    2573 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 08 13:15:50.028513 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.020373    2573 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 08 13:15:50.028513 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.020376    2573 flags.go:64] FLAG: --storage-driver-user="root"
Sep 08 13:15:50.028513 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.020380    2573 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 08 13:15:50.028513 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.020384    2573 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 08 13:15:50.028513 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.020388    2573 flags.go:64] FLAG: --system-cgroups=""
Sep 08 13:15:50.028513 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.020391    2573 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 08 13:15:50.028513 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.020397    2573 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 08 13:15:50.028513 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.020400    2573 flags.go:64] FLAG: --tls-cert-file=""
Sep 08 13:15:50.028513 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.020402    2573 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 08 13:15:50.028513 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.020406    2573 flags.go:64] FLAG: --tls-min-version=""
Sep 08 13:15:50.028513 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.020409    2573 flags.go:64] FLAG: --tls-private-key-file=""
Sep 08 13:15:50.028513 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.020412    2573 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 08 13:15:50.028513 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.020415    2573 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 08 13:15:50.028513 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.020418    2573 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 08 13:15:50.029134 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.020421    2573 flags.go:64] FLAG: --v="2"
Sep 08 13:15:50.029134 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.020425    2573 flags.go:64] FLAG: --version="false"
Sep 08 13:15:50.029134 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.020429    2573 flags.go:64] FLAG: --vmodule=""
Sep 08 13:15:50.029134 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.020432    2573 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 08 13:15:50.029134 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.020435    2573 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 08 13:15:50.029134 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.020565    2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 13:15:50.029134 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.020569    2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 13:15:50.029134 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.020572    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 13:15:50.029134 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.020574    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 13:15:50.029134 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.020578    2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 13:15:50.029134 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.020581    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 13:15:50.029134 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.020584    2573 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 13:15:50.029134 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.020586    2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 13:15:50.029134 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.020589    2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 13:15:50.029134 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.020592    2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 13:15:50.029134 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.020594    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 13:15:50.029134 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.020597    2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 13:15:50.029134 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.020600    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 13:15:50.029134 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.020602    2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 13:15:50.029134 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.020605    2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 13:15:50.029134 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.020607    2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 13:15:50.029134 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.020610    2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 13:15:50.029699 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.020614    2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 13:15:50.029699 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.020617    2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 13:15:50.029699 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.020620    2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 13:15:50.029699 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.020622    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 13:15:50.029699 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.020626    2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 13:15:50.029699 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.020630    2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 13:15:50.029699 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.020633    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 13:15:50.029699 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.020636    2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 13:15:50.029699 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.020640    2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 13:15:50.029699 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.020644    2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 13:15:50.029699 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.020647    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 13:15:50.029699 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.020650    2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 13:15:50.029699 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.020652    2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 13:15:50.029699 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.020655    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 13:15:50.029699 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.020657    2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 13:15:50.029699 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.020660    2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 13:15:50.029699 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.020664    2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 13:15:50.029699 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.020667    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 13:15:50.030141 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.020670    2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 13:15:50.030141 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.020673    2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 13:15:50.030141 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.020676    2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 13:15:50.030141 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.020679    2573 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 13:15:50.030141 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.020682    2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 13:15:50.030141 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.020685    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 13:15:50.030141 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.020687    2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 13:15:50.030141 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.020690    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 13:15:50.030141 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.020692    2573 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 13:15:50.030141 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.020695    2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 13:15:50.030141 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.020697    2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 13:15:50.030141 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.020700    2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 13:15:50.030141 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.020702    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 13:15:50.030141 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.020705    2573 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 13:15:50.030141 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.020709    2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 13:15:50.030141 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.020713    2573 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 13:15:50.030141 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.020715    2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 13:15:50.030141 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.020718    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 13:15:50.030141 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.020720    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 13:15:50.030619 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.020723    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 13:15:50.030619 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.020726    2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 13:15:50.030619 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.020728    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 13:15:50.030619 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.020731    2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 13:15:50.030619 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.020733    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 13:15:50.030619 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.020736    2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 13:15:50.030619 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.020738    2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 13:15:50.030619 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.020741    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 13:15:50.030619 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.020743    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 13:15:50.030619 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.020746    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 13:15:50.030619 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.020749    2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 13:15:50.030619 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.020751    2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 13:15:50.030619 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.020754    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 13:15:50.030619 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.020756    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 13:15:50.030619 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.020758    2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 13:15:50.030619 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.020761    2573 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 13:15:50.030619 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.020763    2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 13:15:50.030619 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.020766    2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 13:15:50.030619 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.020768    2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 13:15:50.030619 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.020771    2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 13:15:50.031119 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.020774    2573 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 13:15:50.031119 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.020776    2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 13:15:50.031119 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.020779    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 13:15:50.031119 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.020781    2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 13:15:50.031119 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.020784    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 13:15:50.031119 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.020786    2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 13:15:50.031119 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.020789    2573 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 13:15:50.031119 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.020793    2573 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 13:15:50.031119 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.021149    2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 13:15:50.031119 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.021153    2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 13:15:50.031119 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.021155    2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 13:15:50.031119 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.021158    2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 13:15:50.031119 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.021161    2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 13:15:50.031119 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.021163    2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 13:15:50.031119 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.021166    2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 13:15:50.031119 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.021168    2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 13:15:50.031119 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.021171    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 13:15:50.031119 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.021174    2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 13:15:50.031119 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.021176    2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 13:15:50.031119 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.021179    2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 13:15:50.031872 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.021181    2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 13:15:50.031872 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.021184    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 13:15:50.031872 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.021186    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 13:15:50.031872 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.021189    2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 13:15:50.031872 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.021191    2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 13:15:50.031872 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.021199    2573 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 08 13:15:50.031872 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.030821    2573 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 08 13:15:50.031872 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.030836    2573 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 08 13:15:50.031872 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.030849    2573 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 08 13:15:50.031872 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.030858    2573 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 08 13:15:50.031872 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.030922    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 13:15:50.031872 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.030928    2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 13:15:50.031872 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.030931    2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 13:15:50.031872 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.030934    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 13:15:50.031872 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.030937    2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 13:15:50.032474 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.030940    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 13:15:50.032474 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.030942    2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 13:15:50.032474 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.030945    2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 13:15:50.032474 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.030948    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 13:15:50.032474 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.030950    2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 13:15:50.032474 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.030954    2573 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 13:15:50.032474 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.030957    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 13:15:50.032474 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.030960    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 13:15:50.032474 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.030962    2573 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 13:15:50.032474 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.030965    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 13:15:50.032474 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.030968    2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 13:15:50.032474 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.030970    2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 13:15:50.032474 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.030973    2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 13:15:50.032474 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.030975    2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 13:15:50.032474 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.030977    2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 13:15:50.032474 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.030980    2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 13:15:50.032474 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.030982    2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 13:15:50.032474 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.030985    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 13:15:50.032474 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.030988    2573 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 13:15:50.032474 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.030990    2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 13:15:50.033122 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.030993    2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 13:15:50.033122 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.030995    2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 13:15:50.033122 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.030998    2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 13:15:50.033122 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.031001    2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 13:15:50.033122 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.031004    2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 13:15:50.033122 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.031006    2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 13:15:50.033122 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.031009    2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 13:15:50.033122 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.031012    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 13:15:50.033122 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.031014    2573 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 13:15:50.033122 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.031017    2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 13:15:50.033122 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.031019    2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 13:15:50.033122 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.031022    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 13:15:50.033122 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.031024    2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 13:15:50.033122 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.031027    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 13:15:50.033122 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.031029    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 13:15:50.033122 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.031032    2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 13:15:50.033122 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.031035    2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 13:15:50.033122 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.031037    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 13:15:50.033122 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.031040    2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 13:15:50.033662 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.031043    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 13:15:50.033662 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.031045    2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 13:15:50.033662 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.031048    2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 13:15:50.033662 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.031050    2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 13:15:50.033662 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.031053    2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 13:15:50.033662 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.031055    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 13:15:50.033662 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.031066    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 13:15:50.033662 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.031069    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 13:15:50.033662 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.031072    2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 13:15:50.033662 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.031074    2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 13:15:50.033662 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.031077    2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 13:15:50.033662 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.031079    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 13:15:50.033662 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.031082    2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 13:15:50.033662 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.031084    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 13:15:50.033662 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.031087    2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 13:15:50.033662 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.031090    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 13:15:50.033662 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.031093    2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 13:15:50.033662 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.031096    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 13:15:50.033662 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.031099    2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 13:15:50.033662 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.031101    2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 13:15:50.034143 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.031104    2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 13:15:50.034143 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.031107    2573 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 13:15:50.034143 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.031109    2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 13:15:50.034143 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.031112    2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 13:15:50.034143 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.031115    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 13:15:50.034143 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.031117    2573 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 13:15:50.034143 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.031120    2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 13:15:50.034143 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.031123    2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 13:15:50.034143 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.031125    2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 13:15:50.034143 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.031128    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 13:15:50.034143 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.031131    2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 13:15:50.034143 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.031133    2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 13:15:50.034143 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.031137    2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 13:15:50.034143 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.031139    2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 13:15:50.034143 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.031142    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 13:15:50.034143 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.031145    2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 13:15:50.034143 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.031147    2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 13:15:50.034143 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.031150    2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 13:15:50.034143 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.031152    2573 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 13:15:50.034143 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.031155    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 13:15:50.034664 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.031158    2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 13:15:50.034664 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.031160    2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 13:15:50.034664 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.031163    2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 13:15:50.034664 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.031165    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 13:15:50.034664 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.031168    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 13:15:50.034664 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.031170    2573 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 13:15:50.034664 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.031175    2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 13:15:50.034664 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.031179    2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 13:15:50.034664 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.031183    2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 13:15:50.034664 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.031186    2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 13:15:50.034664 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.031189    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 13:15:50.034664 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.031192    2573 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 13:15:50.034664 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.031195    2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 13:15:50.034664 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.031197    2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 13:15:50.034664 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.031201    2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 13:15:50.035042 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.031207    2573 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 08 13:15:50.035042 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.031784    2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 13:15:50.035042 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.031800    2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 13:15:50.035042 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.031806    2573 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 13:15:50.035042 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.031810    2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 13:15:50.035042 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.031816    2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 13:15:50.035042 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.031820    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 13:15:50.035042 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.031826    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 13:15:50.035042 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.031831    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 13:15:50.035042 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.031835    2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 13:15:50.035042 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.031841    2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 13:15:50.035042 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.031845    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 13:15:50.035042 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.031854    2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 13:15:50.035042 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.031859    2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 13:15:50.035042 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.031862    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 13:15:50.035426 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.031867    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 13:15:50.035426 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.031871    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 13:15:50.035426 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.031876    2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 13:15:50.035426 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.031881    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 13:15:50.035426 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.031886    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 13:15:50.035426 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.031890    2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 13:15:50.035426 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.031895    2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 13:15:50.035426 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.031900    2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 13:15:50.035426 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.031904    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 13:15:50.035426 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.031912    2573 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 13:15:50.035426 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.031917    2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 13:15:50.035426 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.031922    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 13:15:50.035426 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.031927    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 13:15:50.035426 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.031931    2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 13:15:50.035426 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.031935    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 13:15:50.035426 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.031939    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 13:15:50.035426 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.031943    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 13:15:50.035426 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.031950    2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 13:15:50.035426 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.031956    2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 13:15:50.035426 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.031963    2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 13:15:50.035975 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.031967    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 13:15:50.035975 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.031972    2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 13:15:50.035975 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.031983    2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 13:15:50.035975 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.031987    2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 13:15:50.035975 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.031994    2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 13:15:50.035975 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.032000    2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 13:15:50.035975 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.032004    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 13:15:50.035975 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.032010    2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 13:15:50.035975 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.032015    2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 13:15:50.035975 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.032019    2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 13:15:50.035975 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.032024    2573 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 13:15:50.035975 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.032028    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 13:15:50.035975 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.032032    2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 13:15:50.035975 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.032042    2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 13:15:50.035975 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.032046    2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 13:15:50.035975 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.032050    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 13:15:50.035975 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.032054    2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 13:15:50.035975 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.032058    2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 13:15:50.035975 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.032063    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 13:15:50.036437 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.032067    2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 13:15:50.036437 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.032071    2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 13:15:50.036437 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.032075    2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 13:15:50.036437 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.032080    2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 13:15:50.036437 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.032084    2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 13:15:50.036437 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.032089    2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 13:15:50.036437 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.032093    2573 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 13:15:50.036437 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.032103    2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 13:15:50.036437 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.032107    2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 13:15:50.036437 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.032112    2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 13:15:50.036437 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.032116    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 13:15:50.036437 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.032120    2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 13:15:50.036437 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.032125    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 13:15:50.036437 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.032129    2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 13:15:50.036437 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.032133    2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 13:15:50.036437 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.032138    2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 13:15:50.036437 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.032142    2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 13:15:50.036437 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.032147    2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 13:15:50.036437 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.032151    2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 13:15:50.036965 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.032155    2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 13:15:50.036965 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.032165    2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 13:15:50.036965 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.032169    2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 13:15:50.036965 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.032174    2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 13:15:50.036965 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.032178    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 13:15:50.036965 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.032182    2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 13:15:50.036965 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.032187    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 13:15:50.036965 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.032191    2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 13:15:50.036965 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.032195    2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 13:15:50.036965 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.032200    2573 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 13:15:50.036965 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.032203    2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 13:15:50.036965 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.032208    2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 13:15:50.036965 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.032212    2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 13:15:50.036965 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.032219    2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 13:15:50.036965 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.032228    2573 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 13:15:50.036965 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.032233    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 13:15:50.036965 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.032237    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 13:15:50.036965 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.032242    2573 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 13:15:50.036965 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.032247    2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 13:15:50.036965 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.032251    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 13:15:50.037465 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.032255    2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 13:15:50.037465 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.032259    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 13:15:50.037465 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.032264    2573 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 13:15:50.037465 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.032268    2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 13:15:50.037465 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.032274    2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 13:15:50.037465 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.032278    2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 13:15:50.037465 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:50.032287    2573 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 13:15:50.037465 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.032295    2573 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 08 13:15:50.037465 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.032690    2573 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 08 13:15:50.037465 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.034520    2573 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 08 13:15:50.037465 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.035651    2573 server.go:1019] "Starting client certificate rotation"
Sep 08 13:15:50.037465 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.035753    2573 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 08 13:15:50.037465 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.035781    2573 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 08 13:15:50.038470 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.038456    2573 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 08 13:15:50.039727 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.039707    2573 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 08 13:15:50.050343 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.050325    2573 log.go:25] "Validated CRI v1 runtime API"
Sep 08 13:15:50.053684 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.053670    2573 log.go:25] "Validated CRI v1 image API"
Sep 08 13:15:50.054532 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.054496    2573 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 08 13:15:50.057980 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.057956    2573 fs.go:135] Filesystem UUIDs: map[35c08b31-482c-4e53-adca-88ec35c6ed24:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 becfc3b2-6886-4dbe-8771-d90664c38122:/dev/nvme0n1p4]
Sep 08 13:15:50.058044 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.057979    2573 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 08 13:15:50.063395 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.063274    2573 manager.go:216] Machine: {Timestamp:2026-09-08 13:15:50.061834892 +0000 UTC m=+0.333282829 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3081006 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec29776967223f4f5a6d9081fdb2e5ae SystemUUID:ec297769-6722-3f4f-5a6d-9081fdb2e5ae BootID:b5562624-b64c-499d-8971-7585f85dcb73 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:dd:80:14:17:6b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:dd:80:14:17:6b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:f2:98:8f:ce:67:e8 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 08 13:15:50.063395 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.063387    2573 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 08 13:15:50.063543 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.063458    2573 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 08 13:15:50.066442 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.066420    2573 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 08 13:15:50.067043 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.067018    2573 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 08 13:15:50.067243 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.067047    2573 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-132-254.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 08 13:15:50.068607 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.068594    2573 topology_manager.go:138] "Creating topology manager with none policy"
Sep 08 13:15:50.068639 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.068611    2573 container_manager_linux.go:309] "Creating device plugin manager"
Sep 08 13:15:50.068639 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.068620    2573 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 08 13:15:50.069519 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.069509    2573 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 08 13:15:50.071430 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.071420    2573 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 08 13:15:50.076136 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.076123    2573 state_mem.go:36] "Initialized new in-memory state store"
Sep 08 13:15:50.076267 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.076257    2573 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 08 13:15:50.076302 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.076289    2573 kubelet.go:486] "Attempting to sync node with API server"
Sep 08 13:15:50.076330 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.076309    2573 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 08 13:15:50.076330 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.076323    2573 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 08 13:15:50.076387 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.076333    2573 kubelet.go:398] "Adding apiserver pod source"
Sep 08 13:15:50.076387 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.076342    2573 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 08 13:15:50.076757 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.076740    2573 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 08 13:15:50.076757 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.076759    2573 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 08 13:15:50.079321 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.079304    2573 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 08 13:15:50.083368 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.083350    2573 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 08 13:15:50.083429 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.083382    2573 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 08 13:15:50.084170 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.084157    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 08 13:15:50.084211 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.084178    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 08 13:15:50.084211 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.084183    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 08 13:15:50.084211 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.084188    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 08 13:15:50.084211 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.084195    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 08 13:15:50.084211 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.084200    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 08 13:15:50.084346 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.084217    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 08 13:15:50.084346 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.084222    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 08 13:15:50.084346 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.084229    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 08 13:15:50.084346 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.084234    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 08 13:15:50.084346 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.084242    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 08 13:15:50.084346 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.084250    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 08 13:15:50.087445 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.085097    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 08 13:15:50.087445 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.087427    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 08 13:15:50.091234 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.091221    2573 server.go:1268] "Started kubelet"
Sep 08 13:15:50.091447 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.091376    2573 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 08 13:15:50.091544 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.091474    2573 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 08 13:15:50.091685 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.091657    2573 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 08 13:15:50.092116 ip-10-0-132-254 systemd[1]: Started Kubernetes Kubelet.
Sep 08 13:15:50.093144 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.093120    2573 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 08 13:15:50.093757 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.093623    2573 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-254.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 08 13:15:50.093757 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.093723    2573 server.go:310] "Adding debug handlers to kubelet server"
Sep 08 13:15:50.094402 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:15:50.094380    2573 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-132-254.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 08 13:15:50.094477 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:15:50.094378    2573 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 08 13:15:50.095701 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.095659    2573 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-ccrjq"
Sep 08 13:15:50.095948 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:15:50.095929    2573 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 08 13:15:50.097847 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:15:50.096769    2573 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-254.ec2.internal.18d35ad5845933aa  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-254.ec2.internal,UID:ip-10-0-132-254.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-132-254.ec2.internal,},FirstTimestamp:2026-09-08 13:15:50.09119121 +0000 UTC m=+0.362639135,LastTimestamp:2026-09-08 13:15:50.09119121 +0000 UTC m=+0.362639135,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-254.ec2.internal,}"
Sep 08 13:15:50.099682 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.099665    2573 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 08 13:15:50.100402 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.100374    2573 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 08 13:15:50.100612 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:15:50.100590    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-254.ec2.internal\" not found"
Sep 08 13:15:50.101161 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.101127    2573 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 08 13:15:50.101161 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.101129    2573 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 08 13:15:50.101308 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.101187    2573 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 08 13:15:50.101355 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.101327    2573 reconstruct.go:97] "Volume reconstruction finished"
Sep 08 13:15:50.101355 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.101336    2573 reconciler.go:29] "Reconciler: start to sync state"
Sep 08 13:15:50.101355 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.101349    2573 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 08 13:15:50.101483 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.101362    2573 factory.go:55] Registering systemd factory
Sep 08 13:15:50.101483 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.101369    2573 factory.go:223] Registration of the systemd container factory successfully
Sep 08 13:15:50.101605 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.101583    2573 factory.go:153] Registering CRI-O factory
Sep 08 13:15:50.101605 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.101594    2573 factory.go:223] Registration of the crio container factory successfully
Sep 08 13:15:50.101703 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.101610    2573 factory.go:103] Registering Raw factory
Sep 08 13:15:50.101703 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.101622    2573 manager.go:1196] Started watching for new ooms in manager
Sep 08 13:15:50.101939 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.101927    2573 manager.go:349] Starting recovery of all containers
Sep 08 13:15:50.102158 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.102136    2573 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-ccrjq"
Sep 08 13:15:50.106732 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:15:50.106573    2573 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-254.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 08 13:15:50.106870 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:15:50.106849    2573 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 08 13:15:50.113620 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.113468    2573 manager.go:354] Recovery completed
Sep 08 13:15:50.118428 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.118408    2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 13:15:50.120739 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.120719    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-254.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 13:15:50.120831 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.120749    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-254.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 13:15:50.120831 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.120759    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-254.ec2.internal" event="NodeHasSufficientPID"
Sep 08 13:15:50.121382 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.121363    2573 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 08 13:15:50.121382 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.121375    2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 13:15:50.121382 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.121380    2573 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 08 13:15:50.121563 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.121399    2573 state_mem.go:36] "Initialized new in-memory state store"
Sep 08 13:15:50.122088 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.122072    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-254.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 13:15:50.122168 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.122101    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-254.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 13:15:50.122168 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.122111    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-254.ec2.internal" event="NodeHasSufficientPID"
Sep 08 13:15:50.123426 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.123415    2573 policy_none.go:49] "None policy: Start"
Sep 08 13:15:50.123469 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.123431    2573 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 08 13:15:50.123469 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.123441    2573 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 08 13:15:50.125443 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.125433    2573 policy_none.go:47] "Start"
Sep 08 13:15:50.161112 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.161097    2573 manager.go:337] "Starting Device Plugin manager"
Sep 08 13:15:50.177002 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:15:50.161129    2573 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 08 13:15:50.177002 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.161139    2573 server.go:87] "Starting device plugin registration server"
Sep 08 13:15:50.177002 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.161424    2573 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 08 13:15:50.177002 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.161443    2573 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 08 13:15:50.177002 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.161544    2573 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 08 13:15:50.177002 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.161612    2573 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 08 13:15:50.177002 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.161621    2573 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 08 13:15:50.177002 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:15:50.162149    2573 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 08 13:15:50.177002 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:15:50.162200    2573 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-254.ec2.internal\" not found"
Sep 08 13:15:50.177002 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.167770    2573 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 08 13:15:50.177002 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.168901    2573 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 08 13:15:50.177002 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.168941    2573 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 08 13:15:50.177002 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.168962    2573 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 08 13:15:50.177002 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:15:50.168996    2573 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 08 13:15:50.177002 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.171304    2573 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 13:15:50.262183 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.262117    2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 13:15:50.263083 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.263065    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-254.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 13:15:50.263160 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.263112    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-254.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 13:15:50.263160 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.263128    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-254.ec2.internal" event="NodeHasSufficientPID"
Sep 08 13:15:50.263160 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.263157    2573 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-254.ec2.internal"
Sep 08 13:15:50.269708 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.269684    2573 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-254.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-132-254.ec2.internal"]
Sep 08 13:15:50.269759 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.269750    2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 13:15:50.270892 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.270871    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-254.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 13:15:50.270976 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.270908    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-254.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 13:15:50.270976 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.270923    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-254.ec2.internal" event="NodeHasSufficientPID"
Sep 08 13:15:50.271781 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.271765    2573 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-132-254.ec2.internal"
Sep 08 13:15:50.271834 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:15:50.271793    2573 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-132-254.ec2.internal\": node \"ip-10-0-132-254.ec2.internal\" not found"
Sep 08 13:15:50.273267 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.273254    2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 13:15:50.273365 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.273351    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-254.ec2.internal"
Sep 08 13:15:50.273403 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.273379    2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 13:15:50.273963 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.273949    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-254.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 13:15:50.274030 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.273974    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-254.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 13:15:50.274030 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.273983    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-254.ec2.internal" event="NodeHasSufficientPID"
Sep 08 13:15:50.274093 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.273952    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-254.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 13:15:50.274093 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.274048    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-254.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 13:15:50.274093 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.274061    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-254.ec2.internal" event="NodeHasSufficientPID"
Sep 08 13:15:50.276130 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.276115    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-254.ec2.internal"
Sep 08 13:15:50.276237 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.276139    2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 13:15:50.276807 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.276793    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-254.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 13:15:50.276878 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.276816    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-254.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 13:15:50.276878 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.276827    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-254.ec2.internal" event="NodeHasSufficientPID"
Sep 08 13:15:50.290354 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:15:50.290333    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-254.ec2.internal\" not found"
Sep 08 13:15:50.301076 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:15:50.301062    2573 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-254.ec2.internal\" not found" node="ip-10-0-132-254.ec2.internal"
Sep 08 13:15:50.303033 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.303013    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6a0e339a5ac67687e98c7fee9b0ffa43-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-254.ec2.internal\" (UID: \"6a0e339a5ac67687e98c7fee9b0ffa43\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-254.ec2.internal"
Sep 08 13:15:50.303096 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.303042    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6a0e339a5ac67687e98c7fee9b0ffa43-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-254.ec2.internal\" (UID: \"6a0e339a5ac67687e98c7fee9b0ffa43\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-254.ec2.internal"
Sep 08 13:15:50.303096 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.303073    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7dfffcd72254a48b318c2c1ea59c531d-config\") pod \"kube-apiserver-proxy-ip-10-0-132-254.ec2.internal\" (UID: \"7dfffcd72254a48b318c2c1ea59c531d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-254.ec2.internal"
Sep 08 13:15:50.305349 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:15:50.305335    2573 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-254.ec2.internal\" not found" node="ip-10-0-132-254.ec2.internal"
Sep 08 13:15:50.391009 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:15:50.390989    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-254.ec2.internal\" not found"
Sep 08 13:15:50.403464 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.403446    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6a0e339a5ac67687e98c7fee9b0ffa43-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-254.ec2.internal\" (UID: \"6a0e339a5ac67687e98c7fee9b0ffa43\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-254.ec2.internal"
Sep 08 13:15:50.403549 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.403494    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6a0e339a5ac67687e98c7fee9b0ffa43-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-254.ec2.internal\" (UID: \"6a0e339a5ac67687e98c7fee9b0ffa43\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-254.ec2.internal"
Sep 08 13:15:50.403549 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.403530    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7dfffcd72254a48b318c2c1ea59c531d-config\") pod \"kube-apiserver-proxy-ip-10-0-132-254.ec2.internal\" (UID: \"7dfffcd72254a48b318c2c1ea59c531d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-254.ec2.internal"
Sep 08 13:15:50.403549 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.403533    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6a0e339a5ac67687e98c7fee9b0ffa43-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-254.ec2.internal\" (UID: \"6a0e339a5ac67687e98c7fee9b0ffa43\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-254.ec2.internal"
Sep 08 13:15:50.403675 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.403549    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7dfffcd72254a48b318c2c1ea59c531d-config\") pod \"kube-apiserver-proxy-ip-10-0-132-254.ec2.internal\" (UID: \"7dfffcd72254a48b318c2c1ea59c531d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-254.ec2.internal"
Sep 08 13:15:50.403675 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.403476    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6a0e339a5ac67687e98c7fee9b0ffa43-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-254.ec2.internal\" (UID: \"6a0e339a5ac67687e98c7fee9b0ffa43\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-254.ec2.internal"
Sep 08 13:15:50.491974 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:15:50.491955    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-254.ec2.internal\" not found"
Sep 08 13:15:50.592828 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:15:50.592758    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-254.ec2.internal\" not found"
Sep 08 13:15:50.605145 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.605111    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-254.ec2.internal"
Sep 08 13:15:50.609175 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:50.609158    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-254.ec2.internal"
Sep 08 13:15:50.693578 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:15:50.693546    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-254.ec2.internal\" not found"
Sep 08 13:15:50.794053 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:15:50.794023    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-254.ec2.internal\" not found"
Sep 08 13:15:50.894606 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:15:50.894523    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-254.ec2.internal\" not found"
Sep 08 13:15:50.995019 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:15:50.994977    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-254.ec2.internal\" not found"
Sep 08 13:15:51.005123 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:51.005095    2573 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 13:15:51.035386 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:51.035368    2573 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 08 13:15:51.035787 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:51.035478    2573 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 08 13:15:51.035787 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:51.035483    2573 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 08 13:15:51.066574 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:51.066552    2573 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 13:15:51.095775 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:15:51.095753    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-254.ec2.internal\" not found"
Sep 08 13:15:51.105558 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:51.105527    2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-07 13:10:50 +0000 UTC" deadline="2028-05-26 08:45:34.335821248 +0000 UTC"
Sep 08 13:15:51.105558 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:51.105556    2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15019h29m43.230268205s"
Sep 08 13:15:51.197883 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:15:51.197805    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-254.ec2.internal\" not found"
Sep 08 13:15:51.298738 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:15:51.298698    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-254.ec2.internal\" not found"
Sep 08 13:15:51.390835 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:51.390804    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dfffcd72254a48b318c2c1ea59c531d.slice/crio-7c9fd642b3246f006946109ab935a222be38f4f59395f6b0af7c369f8ed0b278 WatchSource:0}: Error finding container 7c9fd642b3246f006946109ab935a222be38f4f59395f6b0af7c369f8ed0b278: Status 404 returned error can't find the container with id 7c9fd642b3246f006946109ab935a222be38f4f59395f6b0af7c369f8ed0b278
Sep 08 13:15:51.395383 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:51.395362    2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 08 13:15:51.399238 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:15:51.399219    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-254.ec2.internal\" not found"
Sep 08 13:15:51.438777 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:51.438753    2573 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 13:15:51.501181 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:51.501132    2573 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-254.ec2.internal"
Sep 08 13:15:51.513222 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:51.513201    2573 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 08 13:15:51.514349 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:51.514334    2573 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-254.ec2.internal"
Sep 08 13:15:51.524000 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:51.523985    2573 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 08 13:15:52.076772 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.076739    2573 apiserver.go:52] "Watching apiserver"
Sep 08 13:15:52.087363 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.087335    2573 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 08 13:15:52.087755 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.087727    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-rlj5t","openshift-network-operator/iptables-alerter-kmf95","kube-system/global-pull-secret-syncer-6qqfz","openshift-cluster-node-tuning-operator/tuned-8ghvz","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-254.ec2.internal","openshift-multus/multus-additional-cni-plugins-krqr9","openshift-multus/network-metrics-daemon-kpscq","openshift-network-diagnostics/network-check-target-mwdn6","openshift-ovn-kubernetes/ovnkube-node-8h8cn","kube-system/konnectivity-agent-7wrhb","kube-system/kube-apiserver-proxy-ip-10-0-132-254.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hq2vz","openshift-dns/node-resolver-rbnnh","openshift-image-registry/node-ca-24f78"]
Sep 08 13:15:52.090435 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.090411    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kpscq"
Sep 08 13:15:52.090572 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:15:52.090501    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kpscq" podUID="217a8762-14bb-4990-8a53-3b3bcd6984ac"
Sep 08 13:15:52.092683 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.092659    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mwdn6"
Sep 08 13:15:52.092792 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:15:52.092735    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mwdn6" podUID="0be74f6e-cde6-42fd-8772-4fca7f6bab00"
Sep 08 13:15:52.094982 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.094954    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6qqfz"
Sep 08 13:15:52.095084 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:15:52.095029    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6qqfz" podUID="9e7df160-d688-4877-895e-edc5f73db16a"
Sep 08 13:15:52.098073 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.098044    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-8ghvz"
Sep 08 13:15:52.100498 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.100390    2573 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 08 13:15:52.100498 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.100474    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 08 13:15:52.100680 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.100529    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-5fg24\""
Sep 08 13:15:52.100680 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.100644    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-rlj5t"
Sep 08 13:15:52.100680 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.100481    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 08 13:15:52.102923 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.102900    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 08 13:15:52.103058 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.103039    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 08 13:15:52.103283 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.103266    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 08 13:15:52.103349 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.103297    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-krqr9"
Sep 08 13:15:52.103490 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.103472    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 08 13:15:52.103561 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.103505    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-g99sc\""
Sep 08 13:15:52.105311 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.105294    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-pbcsr\""
Sep 08 13:15:52.105415 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.105379    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 08 13:15:52.105642 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.105621    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-8h8cn"
Sep 08 13:15:52.106035 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.105755    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 08 13:15:52.107728 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.107708    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 08 13:15:52.108264 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.108247    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-rbnnh"
Sep 08 13:15:52.108865 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.108845    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-wgq8q\""
Sep 08 13:15:52.108968 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.108904    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 08 13:15:52.108968 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.108925    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 08 13:15:52.109099 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.108967    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 08 13:15:52.109099 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.108979    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 08 13:15:52.109099 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.108908    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 08 13:15:52.110517 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.110481    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 08 13:15:52.110772 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.110749    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-kmf95"
Sep 08 13:15:52.111385 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.111365    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-rlhvg\""
Sep 08 13:15:52.111478 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.111455    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 08 13:15:52.113195 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.113175    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-zf4jc\""
Sep 08 13:15:52.113289 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.113202    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 08 13:15:52.113289 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.113188    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 08 13:15:52.113289 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.113284    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 08 13:15:52.113591 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.113573    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-7wrhb"
Sep 08 13:15:52.114834 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.114807    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a88be331-067a-4b30-bdaa-3b36b7774d2f-host-var-lib-cni-bin\") pod \"multus-rlj5t\" (UID: \"a88be331-067a-4b30-bdaa-3b36b7774d2f\") " pod="openshift-multus/multus-rlj5t"
Sep 08 13:15:52.114963 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.114855    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a88be331-067a-4b30-bdaa-3b36b7774d2f-multus-conf-dir\") pod \"multus-rlj5t\" (UID: \"a88be331-067a-4b30-bdaa-3b36b7774d2f\") " pod="openshift-multus/multus-rlj5t"
Sep 08 13:15:52.114963 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.114894    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkzhj\" (UniqueName: \"kubernetes.io/projected/a88be331-067a-4b30-bdaa-3b36b7774d2f-kube-api-access-nkzhj\") pod \"multus-rlj5t\" (UID: \"a88be331-067a-4b30-bdaa-3b36b7774d2f\") " pod="openshift-multus/multus-rlj5t"
Sep 08 13:15:52.115080 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.114989    2573 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 08 13:15:52.115080 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.115037    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8e45750b-bf27-4d49-8eba-1ba76fadd028-cni-binary-copy\") pod \"multus-additional-cni-plugins-krqr9\" (UID: \"8e45750b-bf27-4d49-8eba-1ba76fadd028\") " pod="openshift-multus/multus-additional-cni-plugins-krqr9"
Sep 08 13:15:52.115080 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.115068    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqhq8\" (UniqueName: \"kubernetes.io/projected/8e45750b-bf27-4d49-8eba-1ba76fadd028-kube-api-access-xqhq8\") pod \"multus-additional-cni-plugins-krqr9\" (UID: \"8e45750b-bf27-4d49-8eba-1ba76fadd028\") " pod="openshift-multus/multus-additional-cni-plugins-krqr9"
Sep 08 13:15:52.115230 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.115101    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c6f2aedb-3459-4350-be14-4a64978cd32a-etc-sysctl-d\") pod \"tuned-8ghvz\" (UID: \"c6f2aedb-3459-4350-be14-4a64978cd32a\") " pod="openshift-cluster-node-tuning-operator/tuned-8ghvz"
Sep 08 13:15:52.115230 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.115131    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c6f2aedb-3459-4350-be14-4a64978cd32a-etc-systemd\") pod \"tuned-8ghvz\" (UID: \"c6f2aedb-3459-4350-be14-4a64978cd32a\") " pod="openshift-cluster-node-tuning-operator/tuned-8ghvz"
Sep 08 13:15:52.115230 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.115161    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c6f2aedb-3459-4350-be14-4a64978cd32a-tmp\") pod \"tuned-8ghvz\" (UID: \"c6f2aedb-3459-4350-be14-4a64978cd32a\") " pod="openshift-cluster-node-tuning-operator/tuned-8ghvz"
Sep 08 13:15:52.115399 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.115323    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a88be331-067a-4b30-bdaa-3b36b7774d2f-host-var-lib-cni-multus\") pod \"multus-rlj5t\" (UID: \"a88be331-067a-4b30-bdaa-3b36b7774d2f\") " pod="openshift-multus/multus-rlj5t"
Sep 08 13:15:52.115399 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.115350    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a88be331-067a-4b30-bdaa-3b36b7774d2f-host-run-multus-certs\") pod \"multus-rlj5t\" (UID: \"a88be331-067a-4b30-bdaa-3b36b7774d2f\") " pod="openshift-multus/multus-rlj5t"
Sep 08 13:15:52.115399 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.115384    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8e45750b-bf27-4d49-8eba-1ba76fadd028-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-krqr9\" (UID: \"8e45750b-bf27-4d49-8eba-1ba76fadd028\") " pod="openshift-multus/multus-additional-cni-plugins-krqr9"
Sep 08 13:15:52.115551 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.115417    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/217a8762-14bb-4990-8a53-3b3bcd6984ac-metrics-certs\") pod \"network-metrics-daemon-kpscq\" (UID: \"217a8762-14bb-4990-8a53-3b3bcd6984ac\") " pod="openshift-multus/network-metrics-daemon-kpscq"
Sep 08 13:15:52.115551 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.115448    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdzwv\" (UniqueName: \"kubernetes.io/projected/217a8762-14bb-4990-8a53-3b3bcd6984ac-kube-api-access-fdzwv\") pod \"network-metrics-daemon-kpscq\" (UID: \"217a8762-14bb-4990-8a53-3b3bcd6984ac\") " pod="openshift-multus/network-metrics-daemon-kpscq"
Sep 08 13:15:52.115551 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.115482    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c6f2aedb-3459-4350-be14-4a64978cd32a-etc-tuned\") pod \"tuned-8ghvz\" (UID: \"c6f2aedb-3459-4350-be14-4a64978cd32a\") " pod="openshift-cluster-node-tuning-operator/tuned-8ghvz"
Sep 08 13:15:52.115551 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.115548    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a88be331-067a-4b30-bdaa-3b36b7774d2f-host-var-lib-kubelet\") pod \"multus-rlj5t\" (UID: \"a88be331-067a-4b30-bdaa-3b36b7774d2f\") " pod="openshift-multus/multus-rlj5t"
Sep 08 13:15:52.115785 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.115643    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a88be331-067a-4b30-bdaa-3b36b7774d2f-cni-binary-copy\") pod \"multus-rlj5t\" (UID: \"a88be331-067a-4b30-bdaa-3b36b7774d2f\") " pod="openshift-multus/multus-rlj5t"
Sep 08 13:15:52.115870 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.115818    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a88be331-067a-4b30-bdaa-3b36b7774d2f-host-run-netns\") pod \"multus-rlj5t\" (UID: \"a88be331-067a-4b30-bdaa-3b36b7774d2f\") " pod="openshift-multus/multus-rlj5t"
Sep 08 13:15:52.115914 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.115868    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8e45750b-bf27-4d49-8eba-1ba76fadd028-tuning-conf-dir\") pod \"multus-additional-cni-plugins-krqr9\" (UID: \"8e45750b-bf27-4d49-8eba-1ba76fadd028\") " pod="openshift-multus/multus-additional-cni-plugins-krqr9"
Sep 08 13:15:52.115957 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.115926    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a88be331-067a-4b30-bdaa-3b36b7774d2f-multus-cni-dir\") pod \"multus-rlj5t\" (UID: \"a88be331-067a-4b30-bdaa-3b36b7774d2f\") " pod="openshift-multus/multus-rlj5t"
Sep 08 13:15:52.116001 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.115963    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a88be331-067a-4b30-bdaa-3b36b7774d2f-cnibin\") pod \"multus-rlj5t\" (UID: \"a88be331-067a-4b30-bdaa-3b36b7774d2f\") " pod="openshift-multus/multus-rlj5t"
Sep 08 13:15:52.116001 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.115993    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a88be331-067a-4b30-bdaa-3b36b7774d2f-multus-socket-dir-parent\") pod \"multus-rlj5t\" (UID: \"a88be331-067a-4b30-bdaa-3b36b7774d2f\") " pod="openshift-multus/multus-rlj5t"
Sep 08 13:15:52.116107 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.116026    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8e45750b-bf27-4d49-8eba-1ba76fadd028-system-cni-dir\") pod \"multus-additional-cni-plugins-krqr9\" (UID: \"8e45750b-bf27-4d49-8eba-1ba76fadd028\") " pod="openshift-multus/multus-additional-cni-plugins-krqr9"
Sep 08 13:15:52.116193 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.116106    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9e7df160-d688-4877-895e-edc5f73db16a-kubelet-config\") pod \"global-pull-secret-syncer-6qqfz\" (UID: \"9e7df160-d688-4877-895e-edc5f73db16a\") " pod="kube-system/global-pull-secret-syncer-6qqfz"
Sep 08 13:15:52.116193 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.116178    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9e7df160-d688-4877-895e-edc5f73db16a-original-pull-secret\") pod \"global-pull-secret-syncer-6qqfz\" (UID: \"9e7df160-d688-4877-895e-edc5f73db16a\") " pod="kube-system/global-pull-secret-syncer-6qqfz"
Sep 08 13:15:52.116641 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.116615    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-wqcfl\""
Sep 08 13:15:52.116763 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.116666    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 08 13:15:52.116819 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.116786    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c6f2aedb-3459-4350-be14-4a64978cd32a-etc-sysconfig\") pod \"tuned-8ghvz\" (UID: \"c6f2aedb-3459-4350-be14-4a64978cd32a\") " pod="openshift-cluster-node-tuning-operator/tuned-8ghvz"
Sep 08 13:15:52.116892 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.116872    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c6f2aedb-3459-4350-be14-4a64978cd32a-etc-kubernetes\") pod \"tuned-8ghvz\" (UID: \"c6f2aedb-3459-4350-be14-4a64978cd32a\") " pod="openshift-cluster-node-tuning-operator/tuned-8ghvz"
Sep 08 13:15:52.116951 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.116925    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 08 13:15:52.117004 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.116925    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c6f2aedb-3459-4350-be14-4a64978cd32a-var-lib-kubelet\") pod \"tuned-8ghvz\" (UID: \"c6f2aedb-3459-4350-be14-4a64978cd32a\") " pod="openshift-cluster-node-tuning-operator/tuned-8ghvz"
Sep 08 13:15:52.117064 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.117024    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fp4sq\" (UniqueName: \"kubernetes.io/projected/c6f2aedb-3459-4350-be14-4a64978cd32a-kube-api-access-fp4sq\") pod \"tuned-8ghvz\" (UID: \"c6f2aedb-3459-4350-be14-4a64978cd32a\") " pod="openshift-cluster-node-tuning-operator/tuned-8ghvz"
Sep 08 13:15:52.117120 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.117070    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a88be331-067a-4b30-bdaa-3b36b7774d2f-hostroot\") pod \"multus-rlj5t\" (UID: \"a88be331-067a-4b30-bdaa-3b36b7774d2f\") " pod="openshift-multus/multus-rlj5t"
Sep 08 13:15:52.117120 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.117105    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4trk\" (UniqueName: \"kubernetes.io/projected/0be74f6e-cde6-42fd-8772-4fca7f6bab00-kube-api-access-m4trk\") pod \"network-check-target-mwdn6\" (UID: \"0be74f6e-cde6-42fd-8772-4fca7f6bab00\") " pod="openshift-network-diagnostics/network-check-target-mwdn6"
Sep 08 13:15:52.117255 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.117139    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c6f2aedb-3459-4350-be14-4a64978cd32a-run\") pod \"tuned-8ghvz\" (UID: \"c6f2aedb-3459-4350-be14-4a64978cd32a\") " pod="openshift-cluster-node-tuning-operator/tuned-8ghvz"
Sep 08 13:15:52.117255 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.117201    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a88be331-067a-4b30-bdaa-3b36b7774d2f-system-cni-dir\") pod \"multus-rlj5t\" (UID: \"a88be331-067a-4b30-bdaa-3b36b7774d2f\") " pod="openshift-multus/multus-rlj5t"
Sep 08 13:15:52.117255 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.117246    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a88be331-067a-4b30-bdaa-3b36b7774d2f-multus-daemon-config\") pod \"multus-rlj5t\" (UID: \"a88be331-067a-4b30-bdaa-3b36b7774d2f\") " pod="openshift-multus/multus-rlj5t"
Sep 08 13:15:52.117425 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.117282    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a88be331-067a-4b30-bdaa-3b36b7774d2f-etc-kubernetes\") pod \"multus-rlj5t\" (UID: \"a88be331-067a-4b30-bdaa-3b36b7774d2f\") " pod="openshift-multus/multus-rlj5t"
Sep 08 13:15:52.117425 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.117348    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8e45750b-bf27-4d49-8eba-1ba76fadd028-cnibin\") pod \"multus-additional-cni-plugins-krqr9\" (UID: \"8e45750b-bf27-4d49-8eba-1ba76fadd028\") " pod="openshift-multus/multus-additional-cni-plugins-krqr9"
Sep 08 13:15:52.117537 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.117397    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8e45750b-bf27-4d49-8eba-1ba76fadd028-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-krqr9\" (UID: \"8e45750b-bf27-4d49-8eba-1ba76fadd028\") " pod="openshift-multus/multus-additional-cni-plugins-krqr9"
Sep 08 13:15:52.117537 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.117476    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9e7df160-d688-4877-895e-edc5f73db16a-dbus\") pod \"global-pull-secret-syncer-6qqfz\" (UID: \"9e7df160-d688-4877-895e-edc5f73db16a\") " pod="kube-system/global-pull-secret-syncer-6qqfz"
Sep 08 13:15:52.117537 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.117523    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c6f2aedb-3459-4350-be14-4a64978cd32a-etc-sysctl-conf\") pod \"tuned-8ghvz\" (UID: \"c6f2aedb-3459-4350-be14-4a64978cd32a\") " pod="openshift-cluster-node-tuning-operator/tuned-8ghvz"
Sep 08 13:15:52.117696 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.117643    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c6f2aedb-3459-4350-be14-4a64978cd32a-lib-modules\") pod \"tuned-8ghvz\" (UID: \"c6f2aedb-3459-4350-be14-4a64978cd32a\") " pod="openshift-cluster-node-tuning-operator/tuned-8ghvz"
Sep 08 13:15:52.117696 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.117674    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c6f2aedb-3459-4350-be14-4a64978cd32a-host\") pod \"tuned-8ghvz\" (UID: \"c6f2aedb-3459-4350-be14-4a64978cd32a\") " pod="openshift-cluster-node-tuning-operator/tuned-8ghvz"
Sep 08 13:15:52.117820 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.117719    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a88be331-067a-4b30-bdaa-3b36b7774d2f-host-run-k8s-cni-cncf-io\") pod \"multus-rlj5t\" (UID: \"a88be331-067a-4b30-bdaa-3b36b7774d2f\") " pod="openshift-multus/multus-rlj5t"
Sep 08 13:15:52.117820 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.117756    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c6f2aedb-3459-4350-be14-4a64978cd32a-etc-modprobe-d\") pod \"tuned-8ghvz\" (UID: \"c6f2aedb-3459-4350-be14-4a64978cd32a\") " pod="openshift-cluster-node-tuning-operator/tuned-8ghvz"
Sep 08 13:15:52.117914 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.117818    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c6f2aedb-3459-4350-be14-4a64978cd32a-sys\") pod \"tuned-8ghvz\" (UID: \"c6f2aedb-3459-4350-be14-4a64978cd32a\") " pod="openshift-cluster-node-tuning-operator/tuned-8ghvz"
Sep 08 13:15:52.118445 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.118426    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-24f78"
Sep 08 13:15:52.120856 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.120831    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 08 13:15:52.120942 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.120876    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 08 13:15:52.121094 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.121072    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 08 13:15:52.121198 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.121125    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-9db9d\""
Sep 08 13:15:52.121412 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.121395    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hq2vz"
Sep 08 13:15:52.123948 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.123921    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 08 13:15:52.124051 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.123933    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-h2ngk\""
Sep 08 13:15:52.124051 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.123995    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 08 13:15:52.124051 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.124009    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 08 13:15:52.130273 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:52.130250    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a0e339a5ac67687e98c7fee9b0ffa43.slice/crio-0178a42c7017d173412cc5fefde1fb5e939b498d8129a66b2c17d1c5775fcb95 WatchSource:0}: Error finding container 0178a42c7017d173412cc5fefde1fb5e939b498d8129a66b2c17d1c5775fcb95: Status 404 returned error can't find the container with id 0178a42c7017d173412cc5fefde1fb5e939b498d8129a66b2c17d1c5775fcb95
Sep 08 13:15:52.140201 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.140179    2573 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-kkzzl"
Sep 08 13:15:52.148552 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.148531    2573 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-kkzzl"
Sep 08 13:15:52.174504 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.174457    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-254.ec2.internal" event={"ID":"6a0e339a5ac67687e98c7fee9b0ffa43","Type":"ContainerStarted","Data":"0178a42c7017d173412cc5fefde1fb5e939b498d8129a66b2c17d1c5775fcb95"}
Sep 08 13:15:52.177512 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.177483    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-254.ec2.internal" event={"ID":"7dfffcd72254a48b318c2c1ea59c531d","Type":"ContainerStarted","Data":"7c9fd642b3246f006946109ab935a222be38f4f59395f6b0af7c369f8ed0b278"}
Sep 08 13:15:52.202699 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.202670    2573 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 08 13:15:52.218383 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.218355    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c6f2aedb-3459-4350-be14-4a64978cd32a-etc-sysctl-d\") pod \"tuned-8ghvz\" (UID: \"c6f2aedb-3459-4350-be14-4a64978cd32a\") " pod="openshift-cluster-node-tuning-operator/tuned-8ghvz"
Sep 08 13:15:52.218501 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.218395    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c6f2aedb-3459-4350-be14-4a64978cd32a-etc-systemd\") pod \"tuned-8ghvz\" (UID: \"c6f2aedb-3459-4350-be14-4a64978cd32a\") " pod="openshift-cluster-node-tuning-operator/tuned-8ghvz"
Sep 08 13:15:52.218501 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.218420    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c6f2aedb-3459-4350-be14-4a64978cd32a-tmp\") pod \"tuned-8ghvz\" (UID: \"c6f2aedb-3459-4350-be14-4a64978cd32a\") " pod="openshift-cluster-node-tuning-operator/tuned-8ghvz"
Sep 08 13:15:52.218658 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.218532    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c6f2aedb-3459-4350-be14-4a64978cd32a-etc-systemd\") pod \"tuned-8ghvz\" (UID: \"c6f2aedb-3459-4350-be14-4a64978cd32a\") " pod="openshift-cluster-node-tuning-operator/tuned-8ghvz"
Sep 08 13:15:52.218658 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.218566    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a88be331-067a-4b30-bdaa-3b36b7774d2f-host-var-lib-cni-multus\") pod \"multus-rlj5t\" (UID: \"a88be331-067a-4b30-bdaa-3b36b7774d2f\") " pod="openshift-multus/multus-rlj5t"
Sep 08 13:15:52.218658 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.218582    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c6f2aedb-3459-4350-be14-4a64978cd32a-etc-sysctl-d\") pod \"tuned-8ghvz\" (UID: \"c6f2aedb-3459-4350-be14-4a64978cd32a\") " pod="openshift-cluster-node-tuning-operator/tuned-8ghvz"
Sep 08 13:15:52.218658 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.218616    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8e45750b-bf27-4d49-8eba-1ba76fadd028-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-krqr9\" (UID: \"8e45750b-bf27-4d49-8eba-1ba76fadd028\") " pod="openshift-multus/multus-additional-cni-plugins-krqr9"
Sep 08 13:15:52.218658 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.218644    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a88be331-067a-4b30-bdaa-3b36b7774d2f-host-var-lib-cni-multus\") pod \"multus-rlj5t\" (UID: \"a88be331-067a-4b30-bdaa-3b36b7774d2f\") " pod="openshift-multus/multus-rlj5t"
Sep 08 13:15:52.218906 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.218655    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b936238e-bdf1-4dac-8bf8-6417c313597c-etc-openvswitch\") pod \"ovnkube-node-8h8cn\" (UID: \"b936238e-bdf1-4dac-8bf8-6417c313597c\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h8cn"
Sep 08 13:15:52.218906 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.218697    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b936238e-bdf1-4dac-8bf8-6417c313597c-log-socket\") pod \"ovnkube-node-8h8cn\" (UID: \"b936238e-bdf1-4dac-8bf8-6417c313597c\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h8cn"
Sep 08 13:15:52.218906 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.218728    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c6f2aedb-3459-4350-be14-4a64978cd32a-etc-tuned\") pod \"tuned-8ghvz\" (UID: \"c6f2aedb-3459-4350-be14-4a64978cd32a\") " pod="openshift-cluster-node-tuning-operator/tuned-8ghvz"
Sep 08 13:15:52.218906 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.218760    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a88be331-067a-4b30-bdaa-3b36b7774d2f-host-var-lib-kubelet\") pod \"multus-rlj5t\" (UID: \"a88be331-067a-4b30-bdaa-3b36b7774d2f\") " pod="openshift-multus/multus-rlj5t"
Sep 08 13:15:52.218906 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.218779    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/28b8b2ef-528a-4802-816c-c3c1de322861-device-dir\") pod \"aws-ebs-csi-driver-node-hq2vz\" (UID: \"28b8b2ef-528a-4802-816c-c3c1de322861\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hq2vz"
Sep 08 13:15:52.218906 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.218797    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/28b8b2ef-528a-4802-816c-c3c1de322861-sys-fs\") pod \"aws-ebs-csi-driver-node-hq2vz\" (UID: \"28b8b2ef-528a-4802-816c-c3c1de322861\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hq2vz"
Sep 08 13:15:52.218906 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.218821    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a88be331-067a-4b30-bdaa-3b36b7774d2f-cni-binary-copy\") pod \"multus-rlj5t\" (UID: \"a88be331-067a-4b30-bdaa-3b36b7774d2f\") " pod="openshift-multus/multus-rlj5t"
Sep 08 13:15:52.218906 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.218849    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a88be331-067a-4b30-bdaa-3b36b7774d2f-host-run-netns\") pod \"multus-rlj5t\" (UID: \"a88be331-067a-4b30-bdaa-3b36b7774d2f\") " pod="openshift-multus/multus-rlj5t"
Sep 08 13:15:52.218906 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.218876    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a88be331-067a-4b30-bdaa-3b36b7774d2f-host-var-lib-kubelet\") pod \"multus-rlj5t\" (UID: \"a88be331-067a-4b30-bdaa-3b36b7774d2f\") " pod="openshift-multus/multus-rlj5t"
Sep 08 13:15:52.219422 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.218898    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b936238e-bdf1-4dac-8bf8-6417c313597c-host-slash\") pod \"ovnkube-node-8h8cn\" (UID: \"b936238e-bdf1-4dac-8bf8-6417c313597c\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h8cn"
Sep 08 13:15:52.219422 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.218935    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b936238e-bdf1-4dac-8bf8-6417c313597c-host-cni-netd\") pod \"ovnkube-node-8h8cn\" (UID: \"b936238e-bdf1-4dac-8bf8-6417c313597c\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h8cn"
Sep 08 13:15:52.219422 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.218950    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/137be755-3092-45ba-a6d2-8a025c5e6226-serviceca\") pod \"node-ca-24f78\" (UID: \"137be755-3092-45ba-a6d2-8a025c5e6226\") " pod="openshift-image-registry/node-ca-24f78"
Sep 08 13:15:52.219422 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.218968    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a88be331-067a-4b30-bdaa-3b36b7774d2f-multus-socket-dir-parent\") pod \"multus-rlj5t\" (UID: \"a88be331-067a-4b30-bdaa-3b36b7774d2f\") " pod="openshift-multus/multus-rlj5t"
Sep 08 13:15:52.219422 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.218991    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8e45750b-bf27-4d49-8eba-1ba76fadd028-system-cni-dir\") pod \"multus-additional-cni-plugins-krqr9\" (UID: \"8e45750b-bf27-4d49-8eba-1ba76fadd028\") " pod="openshift-multus/multus-additional-cni-plugins-krqr9"
Sep 08 13:15:52.219422 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.219012    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/28b8b2ef-528a-4802-816c-c3c1de322861-etc-selinux\") pod \"aws-ebs-csi-driver-node-hq2vz\" (UID: \"28b8b2ef-528a-4802-816c-c3c1de322861\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hq2vz"
Sep 08 13:15:52.219422 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.219031    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c6f2aedb-3459-4350-be14-4a64978cd32a-etc-kubernetes\") pod \"tuned-8ghvz\" (UID: \"c6f2aedb-3459-4350-be14-4a64978cd32a\") " pod="openshift-cluster-node-tuning-operator/tuned-8ghvz"
Sep 08 13:15:52.219422 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.219046    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c6f2aedb-3459-4350-be14-4a64978cd32a-var-lib-kubelet\") pod \"tuned-8ghvz\" (UID: \"c6f2aedb-3459-4350-be14-4a64978cd32a\") " pod="openshift-cluster-node-tuning-operator/tuned-8ghvz"
Sep 08 13:15:52.219422 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.219077    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fp4sq\" (UniqueName: \"kubernetes.io/projected/c6f2aedb-3459-4350-be14-4a64978cd32a-kube-api-access-fp4sq\") pod \"tuned-8ghvz\" (UID: \"c6f2aedb-3459-4350-be14-4a64978cd32a\") " pod="openshift-cluster-node-tuning-operator/tuned-8ghvz"
Sep 08 13:15:52.219422 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.219119    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4trk\" (UniqueName: \"kubernetes.io/projected/0be74f6e-cde6-42fd-8772-4fca7f6bab00-kube-api-access-m4trk\") pod \"network-check-target-mwdn6\" (UID: \"0be74f6e-cde6-42fd-8772-4fca7f6bab00\") " pod="openshift-network-diagnostics/network-check-target-mwdn6"
Sep 08 13:15:52.219422 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.219138    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b936238e-bdf1-4dac-8bf8-6417c313597c-run-ovn\") pod \"ovnkube-node-8h8cn\" (UID: \"b936238e-bdf1-4dac-8bf8-6417c313597c\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h8cn"
Sep 08 13:15:52.219422 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.219157    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5v5g\" (UniqueName: \"kubernetes.io/projected/b936238e-bdf1-4dac-8bf8-6417c313597c-kube-api-access-t5v5g\") pod \"ovnkube-node-8h8cn\" (UID: \"b936238e-bdf1-4dac-8bf8-6417c313597c\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h8cn"
Sep 08 13:15:52.219422 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.219184    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c6f2aedb-3459-4350-be14-4a64978cd32a-run\") pod \"tuned-8ghvz\" (UID: \"c6f2aedb-3459-4350-be14-4a64978cd32a\") " pod="openshift-cluster-node-tuning-operator/tuned-8ghvz"
Sep 08 13:15:52.219422 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.219203    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a88be331-067a-4b30-bdaa-3b36b7774d2f-multus-daemon-config\") pod \"multus-rlj5t\" (UID: \"a88be331-067a-4b30-bdaa-3b36b7774d2f\") " pod="openshift-multus/multus-rlj5t"
Sep 08 13:15:52.219422 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.219219    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a88be331-067a-4b30-bdaa-3b36b7774d2f-etc-kubernetes\") pod \"multus-rlj5t\" (UID: \"a88be331-067a-4b30-bdaa-3b36b7774d2f\") " pod="openshift-multus/multus-rlj5t"
Sep 08 13:15:52.219422 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.219234    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8e45750b-bf27-4d49-8eba-1ba76fadd028-cnibin\") pod \"multus-additional-cni-plugins-krqr9\" (UID: \"8e45750b-bf27-4d49-8eba-1ba76fadd028\") " pod="openshift-multus/multus-additional-cni-plugins-krqr9"
Sep 08 13:15:52.219422 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.219254    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8e45750b-bf27-4d49-8eba-1ba76fadd028-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-krqr9\" (UID: \"8e45750b-bf27-4d49-8eba-1ba76fadd028\") " pod="openshift-multus/multus-additional-cni-plugins-krqr9"
Sep 08 13:15:52.220327 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.219277    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/28b8b2ef-528a-4802-816c-c3c1de322861-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hq2vz\" (UID: \"28b8b2ef-528a-4802-816c-c3c1de322861\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hq2vz"
Sep 08 13:15:52.220327 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.219292    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8e45750b-bf27-4d49-8eba-1ba76fadd028-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-krqr9\" (UID: \"8e45750b-bf27-4d49-8eba-1ba76fadd028\") " pod="openshift-multus/multus-additional-cni-plugins-krqr9"
Sep 08 13:15:52.220327 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.219294    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c6f2aedb-3459-4350-be14-4a64978cd32a-host\") pod \"tuned-8ghvz\" (UID: \"c6f2aedb-3459-4350-be14-4a64978cd32a\") " pod="openshift-cluster-node-tuning-operator/tuned-8ghvz"
Sep 08 13:15:52.220327 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.219323    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a88be331-067a-4b30-bdaa-3b36b7774d2f-host-run-k8s-cni-cncf-io\") pod \"multus-rlj5t\" (UID: \"a88be331-067a-4b30-bdaa-3b36b7774d2f\") " pod="openshift-multus/multus-rlj5t"
Sep 08 13:15:52.220327 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.219346    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b936238e-bdf1-4dac-8bf8-6417c313597c-host-run-netns\") pod \"ovnkube-node-8h8cn\" (UID: \"b936238e-bdf1-4dac-8bf8-6417c313597c\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h8cn"
Sep 08 13:15:52.220327 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.219411    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a88be331-067a-4b30-bdaa-3b36b7774d2f-host-run-netns\") pod \"multus-rlj5t\" (UID: \"a88be331-067a-4b30-bdaa-3b36b7774d2f\") " pod="openshift-multus/multus-rlj5t"
Sep 08 13:15:52.220327 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.219441    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c6f2aedb-3459-4350-be14-4a64978cd32a-run\") pod \"tuned-8ghvz\" (UID: \"c6f2aedb-3459-4350-be14-4a64978cd32a\") " pod="openshift-cluster-node-tuning-operator/tuned-8ghvz"
Sep 08 13:15:52.220327 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.219457    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/42aa8a5e-3aba-4581-9f95-f3b978c250f0-hosts-file\") pod \"node-resolver-rbnnh\" (UID: \"42aa8a5e-3aba-4581-9f95-f3b978c250f0\") " pod="openshift-dns/node-resolver-rbnnh"
Sep 08 13:15:52.220327 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.219492    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42aa8a5e-3aba-4581-9f95-f3b978c250f0-tmp-dir\") pod \"node-resolver-rbnnh\" (UID: \"42aa8a5e-3aba-4581-9f95-f3b978c250f0\") " pod="openshift-dns/node-resolver-rbnnh"
Sep 08 13:15:52.220327 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.219523    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a88be331-067a-4b30-bdaa-3b36b7774d2f-multus-conf-dir\") pod \"multus-rlj5t\" (UID: \"a88be331-067a-4b30-bdaa-3b36b7774d2f\") " pod="openshift-multus/multus-rlj5t"
Sep 08 13:15:52.220327 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.219552    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkzhj\" (UniqueName: \"kubernetes.io/projected/a88be331-067a-4b30-bdaa-3b36b7774d2f-kube-api-access-nkzhj\") pod \"multus-rlj5t\" (UID: \"a88be331-067a-4b30-bdaa-3b36b7774d2f\") " pod="openshift-multus/multus-rlj5t"
Sep 08 13:15:52.220327 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.219596    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/217a8762-14bb-4990-8a53-3b3bcd6984ac-metrics-certs\") pod \"network-metrics-daemon-kpscq\" (UID: \"217a8762-14bb-4990-8a53-3b3bcd6984ac\") " pod="openshift-multus/network-metrics-daemon-kpscq"
Sep 08 13:15:52.220327 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:15:52.219698    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 13:15:52.220327 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.219708    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a88be331-067a-4b30-bdaa-3b36b7774d2f-multus-socket-dir-parent\") pod \"multus-rlj5t\" (UID: \"a88be331-067a-4b30-bdaa-3b36b7774d2f\") " pod="openshift-multus/multus-rlj5t"
Sep 08 13:15:52.220327 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.219733    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8e45750b-bf27-4d49-8eba-1ba76fadd028-system-cni-dir\") pod \"multus-additional-cni-plugins-krqr9\" (UID: \"8e45750b-bf27-4d49-8eba-1ba76fadd028\") " pod="openshift-multus/multus-additional-cni-plugins-krqr9"
Sep 08 13:15:52.220327 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.219745    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdzwv\" (UniqueName: \"kubernetes.io/projected/217a8762-14bb-4990-8a53-3b3bcd6984ac-kube-api-access-fdzwv\") pod \"network-metrics-daemon-kpscq\" (UID: \"217a8762-14bb-4990-8a53-3b3bcd6984ac\") " pod="openshift-multus/network-metrics-daemon-kpscq"
Sep 08 13:15:52.220327 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:15:52.219787    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/217a8762-14bb-4990-8a53-3b3bcd6984ac-metrics-certs podName:217a8762-14bb-4990-8a53-3b3bcd6984ac nodeName:}" failed. No retries permitted until 2026-09-08 13:15:52.719746125 +0000 UTC m=+2.991194042 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/217a8762-14bb-4990-8a53-3b3bcd6984ac-metrics-certs") pod "network-metrics-daemon-kpscq" (UID: "217a8762-14bb-4990-8a53-3b3bcd6984ac") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 13:15:52.221252 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.219844    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c6f2aedb-3459-4350-be14-4a64978cd32a-etc-kubernetes\") pod \"tuned-8ghvz\" (UID: \"c6f2aedb-3459-4350-be14-4a64978cd32a\") " pod="openshift-cluster-node-tuning-operator/tuned-8ghvz"
Sep 08 13:15:52.221252 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.219936    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b936238e-bdf1-4dac-8bf8-6417c313597c-systemd-units\") pod \"ovnkube-node-8h8cn\" (UID: \"b936238e-bdf1-4dac-8bf8-6417c313597c\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h8cn"
Sep 08 13:15:52.221252 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.219971    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b936238e-bdf1-4dac-8bf8-6417c313597c-var-lib-openvswitch\") pod \"ovnkube-node-8h8cn\" (UID: \"b936238e-bdf1-4dac-8bf8-6417c313597c\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h8cn"
Sep 08 13:15:52.221252 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.219997    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b936238e-bdf1-4dac-8bf8-6417c313597c-host-cni-bin\") pod \"ovnkube-node-8h8cn\" (UID: \"b936238e-bdf1-4dac-8bf8-6417c313597c\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h8cn"
Sep 08 13:15:52.221252 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.220025    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a88be331-067a-4b30-bdaa-3b36b7774d2f-host-run-multus-certs\") pod \"multus-rlj5t\" (UID: \"a88be331-067a-4b30-bdaa-3b36b7774d2f\") " pod="openshift-multus/multus-rlj5t"
Sep 08 13:15:52.221252 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.220050    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9c39b80f-3adc-45fd-b1ec-39df3c237b02-agent-certs\") pod \"konnectivity-agent-7wrhb\" (UID: \"9c39b80f-3adc-45fd-b1ec-39df3c237b02\") " pod="kube-system/konnectivity-agent-7wrhb"
Sep 08 13:15:52.221252 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.220076    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9c39b80f-3adc-45fd-b1ec-39df3c237b02-konnectivity-ca\") pod \"konnectivity-agent-7wrhb\" (UID: \"9c39b80f-3adc-45fd-b1ec-39df3c237b02\") " pod="kube-system/konnectivity-agent-7wrhb"
Sep 08 13:15:52.221252 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.220087    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a88be331-067a-4b30-bdaa-3b36b7774d2f-multus-conf-dir\") pod \"multus-rlj5t\" (UID: \"a88be331-067a-4b30-bdaa-3b36b7774d2f\") " pod="openshift-multus/multus-rlj5t"
Sep 08 13:15:52.221252 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.220155    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b936238e-bdf1-4dac-8bf8-6417c313597c-node-log\") pod \"ovnkube-node-8h8cn\" (UID: \"b936238e-bdf1-4dac-8bf8-6417c313597c\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h8cn"
Sep 08 13:15:52.221252 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.220162    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a88be331-067a-4b30-bdaa-3b36b7774d2f-cni-binary-copy\") pod \"multus-rlj5t\" (UID: \"a88be331-067a-4b30-bdaa-3b36b7774d2f\") " pod="openshift-multus/multus-rlj5t"
Sep 08 13:15:52.221252 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.220180    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/137be755-3092-45ba-a6d2-8a025c5e6226-host\") pod \"node-ca-24f78\" (UID: \"137be755-3092-45ba-a6d2-8a025c5e6226\") " pod="openshift-image-registry/node-ca-24f78"
Sep 08 13:15:52.221252 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.220182    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a88be331-067a-4b30-bdaa-3b36b7774d2f-etc-kubernetes\") pod \"multus-rlj5t\" (UID: \"a88be331-067a-4b30-bdaa-3b36b7774d2f\") " pod="openshift-multus/multus-rlj5t"
Sep 08 13:15:52.221252 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.220216    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a88be331-067a-4b30-bdaa-3b36b7774d2f-host-run-k8s-cni-cncf-io\") pod \"multus-rlj5t\" (UID: \"a88be331-067a-4b30-bdaa-3b36b7774d2f\") " pod="openshift-multus/multus-rlj5t"
Sep 08 13:15:52.221252 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.220227    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8e45750b-bf27-4d49-8eba-1ba76fadd028-tuning-conf-dir\") pod \"multus-additional-cni-plugins-krqr9\" (UID: \"8e45750b-bf27-4d49-8eba-1ba76fadd028\") " pod="openshift-multus/multus-additional-cni-plugins-krqr9"
Sep 08 13:15:52.221252 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.220247    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c6f2aedb-3459-4350-be14-4a64978cd32a-var-lib-kubelet\") pod \"tuned-8ghvz\" (UID: \"c6f2aedb-3459-4350-be14-4a64978cd32a\") " pod="openshift-cluster-node-tuning-operator/tuned-8ghvz"
Sep 08 13:15:52.221252 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.220255    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b936238e-bdf1-4dac-8bf8-6417c313597c-host-kubelet\") pod \"ovnkube-node-8h8cn\" (UID: \"b936238e-bdf1-4dac-8bf8-6417c313597c\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h8cn"
Sep 08 13:15:52.221252 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.220283    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b936238e-bdf1-4dac-8bf8-6417c313597c-ovn-node-metrics-cert\") pod \"ovnkube-node-8h8cn\" (UID: \"b936238e-bdf1-4dac-8bf8-6417c313597c\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h8cn"
Sep 08 13:15:52.222153 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.220258    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a88be331-067a-4b30-bdaa-3b36b7774d2f-host-run-multus-certs\") pod \"multus-rlj5t\" (UID: \"a88be331-067a-4b30-bdaa-3b36b7774d2f\") " pod="openshift-multus/multus-rlj5t"
Sep 08 13:15:52.222153 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.220287    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8e45750b-bf27-4d49-8eba-1ba76fadd028-cnibin\") pod \"multus-additional-cni-plugins-krqr9\" (UID: \"8e45750b-bf27-4d49-8eba-1ba76fadd028\") " pod="openshift-multus/multus-additional-cni-plugins-krqr9"
Sep 08 13:15:52.222153 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.220308    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c6f2aedb-3459-4350-be14-4a64978cd32a-host\") pod \"tuned-8ghvz\" (UID: \"c6f2aedb-3459-4350-be14-4a64978cd32a\") " pod="openshift-cluster-node-tuning-operator/tuned-8ghvz"
Sep 08 13:15:52.222153 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.220376    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a88be331-067a-4b30-bdaa-3b36b7774d2f-multus-cni-dir\") pod \"multus-rlj5t\" (UID: \"a88be331-067a-4b30-bdaa-3b36b7774d2f\") " pod="openshift-multus/multus-rlj5t"
Sep 08 13:15:52.222153 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.220420    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a88be331-067a-4b30-bdaa-3b36b7774d2f-cnibin\") pod \"multus-rlj5t\" (UID: \"a88be331-067a-4b30-bdaa-3b36b7774d2f\") " pod="openshift-multus/multus-rlj5t"
Sep 08 13:15:52.222153 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.220432    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a88be331-067a-4b30-bdaa-3b36b7774d2f-multus-cni-dir\") pod \"multus-rlj5t\" (UID: \"a88be331-067a-4b30-bdaa-3b36b7774d2f\") " pod="openshift-multus/multus-rlj5t"
Sep 08 13:15:52.222153 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.220460    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9e7df160-d688-4877-895e-edc5f73db16a-kubelet-config\") pod \"global-pull-secret-syncer-6qqfz\" (UID: \"9e7df160-d688-4877-895e-edc5f73db16a\") " pod="kube-system/global-pull-secret-syncer-6qqfz"
Sep 08 13:15:52.222153 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.220466    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8e45750b-bf27-4d49-8eba-1ba76fadd028-tuning-conf-dir\") pod \"multus-additional-cni-plugins-krqr9\" (UID: \"8e45750b-bf27-4d49-8eba-1ba76fadd028\") " pod="openshift-multus/multus-additional-cni-plugins-krqr9"
Sep 08 13:15:52.222153 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.220507    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a88be331-067a-4b30-bdaa-3b36b7774d2f-cnibin\") pod \"multus-rlj5t\" (UID: \"a88be331-067a-4b30-bdaa-3b36b7774d2f\") " pod="openshift-multus/multus-rlj5t"
Sep 08 13:15:52.222153 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.220555    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9e7df160-d688-4877-895e-edc5f73db16a-original-pull-secret\") pod \"global-pull-secret-syncer-6qqfz\" (UID: \"9e7df160-d688-4877-895e-edc5f73db16a\") " pod="kube-system/global-pull-secret-syncer-6qqfz"
Sep 08 13:15:52.222153 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.220573    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9e7df160-d688-4877-895e-edc5f73db16a-kubelet-config\") pod \"global-pull-secret-syncer-6qqfz\" (UID: \"9e7df160-d688-4877-895e-edc5f73db16a\") " pod="kube-system/global-pull-secret-syncer-6qqfz"
Sep 08 13:15:52.222153 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.220608    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c6f2aedb-3459-4350-be14-4a64978cd32a-etc-sysconfig\") pod \"tuned-8ghvz\" (UID: \"c6f2aedb-3459-4350-be14-4a64978cd32a\") " pod="openshift-cluster-node-tuning-operator/tuned-8ghvz"
Sep 08 13:15:52.222153 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.220630    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8e45750b-bf27-4d49-8eba-1ba76fadd028-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-krqr9\" (UID: \"8e45750b-bf27-4d49-8eba-1ba76fadd028\") " pod="openshift-multus/multus-additional-cni-plugins-krqr9"
Sep 08 13:15:52.222153 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:15:52.220668    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 13:15:52.222153 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.220719    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c6f2aedb-3459-4350-be14-4a64978cd32a-etc-sysconfig\") pod \"tuned-8ghvz\" (UID: \"c6f2aedb-3459-4350-be14-4a64978cd32a\") " pod="openshift-cluster-node-tuning-operator/tuned-8ghvz"
Sep 08 13:15:52.222153 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.220723    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a88be331-067a-4b30-bdaa-3b36b7774d2f-hostroot\") pod \"multus-rlj5t\" (UID: \"a88be331-067a-4b30-bdaa-3b36b7774d2f\") " pod="openshift-multus/multus-rlj5t"
Sep 08 13:15:52.222153 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.220749    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a88be331-067a-4b30-bdaa-3b36b7774d2f-multus-daemon-config\") pod \"multus-rlj5t\" (UID: \"a88be331-067a-4b30-bdaa-3b36b7774d2f\") " pod="openshift-multus/multus-rlj5t"
Sep 08 13:15:52.223012 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:15:52.220762    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e7df160-d688-4877-895e-edc5f73db16a-original-pull-secret podName:9e7df160-d688-4877-895e-edc5f73db16a nodeName:}" failed. No retries permitted until 2026-09-08 13:15:52.720743397 +0000 UTC m=+2.992191330 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9e7df160-d688-4877-895e-edc5f73db16a-original-pull-secret") pod "global-pull-secret-syncer-6qqfz" (UID: "9e7df160-d688-4877-895e-edc5f73db16a") : object "kube-system"/"original-pull-secret" not registered
Sep 08 13:15:52.223012 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.220809    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b936238e-bdf1-4dac-8bf8-6417c313597c-run-openvswitch\") pod \"ovnkube-node-8h8cn\" (UID: \"b936238e-bdf1-4dac-8bf8-6417c313597c\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h8cn"
Sep 08 13:15:52.223012 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.220818    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a88be331-067a-4b30-bdaa-3b36b7774d2f-hostroot\") pod \"multus-rlj5t\" (UID: \"a88be331-067a-4b30-bdaa-3b36b7774d2f\") " pod="openshift-multus/multus-rlj5t"
Sep 08 13:15:52.223012 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.220842    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b936238e-bdf1-4dac-8bf8-6417c313597c-host-run-ovn-kubernetes\") pod \"ovnkube-node-8h8cn\" (UID: \"b936238e-bdf1-4dac-8bf8-6417c313597c\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h8cn"
Sep 08 13:15:52.223012 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.220876    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b936238e-bdf1-4dac-8bf8-6417c313597c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8h8cn\" (UID: \"b936238e-bdf1-4dac-8bf8-6417c313597c\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h8cn"
Sep 08 13:15:52.223012 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.220920    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b936238e-bdf1-4dac-8bf8-6417c313597c-ovnkube-config\") pod \"ovnkube-node-8h8cn\" (UID: \"b936238e-bdf1-4dac-8bf8-6417c313597c\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h8cn"
Sep 08 13:15:52.223012 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.220942    2573 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 08 13:15:52.223012 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.220950    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4smg\" (UniqueName: \"kubernetes.io/projected/42aa8a5e-3aba-4581-9f95-f3b978c250f0-kube-api-access-x4smg\") pod \"node-resolver-rbnnh\" (UID: \"42aa8a5e-3aba-4581-9f95-f3b978c250f0\") " pod="openshift-dns/node-resolver-rbnnh"
Sep 08 13:15:52.223012 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.220992    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8f4ede58-3523-449b-bce0-9ba36f6d919b-host-slash\") pod \"iptables-alerter-kmf95\" (UID: \"8f4ede58-3523-449b-bce0-9ba36f6d919b\") " pod="openshift-network-operator/iptables-alerter-kmf95"
Sep 08 13:15:52.223012 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.221020    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a88be331-067a-4b30-bdaa-3b36b7774d2f-system-cni-dir\") pod \"multus-rlj5t\" (UID: \"a88be331-067a-4b30-bdaa-3b36b7774d2f\") " pod="openshift-multus/multus-rlj5t"
Sep 08 13:15:52.223012 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.221048    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9e7df160-d688-4877-895e-edc5f73db16a-dbus\") pod \"global-pull-secret-syncer-6qqfz\" (UID: \"9e7df160-d688-4877-895e-edc5f73db16a\") " pod="kube-system/global-pull-secret-syncer-6qqfz"
Sep 08 13:15:52.223012 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.221075    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/28b8b2ef-528a-4802-816c-c3c1de322861-socket-dir\") pod \"aws-ebs-csi-driver-node-hq2vz\" (UID: \"28b8b2ef-528a-4802-816c-c3c1de322861\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hq2vz"
Sep 08 13:15:52.223012 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.221101    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9m6q\" (UniqueName: \"kubernetes.io/projected/28b8b2ef-528a-4802-816c-c3c1de322861-kube-api-access-v9m6q\") pod \"aws-ebs-csi-driver-node-hq2vz\" (UID: \"28b8b2ef-528a-4802-816c-c3c1de322861\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hq2vz"
Sep 08 13:15:52.223012 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.221126    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b936238e-bdf1-4dac-8bf8-6417c313597c-run-systemd\") pod \"ovnkube-node-8h8cn\" (UID: \"b936238e-bdf1-4dac-8bf8-6417c313597c\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h8cn"
Sep 08 13:15:52.224081 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.223913    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a88be331-067a-4b30-bdaa-3b36b7774d2f-system-cni-dir\") pod \"multus-rlj5t\" (UID: \"a88be331-067a-4b30-bdaa-3b36b7774d2f\") " pod="openshift-multus/multus-rlj5t"
Sep 08 13:15:52.224634 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.224608    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b936238e-bdf1-4dac-8bf8-6417c313597c-env-overrides\") pod \"ovnkube-node-8h8cn\" (UID: \"b936238e-bdf1-4dac-8bf8-6417c313597c\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h8cn"
Sep 08 13:15:52.224736 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.224640    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9e7df160-d688-4877-895e-edc5f73db16a-dbus\") pod \"global-pull-secret-syncer-6qqfz\" (UID: \"9e7df160-d688-4877-895e-edc5f73db16a\") " pod="kube-system/global-pull-secret-syncer-6qqfz"
Sep 08 13:15:52.224736 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.224660    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b936238e-bdf1-4dac-8bf8-6417c313597c-ovnkube-script-lib\") pod \"ovnkube-node-8h8cn\" (UID: \"b936238e-bdf1-4dac-8bf8-6417c313597c\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h8cn"
Sep 08 13:15:52.224853 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.224702    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8f4ede58-3523-449b-bce0-9ba36f6d919b-iptables-alerter-script\") pod \"iptables-alerter-kmf95\" (UID: \"8f4ede58-3523-449b-bce0-9ba36f6d919b\") " pod="openshift-network-operator/iptables-alerter-kmf95"
Sep 08 13:15:52.224853 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.224784    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c6f2aedb-3459-4350-be14-4a64978cd32a-etc-sysctl-conf\") pod \"tuned-8ghvz\" (UID: \"c6f2aedb-3459-4350-be14-4a64978cd32a\") " pod="openshift-cluster-node-tuning-operator/tuned-8ghvz"
Sep 08 13:15:52.224853 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.224822    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c6f2aedb-3459-4350-be14-4a64978cd32a-lib-modules\") pod \"tuned-8ghvz\" (UID: \"c6f2aedb-3459-4350-be14-4a64978cd32a\") " pod="openshift-cluster-node-tuning-operator/tuned-8ghvz"
Sep 08 13:15:52.225004 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.224876    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zz7v8\" (UniqueName: \"kubernetes.io/projected/8f4ede58-3523-449b-bce0-9ba36f6d919b-kube-api-access-zz7v8\") pod \"iptables-alerter-kmf95\" (UID: \"8f4ede58-3523-449b-bce0-9ba36f6d919b\") " pod="openshift-network-operator/iptables-alerter-kmf95"
Sep 08 13:15:52.225360 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.225336    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c6f2aedb-3459-4350-be14-4a64978cd32a-etc-sysctl-conf\") pod \"tuned-8ghvz\" (UID: \"c6f2aedb-3459-4350-be14-4a64978cd32a\") " pod="openshift-cluster-node-tuning-operator/tuned-8ghvz"
Sep 08 13:15:52.225447 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.225378    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c6f2aedb-3459-4350-be14-4a64978cd32a-lib-modules\") pod \"tuned-8ghvz\" (UID: \"c6f2aedb-3459-4350-be14-4a64978cd32a\") " pod="openshift-cluster-node-tuning-operator/tuned-8ghvz"
Sep 08 13:15:52.226027 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.226004    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwwdv\" (UniqueName: \"kubernetes.io/projected/137be755-3092-45ba-a6d2-8a025c5e6226-kube-api-access-jwwdv\") pod \"node-ca-24f78\" (UID: \"137be755-3092-45ba-a6d2-8a025c5e6226\") " pod="openshift-image-registry/node-ca-24f78"
Sep 08 13:15:52.226119 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.226061    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c6f2aedb-3459-4350-be14-4a64978cd32a-etc-modprobe-d\") pod \"tuned-8ghvz\" (UID: \"c6f2aedb-3459-4350-be14-4a64978cd32a\") " pod="openshift-cluster-node-tuning-operator/tuned-8ghvz"
Sep 08 13:15:52.226119 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.226115    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c6f2aedb-3459-4350-be14-4a64978cd32a-sys\") pod \"tuned-8ghvz\" (UID: \"c6f2aedb-3459-4350-be14-4a64978cd32a\") " pod="openshift-cluster-node-tuning-operator/tuned-8ghvz"
Sep 08 13:15:52.226224 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.226150    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a88be331-067a-4b30-bdaa-3b36b7774d2f-host-var-lib-cni-bin\") pod \"multus-rlj5t\" (UID: \"a88be331-067a-4b30-bdaa-3b36b7774d2f\") " pod="openshift-multus/multus-rlj5t"
Sep 08 13:15:52.226285 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.226227    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8e45750b-bf27-4d49-8eba-1ba76fadd028-cni-binary-copy\") pod \"multus-additional-cni-plugins-krqr9\" (UID: \"8e45750b-bf27-4d49-8eba-1ba76fadd028\") " pod="openshift-multus/multus-additional-cni-plugins-krqr9"
Sep 08 13:15:52.226285 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.226264    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqhq8\" (UniqueName: \"kubernetes.io/projected/8e45750b-bf27-4d49-8eba-1ba76fadd028-kube-api-access-xqhq8\") pod \"multus-additional-cni-plugins-krqr9\" (UID: \"8e45750b-bf27-4d49-8eba-1ba76fadd028\") " pod="openshift-multus/multus-additional-cni-plugins-krqr9"
Sep 08 13:15:52.226387 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.226305    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/28b8b2ef-528a-4802-816c-c3c1de322861-registration-dir\") pod \"aws-ebs-csi-driver-node-hq2vz\" (UID: \"28b8b2ef-528a-4802-816c-c3c1de322861\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hq2vz"
Sep 08 13:15:52.226507 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.226489    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c6f2aedb-3459-4350-be14-4a64978cd32a-etc-modprobe-d\") pod \"tuned-8ghvz\" (UID: \"c6f2aedb-3459-4350-be14-4a64978cd32a\") " pod="openshift-cluster-node-tuning-operator/tuned-8ghvz"
Sep 08 13:15:52.226590 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.226564    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c6f2aedb-3459-4350-be14-4a64978cd32a-sys\") pod \"tuned-8ghvz\" (UID: \"c6f2aedb-3459-4350-be14-4a64978cd32a\") " pod="openshift-cluster-node-tuning-operator/tuned-8ghvz"
Sep 08 13:15:52.226659 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.226643    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a88be331-067a-4b30-bdaa-3b36b7774d2f-host-var-lib-cni-bin\") pod \"multus-rlj5t\" (UID: \"a88be331-067a-4b30-bdaa-3b36b7774d2f\") " pod="openshift-multus/multus-rlj5t"
Sep 08 13:15:52.227143 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.227121    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8e45750b-bf27-4d49-8eba-1ba76fadd028-cni-binary-copy\") pod \"multus-additional-cni-plugins-krqr9\" (UID: \"8e45750b-bf27-4d49-8eba-1ba76fadd028\") " pod="openshift-multus/multus-additional-cni-plugins-krqr9"
Sep 08 13:15:52.229780 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:15:52.227298    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 13:15:52.229780 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:15:52.227339    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 13:15:52.229780 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:15:52.227354    2573 projected.go:196] Error preparing data for projected volume kube-api-access-m4trk for pod openshift-network-diagnostics/network-check-target-mwdn6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 13:15:52.229780 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:15:52.227425    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0be74f6e-cde6-42fd-8772-4fca7f6bab00-kube-api-access-m4trk podName:0be74f6e-cde6-42fd-8772-4fca7f6bab00 nodeName:}" failed. No retries permitted until 2026-09-08 13:15:52.727406552 +0000 UTC m=+2.998854479 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-m4trk" (UniqueName: "kubernetes.io/projected/0be74f6e-cde6-42fd-8772-4fca7f6bab00-kube-api-access-m4trk") pod "network-check-target-mwdn6" (UID: "0be74f6e-cde6-42fd-8772-4fca7f6bab00") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 13:15:52.231710 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.231529    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c6f2aedb-3459-4350-be14-4a64978cd32a-etc-tuned\") pod \"tuned-8ghvz\" (UID: \"c6f2aedb-3459-4350-be14-4a64978cd32a\") " pod="openshift-cluster-node-tuning-operator/tuned-8ghvz"
Sep 08 13:15:52.231834 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.231591    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fp4sq\" (UniqueName: \"kubernetes.io/projected/c6f2aedb-3459-4350-be14-4a64978cd32a-kube-api-access-fp4sq\") pod \"tuned-8ghvz\" (UID: \"c6f2aedb-3459-4350-be14-4a64978cd32a\") " pod="openshift-cluster-node-tuning-operator/tuned-8ghvz"
Sep 08 13:15:52.231834 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.231633    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkzhj\" (UniqueName: \"kubernetes.io/projected/a88be331-067a-4b30-bdaa-3b36b7774d2f-kube-api-access-nkzhj\") pod \"multus-rlj5t\" (UID: \"a88be331-067a-4b30-bdaa-3b36b7774d2f\") " pod="openshift-multus/multus-rlj5t"
Sep 08 13:15:52.232090 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.232068    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdzwv\" (UniqueName: \"kubernetes.io/projected/217a8762-14bb-4990-8a53-3b3bcd6984ac-kube-api-access-fdzwv\") pod \"network-metrics-daemon-kpscq\" (UID: \"217a8762-14bb-4990-8a53-3b3bcd6984ac\") " pod="openshift-multus/network-metrics-daemon-kpscq"
Sep 08 13:15:52.232660 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.232634    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c6f2aedb-3459-4350-be14-4a64978cd32a-tmp\") pod \"tuned-8ghvz\" (UID: \"c6f2aedb-3459-4350-be14-4a64978cd32a\") " pod="openshift-cluster-node-tuning-operator/tuned-8ghvz"
Sep 08 13:15:52.235652 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.235569    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqhq8\" (UniqueName: \"kubernetes.io/projected/8e45750b-bf27-4d49-8eba-1ba76fadd028-kube-api-access-xqhq8\") pod \"multus-additional-cni-plugins-krqr9\" (UID: \"8e45750b-bf27-4d49-8eba-1ba76fadd028\") " pod="openshift-multus/multus-additional-cni-plugins-krqr9"
Sep 08 13:15:52.327674 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.327595    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42aa8a5e-3aba-4581-9f95-f3b978c250f0-tmp-dir\") pod \"node-resolver-rbnnh\" (UID: \"42aa8a5e-3aba-4581-9f95-f3b978c250f0\") " pod="openshift-dns/node-resolver-rbnnh"
Sep 08 13:15:52.327674 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.327645    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b936238e-bdf1-4dac-8bf8-6417c313597c-systemd-units\") pod \"ovnkube-node-8h8cn\" (UID: \"b936238e-bdf1-4dac-8bf8-6417c313597c\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h8cn"
Sep 08 13:15:52.327674 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.327662    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b936238e-bdf1-4dac-8bf8-6417c313597c-var-lib-openvswitch\") pod \"ovnkube-node-8h8cn\" (UID: \"b936238e-bdf1-4dac-8bf8-6417c313597c\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h8cn"
Sep 08 13:15:52.327965 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.327678    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b936238e-bdf1-4dac-8bf8-6417c313597c-host-cni-bin\") pod \"ovnkube-node-8h8cn\" (UID: \"b936238e-bdf1-4dac-8bf8-6417c313597c\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h8cn"
Sep 08 13:15:52.327965 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.327703    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9c39b80f-3adc-45fd-b1ec-39df3c237b02-agent-certs\") pod \"konnectivity-agent-7wrhb\" (UID: \"9c39b80f-3adc-45fd-b1ec-39df3c237b02\") " pod="kube-system/konnectivity-agent-7wrhb"
Sep 08 13:15:52.327965 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.327726    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9c39b80f-3adc-45fd-b1ec-39df3c237b02-konnectivity-ca\") pod \"konnectivity-agent-7wrhb\" (UID: \"9c39b80f-3adc-45fd-b1ec-39df3c237b02\") " pod="kube-system/konnectivity-agent-7wrhb"
Sep 08 13:15:52.327965 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.327733    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b936238e-bdf1-4dac-8bf8-6417c313597c-systemd-units\") pod \"ovnkube-node-8h8cn\" (UID: \"b936238e-bdf1-4dac-8bf8-6417c313597c\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h8cn"
Sep 08 13:15:52.327965 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.327745    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b936238e-bdf1-4dac-8bf8-6417c313597c-var-lib-openvswitch\") pod \"ovnkube-node-8h8cn\" (UID: \"b936238e-bdf1-4dac-8bf8-6417c313597c\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h8cn"
Sep 08 13:15:52.327965 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.327753    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b936238e-bdf1-4dac-8bf8-6417c313597c-node-log\") pod \"ovnkube-node-8h8cn\" (UID: \"b936238e-bdf1-4dac-8bf8-6417c313597c\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h8cn"
Sep 08 13:15:52.327965 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.327793    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b936238e-bdf1-4dac-8bf8-6417c313597c-node-log\") pod \"ovnkube-node-8h8cn\" (UID: \"b936238e-bdf1-4dac-8bf8-6417c313597c\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h8cn"
Sep 08 13:15:52.327965 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.327803    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b936238e-bdf1-4dac-8bf8-6417c313597c-host-cni-bin\") pod \"ovnkube-node-8h8cn\" (UID: \"b936238e-bdf1-4dac-8bf8-6417c313597c\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h8cn"
Sep 08 13:15:52.327965 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.327913    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/137be755-3092-45ba-a6d2-8a025c5e6226-host\") pod \"node-ca-24f78\" (UID: \"137be755-3092-45ba-a6d2-8a025c5e6226\") " pod="openshift-image-registry/node-ca-24f78"
Sep 08 13:15:52.327965 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.327944    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/137be755-3092-45ba-a6d2-8a025c5e6226-host\") pod \"node-ca-24f78\" (UID: \"137be755-3092-45ba-a6d2-8a025c5e6226\") " pod="openshift-image-registry/node-ca-24f78"
Sep 08 13:15:52.327965 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.327967    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b936238e-bdf1-4dac-8bf8-6417c313597c-host-kubelet\") pod \"ovnkube-node-8h8cn\" (UID: \"b936238e-bdf1-4dac-8bf8-6417c313597c\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h8cn"
Sep 08 13:15:52.328537 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.327985    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42aa8a5e-3aba-4581-9f95-f3b978c250f0-tmp-dir\") pod \"node-resolver-rbnnh\" (UID: \"42aa8a5e-3aba-4581-9f95-f3b978c250f0\") " pod="openshift-dns/node-resolver-rbnnh"
Sep 08 13:15:52.328537 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.327992    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b936238e-bdf1-4dac-8bf8-6417c313597c-ovn-node-metrics-cert\") pod \"ovnkube-node-8h8cn\" (UID: \"b936238e-bdf1-4dac-8bf8-6417c313597c\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h8cn"
Sep 08 13:15:52.328537 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.328023    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b936238e-bdf1-4dac-8bf8-6417c313597c-run-openvswitch\") pod \"ovnkube-node-8h8cn\" (UID: \"b936238e-bdf1-4dac-8bf8-6417c313597c\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h8cn"
Sep 08 13:15:52.328537 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.328039    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b936238e-bdf1-4dac-8bf8-6417c313597c-host-run-ovn-kubernetes\") pod \"ovnkube-node-8h8cn\" (UID: \"b936238e-bdf1-4dac-8bf8-6417c313597c\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h8cn"
Sep 08 13:15:52.328537 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.328065    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b936238e-bdf1-4dac-8bf8-6417c313597c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8h8cn\" (UID: \"b936238e-bdf1-4dac-8bf8-6417c313597c\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h8cn"
Sep 08 13:15:52.328537 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.328089    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b936238e-bdf1-4dac-8bf8-6417c313597c-ovnkube-config\") pod \"ovnkube-node-8h8cn\" (UID: \"b936238e-bdf1-4dac-8bf8-6417c313597c\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h8cn"
Sep 08 13:15:52.328537 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.328107    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4smg\" (UniqueName: \"kubernetes.io/projected/42aa8a5e-3aba-4581-9f95-f3b978c250f0-kube-api-access-x4smg\") pod \"node-resolver-rbnnh\" (UID: \"42aa8a5e-3aba-4581-9f95-f3b978c250f0\") " pod="openshift-dns/node-resolver-rbnnh"
Sep 08 13:15:52.328537 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.328125    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8f4ede58-3523-449b-bce0-9ba36f6d919b-host-slash\") pod \"iptables-alerter-kmf95\" (UID: \"8f4ede58-3523-449b-bce0-9ba36f6d919b\") " pod="openshift-network-operator/iptables-alerter-kmf95"
Sep 08 13:15:52.328537 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.328145    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/28b8b2ef-528a-4802-816c-c3c1de322861-socket-dir\") pod \"aws-ebs-csi-driver-node-hq2vz\" (UID: \"28b8b2ef-528a-4802-816c-c3c1de322861\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hq2vz"
Sep 08 13:15:52.328537 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.328168    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9m6q\" (UniqueName: \"kubernetes.io/projected/28b8b2ef-528a-4802-816c-c3c1de322861-kube-api-access-v9m6q\") pod \"aws-ebs-csi-driver-node-hq2vz\" (UID: \"28b8b2ef-528a-4802-816c-c3c1de322861\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hq2vz"
Sep 08 13:15:52.328537 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.328184    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b936238e-bdf1-4dac-8bf8-6417c313597c-run-systemd\") pod \"ovnkube-node-8h8cn\" (UID: \"b936238e-bdf1-4dac-8bf8-6417c313597c\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h8cn"
Sep 08 13:15:52.328537 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.328199    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b936238e-bdf1-4dac-8bf8-6417c313597c-env-overrides\") pod \"ovnkube-node-8h8cn\" (UID: \"b936238e-bdf1-4dac-8bf8-6417c313597c\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h8cn"
Sep 08 13:15:52.328537 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.328215    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b936238e-bdf1-4dac-8bf8-6417c313597c-ovnkube-script-lib\") pod \"ovnkube-node-8h8cn\" (UID: \"b936238e-bdf1-4dac-8bf8-6417c313597c\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h8cn"
Sep 08 13:15:52.328537 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.328231    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8f4ede58-3523-449b-bce0-9ba36f6d919b-iptables-alerter-script\") pod \"iptables-alerter-kmf95\" (UID: \"8f4ede58-3523-449b-bce0-9ba36f6d919b\") " pod="openshift-network-operator/iptables-alerter-kmf95"
Sep 08 13:15:52.328537 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.328257    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zz7v8\" (UniqueName: \"kubernetes.io/projected/8f4ede58-3523-449b-bce0-9ba36f6d919b-kube-api-access-zz7v8\") pod \"iptables-alerter-kmf95\" (UID: \"8f4ede58-3523-449b-bce0-9ba36f6d919b\") " pod="openshift-network-operator/iptables-alerter-kmf95"
Sep 08 13:15:52.328537 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.328281    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwwdv\" (UniqueName: \"kubernetes.io/projected/137be755-3092-45ba-a6d2-8a025c5e6226-kube-api-access-jwwdv\") pod \"node-ca-24f78\" (UID: \"137be755-3092-45ba-a6d2-8a025c5e6226\") " pod="openshift-image-registry/node-ca-24f78"
Sep 08 13:15:52.328537 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.328306    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/28b8b2ef-528a-4802-816c-c3c1de322861-registration-dir\") pod \"aws-ebs-csi-driver-node-hq2vz\" (UID: \"28b8b2ef-528a-4802-816c-c3c1de322861\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hq2vz"
Sep 08 13:15:52.329336 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.328336    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b936238e-bdf1-4dac-8bf8-6417c313597c-etc-openvswitch\") pod \"ovnkube-node-8h8cn\" (UID: \"b936238e-bdf1-4dac-8bf8-6417c313597c\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h8cn"
Sep 08 13:15:52.329336 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.328357    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b936238e-bdf1-4dac-8bf8-6417c313597c-log-socket\") pod \"ovnkube-node-8h8cn\" (UID: \"b936238e-bdf1-4dac-8bf8-6417c313597c\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h8cn"
Sep 08 13:15:52.329336 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.328362    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9c39b80f-3adc-45fd-b1ec-39df3c237b02-konnectivity-ca\") pod \"konnectivity-agent-7wrhb\" (UID: \"9c39b80f-3adc-45fd-b1ec-39df3c237b02\") " pod="kube-system/konnectivity-agent-7wrhb"
Sep 08 13:15:52.329336 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.328390    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/28b8b2ef-528a-4802-816c-c3c1de322861-device-dir\") pod \"aws-ebs-csi-driver-node-hq2vz\" (UID: \"28b8b2ef-528a-4802-816c-c3c1de322861\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hq2vz"
Sep 08 13:15:52.329336 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.328422    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/28b8b2ef-528a-4802-816c-c3c1de322861-sys-fs\") pod \"aws-ebs-csi-driver-node-hq2vz\" (UID: \"28b8b2ef-528a-4802-816c-c3c1de322861\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hq2vz"
Sep 08 13:15:52.329336 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.328426    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b936238e-bdf1-4dac-8bf8-6417c313597c-host-run-ovn-kubernetes\") pod \"ovnkube-node-8h8cn\" (UID: \"b936238e-bdf1-4dac-8bf8-6417c313597c\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h8cn"
Sep 08 13:15:52.329336 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.328426    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b936238e-bdf1-4dac-8bf8-6417c313597c-run-openvswitch\") pod \"ovnkube-node-8h8cn\" (UID: \"b936238e-bdf1-4dac-8bf8-6417c313597c\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h8cn"
Sep 08 13:15:52.329336 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.328040    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b936238e-bdf1-4dac-8bf8-6417c313597c-host-kubelet\") pod \"ovnkube-node-8h8cn\" (UID: \"b936238e-bdf1-4dac-8bf8-6417c313597c\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h8cn"
Sep 08 13:15:52.329336 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.328451    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b936238e-bdf1-4dac-8bf8-6417c313597c-host-slash\") pod \"ovnkube-node-8h8cn\" (UID: \"b936238e-bdf1-4dac-8bf8-6417c313597c\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h8cn"
Sep 08 13:15:52.329336 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.328483    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b936238e-bdf1-4dac-8bf8-6417c313597c-host-cni-netd\") pod \"ovnkube-node-8h8cn\" (UID: \"b936238e-bdf1-4dac-8bf8-6417c313597c\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h8cn"
Sep 08 13:15:52.329336 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.328489    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b936238e-bdf1-4dac-8bf8-6417c313597c-etc-openvswitch\") pod \"ovnkube-node-8h8cn\" (UID: \"b936238e-bdf1-4dac-8bf8-6417c313597c\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h8cn"
Sep 08 13:15:52.329336 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.328494    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b936238e-bdf1-4dac-8bf8-6417c313597c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8h8cn\" (UID: \"b936238e-bdf1-4dac-8bf8-6417c313597c\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h8cn"
Sep 08 13:15:52.329336 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.328509    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/137be755-3092-45ba-a6d2-8a025c5e6226-serviceca\") pod \"node-ca-24f78\" (UID: \"137be755-3092-45ba-a6d2-8a025c5e6226\") " pod="openshift-image-registry/node-ca-24f78"
Sep 08 13:15:52.329336 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.328554    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b936238e-bdf1-4dac-8bf8-6417c313597c-run-systemd\") pod \"ovnkube-node-8h8cn\" (UID: \"b936238e-bdf1-4dac-8bf8-6417c313597c\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h8cn"
Sep 08 13:15:52.329336 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.328578    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/28b8b2ef-528a-4802-816c-c3c1de322861-etc-selinux\") pod \"aws-ebs-csi-driver-node-hq2vz\" (UID: \"28b8b2ef-528a-4802-816c-c3c1de322861\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hq2vz"
Sep 08 13:15:52.329336 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.328631    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b936238e-bdf1-4dac-8bf8-6417c313597c-run-ovn\") pod \"ovnkube-node-8h8cn\" (UID: \"b936238e-bdf1-4dac-8bf8-6417c313597c\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h8cn"
Sep 08 13:15:52.329336 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.328660    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5v5g\" (UniqueName: \"kubernetes.io/projected/b936238e-bdf1-4dac-8bf8-6417c313597c-kube-api-access-t5v5g\") pod \"ovnkube-node-8h8cn\" (UID: \"b936238e-bdf1-4dac-8bf8-6417c313597c\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h8cn"
Sep 08 13:15:52.330117 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.328693    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/28b8b2ef-528a-4802-816c-c3c1de322861-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hq2vz\" (UID: \"28b8b2ef-528a-4802-816c-c3c1de322861\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hq2vz"
Sep 08 13:15:52.330117 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.328725    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b936238e-bdf1-4dac-8bf8-6417c313597c-host-run-netns\") pod \"ovnkube-node-8h8cn\" (UID: \"b936238e-bdf1-4dac-8bf8-6417c313597c\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h8cn"
Sep 08 13:15:52.330117 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.328751    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/42aa8a5e-3aba-4581-9f95-f3b978c250f0-hosts-file\") pod \"node-resolver-rbnnh\" (UID: \"42aa8a5e-3aba-4581-9f95-f3b978c250f0\") " pod="openshift-dns/node-resolver-rbnnh"
Sep 08 13:15:52.330117 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.328806    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b936238e-bdf1-4dac-8bf8-6417c313597c-ovnkube-config\") pod \"ovnkube-node-8h8cn\" (UID: \"b936238e-bdf1-4dac-8bf8-6417c313597c\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h8cn"
Sep 08 13:15:52.330117 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.328841    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/42aa8a5e-3aba-4581-9f95-f3b978c250f0-hosts-file\") pod \"node-resolver-rbnnh\" (UID: \"42aa8a5e-3aba-4581-9f95-f3b978c250f0\") " pod="openshift-dns/node-resolver-rbnnh"
Sep 08 13:15:52.330117 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.328899    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b936238e-bdf1-4dac-8bf8-6417c313597c-run-ovn\") pod \"ovnkube-node-8h8cn\" (UID: \"b936238e-bdf1-4dac-8bf8-6417c313597c\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h8cn"
Sep 08 13:15:52.330117 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.328983    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8f4ede58-3523-449b-bce0-9ba36f6d919b-host-slash\") pod \"iptables-alerter-kmf95\" (UID: \"8f4ede58-3523-449b-bce0-9ba36f6d919b\") " pod="openshift-network-operator/iptables-alerter-kmf95"
Sep 08 13:15:52.330117 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.328977    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/137be755-3092-45ba-a6d2-8a025c5e6226-serviceca\") pod \"node-ca-24f78\" (UID: \"137be755-3092-45ba-a6d2-8a025c5e6226\") " pod="openshift-image-registry/node-ca-24f78"
Sep 08 13:15:52.330117 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.329018    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b936238e-bdf1-4dac-8bf8-6417c313597c-ovnkube-script-lib\") pod \"ovnkube-node-8h8cn\" (UID: \"b936238e-bdf1-4dac-8bf8-6417c313597c\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h8cn"
Sep 08 13:15:52.330117 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.329040    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/28b8b2ef-528a-4802-816c-c3c1de322861-sys-fs\") pod \"aws-ebs-csi-driver-node-hq2vz\" (UID: \"28b8b2ef-528a-4802-816c-c3c1de322861\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hq2vz"
Sep 08 13:15:52.330117 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.329066    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b936238e-bdf1-4dac-8bf8-6417c313597c-log-socket\") pod \"ovnkube-node-8h8cn\" (UID: \"b936238e-bdf1-4dac-8bf8-6417c313597c\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h8cn"
Sep 08 13:15:52.330117 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.329076    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b936238e-bdf1-4dac-8bf8-6417c313597c-host-slash\") pod \"ovnkube-node-8h8cn\" (UID: \"b936238e-bdf1-4dac-8bf8-6417c313597c\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h8cn"
Sep 08 13:15:52.330117 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.329126    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/28b8b2ef-528a-4802-816c-c3c1de322861-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hq2vz\" (UID: \"28b8b2ef-528a-4802-816c-c3c1de322861\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hq2vz"
Sep 08 13:15:52.330117 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.329129    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/28b8b2ef-528a-4802-816c-c3c1de322861-device-dir\") pod \"aws-ebs-csi-driver-node-hq2vz\" (UID: \"28b8b2ef-528a-4802-816c-c3c1de322861\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hq2vz"
Sep 08 13:15:52.330117 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.328986    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/28b8b2ef-528a-4802-816c-c3c1de322861-etc-selinux\") pod \"aws-ebs-csi-driver-node-hq2vz\" (UID: \"28b8b2ef-528a-4802-816c-c3c1de322861\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hq2vz"
Sep 08 13:15:52.330117 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.329151    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b936238e-bdf1-4dac-8bf8-6417c313597c-host-cni-netd\") pod \"ovnkube-node-8h8cn\" (UID: \"b936238e-bdf1-4dac-8bf8-6417c313597c\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h8cn"
Sep 08 13:15:52.330117 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.328559    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/28b8b2ef-528a-4802-816c-c3c1de322861-socket-dir\") pod \"aws-ebs-csi-driver-node-hq2vz\" (UID: \"28b8b2ef-528a-4802-816c-c3c1de322861\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hq2vz"
Sep 08 13:15:52.330117 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.329175    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b936238e-bdf1-4dac-8bf8-6417c313597c-host-run-netns\") pod \"ovnkube-node-8h8cn\" (UID: \"b936238e-bdf1-4dac-8bf8-6417c313597c\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h8cn"
Sep 08 13:15:52.330946 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.329205    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/28b8b2ef-528a-4802-816c-c3c1de322861-registration-dir\") pod \"aws-ebs-csi-driver-node-hq2vz\" (UID: \"28b8b2ef-528a-4802-816c-c3c1de322861\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hq2vz"
Sep 08 13:15:52.330946 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.329603    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b936238e-bdf1-4dac-8bf8-6417c313597c-env-overrides\") pod \"ovnkube-node-8h8cn\" (UID: \"b936238e-bdf1-4dac-8bf8-6417c313597c\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h8cn"
Sep 08 13:15:52.330946 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.329625    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8f4ede58-3523-449b-bce0-9ba36f6d919b-iptables-alerter-script\") pod \"iptables-alerter-kmf95\" (UID: \"8f4ede58-3523-449b-bce0-9ba36f6d919b\") " pod="openshift-network-operator/iptables-alerter-kmf95"
Sep 08 13:15:52.331238 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.331219    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b936238e-bdf1-4dac-8bf8-6417c313597c-ovn-node-metrics-cert\") pod \"ovnkube-node-8h8cn\" (UID: \"b936238e-bdf1-4dac-8bf8-6417c313597c\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h8cn"
Sep 08 13:15:52.331316 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.331296    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9c39b80f-3adc-45fd-b1ec-39df3c237b02-agent-certs\") pod \"konnectivity-agent-7wrhb\" (UID: \"9c39b80f-3adc-45fd-b1ec-39df3c237b02\") " pod="kube-system/konnectivity-agent-7wrhb"
Sep 08 13:15:52.339483 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.339454    2573 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 13:15:52.341839 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.341808    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9m6q\" (UniqueName: \"kubernetes.io/projected/28b8b2ef-528a-4802-816c-c3c1de322861-kube-api-access-v9m6q\") pod \"aws-ebs-csi-driver-node-hq2vz\" (UID: \"28b8b2ef-528a-4802-816c-c3c1de322861\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hq2vz"
Sep 08 13:15:52.341922 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.341878    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5v5g\" (UniqueName: \"kubernetes.io/projected/b936238e-bdf1-4dac-8bf8-6417c313597c-kube-api-access-t5v5g\") pod \"ovnkube-node-8h8cn\" (UID: \"b936238e-bdf1-4dac-8bf8-6417c313597c\") " pod="openshift-ovn-kubernetes/ovnkube-node-8h8cn"
Sep 08 13:15:52.342297 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.342276    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zz7v8\" (UniqueName: \"kubernetes.io/projected/8f4ede58-3523-449b-bce0-9ba36f6d919b-kube-api-access-zz7v8\") pod \"iptables-alerter-kmf95\" (UID: \"8f4ede58-3523-449b-bce0-9ba36f6d919b\") " pod="openshift-network-operator/iptables-alerter-kmf95"
Sep 08 13:15:52.342297 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.342292    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwwdv\" (UniqueName: \"kubernetes.io/projected/137be755-3092-45ba-a6d2-8a025c5e6226-kube-api-access-jwwdv\") pod \"node-ca-24f78\" (UID: \"137be755-3092-45ba-a6d2-8a025c5e6226\") " pod="openshift-image-registry/node-ca-24f78"
Sep 08 13:15:52.342503 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.342483    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4smg\" (UniqueName: \"kubernetes.io/projected/42aa8a5e-3aba-4581-9f95-f3b978c250f0-kube-api-access-x4smg\") pod \"node-resolver-rbnnh\" (UID: \"42aa8a5e-3aba-4581-9f95-f3b978c250f0\") " pod="openshift-dns/node-resolver-rbnnh"
Sep 08 13:15:52.410671 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.410631    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-8ghvz"
Sep 08 13:15:52.419308 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.419283    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-rlj5t"
Sep 08 13:15:52.423811 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:52.423486    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6f2aedb_3459_4350_be14_4a64978cd32a.slice/crio-78ac5cc48a855d9f466845439398ba2e8a0fa750ed237aaa96ff5e2bd41473ae WatchSource:0}: Error finding container 78ac5cc48a855d9f466845439398ba2e8a0fa750ed237aaa96ff5e2bd41473ae: Status 404 returned error can't find the container with id 78ac5cc48a855d9f466845439398ba2e8a0fa750ed237aaa96ff5e2bd41473ae
Sep 08 13:15:52.429380 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:52.429355    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda88be331_067a_4b30_bdaa_3b36b7774d2f.slice/crio-d3252bebe98b650514ddb8bb4c521a7d97ff75df8821783e9fcdb6e07838425f WatchSource:0}: Error finding container d3252bebe98b650514ddb8bb4c521a7d97ff75df8821783e9fcdb6e07838425f: Status 404 returned error can't find the container with id d3252bebe98b650514ddb8bb4c521a7d97ff75df8821783e9fcdb6e07838425f
Sep 08 13:15:52.430933 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.430851    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-kmf95"
Sep 08 13:15:52.435839 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.435823    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-krqr9"
Sep 08 13:15:52.440205 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:52.440178    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f4ede58_3523_449b_bce0_9ba36f6d919b.slice/crio-65f49628ba54a1ff781ff273b122a182a615749662cc1faf07275e0998c0a20b WatchSource:0}: Error finding container 65f49628ba54a1ff781ff273b122a182a615749662cc1faf07275e0998c0a20b: Status 404 returned error can't find the container with id 65f49628ba54a1ff781ff273b122a182a615749662cc1faf07275e0998c0a20b
Sep 08 13:15:52.444780 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.444764    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-8h8cn"
Sep 08 13:15:52.446298 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:52.446273    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e45750b_bf27_4d49_8eba_1ba76fadd028.slice/crio-ddfa9bbbf66bfb7dc82e88f5e956044bf1ee8da81c72d298983e630b5b7771e5 WatchSource:0}: Error finding container ddfa9bbbf66bfb7dc82e88f5e956044bf1ee8da81c72d298983e630b5b7771e5: Status 404 returned error can't find the container with id ddfa9bbbf66bfb7dc82e88f5e956044bf1ee8da81c72d298983e630b5b7771e5
Sep 08 13:15:52.451121 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.451100    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-rbnnh"
Sep 08 13:15:52.454547 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:52.454518    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb936238e_bdf1_4dac_8bf8_6417c313597c.slice/crio-9427839b32f31fd27c3fd2803ae4041a22f56611816eb5fdafea2e5f75a366a4 WatchSource:0}: Error finding container 9427839b32f31fd27c3fd2803ae4041a22f56611816eb5fdafea2e5f75a366a4: Status 404 returned error can't find the container with id 9427839b32f31fd27c3fd2803ae4041a22f56611816eb5fdafea2e5f75a366a4
Sep 08 13:15:52.460358 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:52.460335    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42aa8a5e_3aba_4581_9f95_f3b978c250f0.slice/crio-d5822f2f6047ffc157f52053ffe163fbb0a46a8989c08dbe1fee1bbb549584d2 WatchSource:0}: Error finding container d5822f2f6047ffc157f52053ffe163fbb0a46a8989c08dbe1fee1bbb549584d2: Status 404 returned error can't find the container with id d5822f2f6047ffc157f52053ffe163fbb0a46a8989c08dbe1fee1bbb549584d2
Sep 08 13:15:52.461100 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.461080    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-7wrhb"
Sep 08 13:15:52.469013 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.468889    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-24f78"
Sep 08 13:15:52.471834 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:52.471800    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c39b80f_3adc_45fd_b1ec_39df3c237b02.slice/crio-1557131fb269b94402e721cb91ca1c2520420756ba2126c54329eed73d549317 WatchSource:0}: Error finding container 1557131fb269b94402e721cb91ca1c2520420756ba2126c54329eed73d549317: Status 404 returned error can't find the container with id 1557131fb269b94402e721cb91ca1c2520420756ba2126c54329eed73d549317
Sep 08 13:15:52.475628 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.475575    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hq2vz"
Sep 08 13:15:52.479600 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:52.479495    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod137be755_3092_45ba_a6d2_8a025c5e6226.slice/crio-dcbf04ea57af9ba17667a3fe697e9f3618c95e8127012a5436b207712a560f31 WatchSource:0}: Error finding container dcbf04ea57af9ba17667a3fe697e9f3618c95e8127012a5436b207712a560f31: Status 404 returned error can't find the container with id dcbf04ea57af9ba17667a3fe697e9f3618c95e8127012a5436b207712a560f31
Sep 08 13:15:52.488762 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:15:52.488735    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28b8b2ef_528a_4802_816c_c3c1de322861.slice/crio-cc6baa75b4d0677adcc8e0cbd5c98a76435d080a2ac538654ac7de22a503f975 WatchSource:0}: Error finding container cc6baa75b4d0677adcc8e0cbd5c98a76435d080a2ac538654ac7de22a503f975: Status 404 returned error can't find the container with id cc6baa75b4d0677adcc8e0cbd5c98a76435d080a2ac538654ac7de22a503f975
Sep 08 13:15:52.731448 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.731354    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/217a8762-14bb-4990-8a53-3b3bcd6984ac-metrics-certs\") pod \"network-metrics-daemon-kpscq\" (UID: \"217a8762-14bb-4990-8a53-3b3bcd6984ac\") " pod="openshift-multus/network-metrics-daemon-kpscq"
Sep 08 13:15:52.731448 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.731406    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9e7df160-d688-4877-895e-edc5f73db16a-original-pull-secret\") pod \"global-pull-secret-syncer-6qqfz\" (UID: \"9e7df160-d688-4877-895e-edc5f73db16a\") " pod="kube-system/global-pull-secret-syncer-6qqfz"
Sep 08 13:15:52.731448 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:52.731449    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4trk\" (UniqueName: \"kubernetes.io/projected/0be74f6e-cde6-42fd-8772-4fca7f6bab00-kube-api-access-m4trk\") pod \"network-check-target-mwdn6\" (UID: \"0be74f6e-cde6-42fd-8772-4fca7f6bab00\") " pod="openshift-network-diagnostics/network-check-target-mwdn6"
Sep 08 13:15:52.731765 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:15:52.731565    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 13:15:52.731765 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:15:52.731578    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 13:15:52.731765 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:15:52.731613    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 13:15:52.731765 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:15:52.731620    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 13:15:52.731765 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:15:52.731641    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/217a8762-14bb-4990-8a53-3b3bcd6984ac-metrics-certs podName:217a8762-14bb-4990-8a53-3b3bcd6984ac nodeName:}" failed. No retries permitted until 2026-09-08 13:15:53.731617874 +0000 UTC m=+4.003065796 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/217a8762-14bb-4990-8a53-3b3bcd6984ac-metrics-certs") pod "network-metrics-daemon-kpscq" (UID: "217a8762-14bb-4990-8a53-3b3bcd6984ac") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 13:15:52.731765 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:15:52.731686    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e7df160-d688-4877-895e-edc5f73db16a-original-pull-secret podName:9e7df160-d688-4877-895e-edc5f73db16a nodeName:}" failed. No retries permitted until 2026-09-08 13:15:53.731666481 +0000 UTC m=+4.003114397 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9e7df160-d688-4877-895e-edc5f73db16a-original-pull-secret") pod "global-pull-secret-syncer-6qqfz" (UID: "9e7df160-d688-4877-895e-edc5f73db16a") : object "kube-system"/"original-pull-secret" not registered
Sep 08 13:15:52.731765 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:15:52.731624    2573 projected.go:196] Error preparing data for projected volume kube-api-access-m4trk for pod openshift-network-diagnostics/network-check-target-mwdn6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 13:15:52.731765 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:15:52.731742    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0be74f6e-cde6-42fd-8772-4fca7f6bab00-kube-api-access-m4trk podName:0be74f6e-cde6-42fd-8772-4fca7f6bab00 nodeName:}" failed. No retries permitted until 2026-09-08 13:15:53.731730232 +0000 UTC m=+4.003178143 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-m4trk" (UniqueName: "kubernetes.io/projected/0be74f6e-cde6-42fd-8772-4fca7f6bab00-kube-api-access-m4trk") pod "network-check-target-mwdn6" (UID: "0be74f6e-cde6-42fd-8772-4fca7f6bab00") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 13:15:53.151838 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:53.151756    2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-07 13:10:52 +0000 UTC" deadline="2028-03-26 11:51:53.011764898 +0000 UTC"
Sep 08 13:15:53.151838 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:53.151790    2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13558h35m59.85997872s"
Sep 08 13:15:53.169212 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:53.169183    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mwdn6"
Sep 08 13:15:53.169382 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:15:53.169338    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mwdn6" podUID="0be74f6e-cde6-42fd-8772-4fca7f6bab00"
Sep 08 13:15:53.181129 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:53.181099    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-24f78" event={"ID":"137be755-3092-45ba-a6d2-8a025c5e6226","Type":"ContainerStarted","Data":"dcbf04ea57af9ba17667a3fe697e9f3618c95e8127012a5436b207712a560f31"}
Sep 08 13:15:53.182622 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:53.182592    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-krqr9" event={"ID":"8e45750b-bf27-4d49-8eba-1ba76fadd028","Type":"ContainerStarted","Data":"ddfa9bbbf66bfb7dc82e88f5e956044bf1ee8da81c72d298983e630b5b7771e5"}
Sep 08 13:15:53.184247 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:53.184213    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-kmf95" event={"ID":"8f4ede58-3523-449b-bce0-9ba36f6d919b","Type":"ContainerStarted","Data":"65f49628ba54a1ff781ff273b122a182a615749662cc1faf07275e0998c0a20b"}
Sep 08 13:15:53.185783 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:53.185755    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-rlj5t" event={"ID":"a88be331-067a-4b30-bdaa-3b36b7774d2f","Type":"ContainerStarted","Data":"d3252bebe98b650514ddb8bb4c521a7d97ff75df8821783e9fcdb6e07838425f"}
Sep 08 13:15:53.187120 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:53.187093    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hq2vz" event={"ID":"28b8b2ef-528a-4802-816c-c3c1de322861","Type":"ContainerStarted","Data":"cc6baa75b4d0677adcc8e0cbd5c98a76435d080a2ac538654ac7de22a503f975"}
Sep 08 13:15:53.189950 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:53.189922    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-7wrhb" event={"ID":"9c39b80f-3adc-45fd-b1ec-39df3c237b02","Type":"ContainerStarted","Data":"1557131fb269b94402e721cb91ca1c2520420756ba2126c54329eed73d549317"}
Sep 08 13:15:53.191728 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:53.191698    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-rbnnh" event={"ID":"42aa8a5e-3aba-4581-9f95-f3b978c250f0","Type":"ContainerStarted","Data":"d5822f2f6047ffc157f52053ffe163fbb0a46a8989c08dbe1fee1bbb549584d2"}
Sep 08 13:15:53.193520 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:53.193475    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8h8cn" event={"ID":"b936238e-bdf1-4dac-8bf8-6417c313597c","Type":"ContainerStarted","Data":"9427839b32f31fd27c3fd2803ae4041a22f56611816eb5fdafea2e5f75a366a4"}
Sep 08 13:15:53.195020 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:53.194974    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-8ghvz" event={"ID":"c6f2aedb-3459-4350-be14-4a64978cd32a","Type":"ContainerStarted","Data":"78ac5cc48a855d9f466845439398ba2e8a0fa750ed237aaa96ff5e2bd41473ae"}
Sep 08 13:15:53.740158 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:53.740068    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/217a8762-14bb-4990-8a53-3b3bcd6984ac-metrics-certs\") pod \"network-metrics-daemon-kpscq\" (UID: \"217a8762-14bb-4990-8a53-3b3bcd6984ac\") " pod="openshift-multus/network-metrics-daemon-kpscq"
Sep 08 13:15:53.740158 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:53.740129    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9e7df160-d688-4877-895e-edc5f73db16a-original-pull-secret\") pod \"global-pull-secret-syncer-6qqfz\" (UID: \"9e7df160-d688-4877-895e-edc5f73db16a\") " pod="kube-system/global-pull-secret-syncer-6qqfz"
Sep 08 13:15:53.740389 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:53.740186    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4trk\" (UniqueName: \"kubernetes.io/projected/0be74f6e-cde6-42fd-8772-4fca7f6bab00-kube-api-access-m4trk\") pod \"network-check-target-mwdn6\" (UID: \"0be74f6e-cde6-42fd-8772-4fca7f6bab00\") " pod="openshift-network-diagnostics/network-check-target-mwdn6"
Sep 08 13:15:53.740389 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:15:53.740321    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 13:15:53.740389 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:15:53.740342    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 13:15:53.740389 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:15:53.740356    2573 projected.go:196] Error preparing data for projected volume kube-api-access-m4trk for pod openshift-network-diagnostics/network-check-target-mwdn6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 13:15:53.740621 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:15:53.740419    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0be74f6e-cde6-42fd-8772-4fca7f6bab00-kube-api-access-m4trk podName:0be74f6e-cde6-42fd-8772-4fca7f6bab00 nodeName:}" failed. No retries permitted until 2026-09-08 13:15:55.740398654 +0000 UTC m=+6.011846582 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-m4trk" (UniqueName: "kubernetes.io/projected/0be74f6e-cde6-42fd-8772-4fca7f6bab00-kube-api-access-m4trk") pod "network-check-target-mwdn6" (UID: "0be74f6e-cde6-42fd-8772-4fca7f6bab00") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 13:15:53.740901 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:15:53.740866    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 13:15:53.740994 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:15:53.740920    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/217a8762-14bb-4990-8a53-3b3bcd6984ac-metrics-certs podName:217a8762-14bb-4990-8a53-3b3bcd6984ac nodeName:}" failed. No retries permitted until 2026-09-08 13:15:55.74090357 +0000 UTC m=+6.012351495 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/217a8762-14bb-4990-8a53-3b3bcd6984ac-metrics-certs") pod "network-metrics-daemon-kpscq" (UID: "217a8762-14bb-4990-8a53-3b3bcd6984ac") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 13:15:53.740994 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:15:53.740991    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 13:15:53.741117 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:15:53.741030    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e7df160-d688-4877-895e-edc5f73db16a-original-pull-secret podName:9e7df160-d688-4877-895e-edc5f73db16a nodeName:}" failed. No retries permitted until 2026-09-08 13:15:55.74101709 +0000 UTC m=+6.012465016 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9e7df160-d688-4877-895e-edc5f73db16a-original-pull-secret") pod "global-pull-secret-syncer-6qqfz" (UID: "9e7df160-d688-4877-895e-edc5f73db16a") : object "kube-system"/"original-pull-secret" not registered
Sep 08 13:15:53.812750 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:53.812719    2573 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 13:15:54.153203 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:54.152922    2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-07 13:10:52 +0000 UTC" deadline="2028-05-10 01:55:17.342868659 +0000 UTC"
Sep 08 13:15:54.153203 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:54.152990    2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14628h39m23.189883934s"
Sep 08 13:15:54.172673 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:54.172555    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kpscq"
Sep 08 13:15:54.174324 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:15:54.173827    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kpscq" podUID="217a8762-14bb-4990-8a53-3b3bcd6984ac"
Sep 08 13:15:54.174324 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:54.172707    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6qqfz"
Sep 08 13:15:54.174324 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:15:54.174282    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6qqfz" podUID="9e7df160-d688-4877-895e-edc5f73db16a"
Sep 08 13:15:54.210266 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:54.210227    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-254.ec2.internal" event={"ID":"7dfffcd72254a48b318c2c1ea59c531d","Type":"ContainerStarted","Data":"adff1215850dc982aac53e1c64fc4c527834202929546b4868a68feb2f4aadd5"}
Sep 08 13:15:55.170520 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:55.169998    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mwdn6"
Sep 08 13:15:55.170520 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:15:55.170120    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mwdn6" podUID="0be74f6e-cde6-42fd-8772-4fca7f6bab00"
Sep 08 13:15:55.762651 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:55.762592    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/217a8762-14bb-4990-8a53-3b3bcd6984ac-metrics-certs\") pod \"network-metrics-daemon-kpscq\" (UID: \"217a8762-14bb-4990-8a53-3b3bcd6984ac\") " pod="openshift-multus/network-metrics-daemon-kpscq"
Sep 08 13:15:55.762839 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:55.762679    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9e7df160-d688-4877-895e-edc5f73db16a-original-pull-secret\") pod \"global-pull-secret-syncer-6qqfz\" (UID: \"9e7df160-d688-4877-895e-edc5f73db16a\") " pod="kube-system/global-pull-secret-syncer-6qqfz"
Sep 08 13:15:55.762839 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:55.762741    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4trk\" (UniqueName: \"kubernetes.io/projected/0be74f6e-cde6-42fd-8772-4fca7f6bab00-kube-api-access-m4trk\") pod \"network-check-target-mwdn6\" (UID: \"0be74f6e-cde6-42fd-8772-4fca7f6bab00\") " pod="openshift-network-diagnostics/network-check-target-mwdn6"
Sep 08 13:15:55.762839 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:15:55.762783    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 13:15:55.763016 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:15:55.762847    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 13:15:55.763016 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:15:55.762867    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 13:15:55.763016 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:15:55.762869    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 13:15:55.763016 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:15:55.762879    2573 projected.go:196] Error preparing data for projected volume kube-api-access-m4trk for pod openshift-network-diagnostics/network-check-target-mwdn6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 13:15:55.763016 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:15:55.762851    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/217a8762-14bb-4990-8a53-3b3bcd6984ac-metrics-certs podName:217a8762-14bb-4990-8a53-3b3bcd6984ac nodeName:}" failed. No retries permitted until 2026-09-08 13:15:59.762832123 +0000 UTC m=+10.034280041 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/217a8762-14bb-4990-8a53-3b3bcd6984ac-metrics-certs") pod "network-metrics-daemon-kpscq" (UID: "217a8762-14bb-4990-8a53-3b3bcd6984ac") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 13:15:55.763016 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:15:55.763001    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e7df160-d688-4877-895e-edc5f73db16a-original-pull-secret podName:9e7df160-d688-4877-895e-edc5f73db16a nodeName:}" failed. No retries permitted until 2026-09-08 13:15:59.762981438 +0000 UTC m=+10.034429361 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9e7df160-d688-4877-895e-edc5f73db16a-original-pull-secret") pod "global-pull-secret-syncer-6qqfz" (UID: "9e7df160-d688-4877-895e-edc5f73db16a") : object "kube-system"/"original-pull-secret" not registered
Sep 08 13:15:55.763345 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:15:55.763022    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0be74f6e-cde6-42fd-8772-4fca7f6bab00-kube-api-access-m4trk podName:0be74f6e-cde6-42fd-8772-4fca7f6bab00 nodeName:}" failed. No retries permitted until 2026-09-08 13:15:59.76301111 +0000 UTC m=+10.034459022 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-m4trk" (UniqueName: "kubernetes.io/projected/0be74f6e-cde6-42fd-8772-4fca7f6bab00-kube-api-access-m4trk") pod "network-check-target-mwdn6" (UID: "0be74f6e-cde6-42fd-8772-4fca7f6bab00") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 13:15:56.171286 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:56.170685    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kpscq"
Sep 08 13:15:56.171286 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:15:56.170836    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kpscq" podUID="217a8762-14bb-4990-8a53-3b3bcd6984ac"
Sep 08 13:15:56.172086 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:56.171909    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6qqfz"
Sep 08 13:15:56.172086 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:15:56.172039    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6qqfz" podUID="9e7df160-d688-4877-895e-edc5f73db16a"
Sep 08 13:15:57.169720 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:57.169687    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mwdn6"
Sep 08 13:15:57.169885 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:15:57.169822    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mwdn6" podUID="0be74f6e-cde6-42fd-8772-4fca7f6bab00"
Sep 08 13:15:58.169368 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:58.169334    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kpscq"
Sep 08 13:15:58.169834 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:15:58.169473    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kpscq" podUID="217a8762-14bb-4990-8a53-3b3bcd6984ac"
Sep 08 13:15:58.169834 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:58.169673    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6qqfz"
Sep 08 13:15:58.169834 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:15:58.169785    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6qqfz" podUID="9e7df160-d688-4877-895e-edc5f73db16a"
Sep 08 13:15:59.169659 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:59.169582    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mwdn6"
Sep 08 13:15:59.170070 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:15:59.169702    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mwdn6" podUID="0be74f6e-cde6-42fd-8772-4fca7f6bab00"
Sep 08 13:15:59.794537 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:59.794486    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4trk\" (UniqueName: \"kubernetes.io/projected/0be74f6e-cde6-42fd-8772-4fca7f6bab00-kube-api-access-m4trk\") pod \"network-check-target-mwdn6\" (UID: \"0be74f6e-cde6-42fd-8772-4fca7f6bab00\") " pod="openshift-network-diagnostics/network-check-target-mwdn6"
Sep 08 13:15:59.794696 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:59.794558    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/217a8762-14bb-4990-8a53-3b3bcd6984ac-metrics-certs\") pod \"network-metrics-daemon-kpscq\" (UID: \"217a8762-14bb-4990-8a53-3b3bcd6984ac\") " pod="openshift-multus/network-metrics-daemon-kpscq"
Sep 08 13:15:59.794696 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:15:59.794597    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9e7df160-d688-4877-895e-edc5f73db16a-original-pull-secret\") pod \"global-pull-secret-syncer-6qqfz\" (UID: \"9e7df160-d688-4877-895e-edc5f73db16a\") " pod="kube-system/global-pull-secret-syncer-6qqfz"
Sep 08 13:15:59.794696 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:15:59.794662    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 13:15:59.794696 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:15:59.794686    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 13:15:59.794939 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:15:59.794700    2573 projected.go:196] Error preparing data for projected volume kube-api-access-m4trk for pod openshift-network-diagnostics/network-check-target-mwdn6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 13:15:59.794939 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:15:59.794722    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 13:15:59.794939 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:15:59.794755    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0be74f6e-cde6-42fd-8772-4fca7f6bab00-kube-api-access-m4trk podName:0be74f6e-cde6-42fd-8772-4fca7f6bab00 nodeName:}" failed. No retries permitted until 2026-09-08 13:16:07.79473719 +0000 UTC m=+18.066185113 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-m4trk" (UniqueName: "kubernetes.io/projected/0be74f6e-cde6-42fd-8772-4fca7f6bab00-kube-api-access-m4trk") pod "network-check-target-mwdn6" (UID: "0be74f6e-cde6-42fd-8772-4fca7f6bab00") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 13:15:59.794939 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:15:59.794723    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 13:15:59.794939 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:15:59.794776    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e7df160-d688-4877-895e-edc5f73db16a-original-pull-secret podName:9e7df160-d688-4877-895e-edc5f73db16a nodeName:}" failed. No retries permitted until 2026-09-08 13:16:07.794766476 +0000 UTC m=+18.066214398 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9e7df160-d688-4877-895e-edc5f73db16a-original-pull-secret") pod "global-pull-secret-syncer-6qqfz" (UID: "9e7df160-d688-4877-895e-edc5f73db16a") : object "kube-system"/"original-pull-secret" not registered
Sep 08 13:15:59.794939 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:15:59.794801    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/217a8762-14bb-4990-8a53-3b3bcd6984ac-metrics-certs podName:217a8762-14bb-4990-8a53-3b3bcd6984ac nodeName:}" failed. No retries permitted until 2026-09-08 13:16:07.794783233 +0000 UTC m=+18.066231146 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/217a8762-14bb-4990-8a53-3b3bcd6984ac-metrics-certs") pod "network-metrics-daemon-kpscq" (UID: "217a8762-14bb-4990-8a53-3b3bcd6984ac") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 13:16:00.173122 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:00.172613    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kpscq"
Sep 08 13:16:00.173122 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:16:00.172741    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kpscq" podUID="217a8762-14bb-4990-8a53-3b3bcd6984ac"
Sep 08 13:16:00.173122 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:00.172788    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6qqfz"
Sep 08 13:16:00.173122 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:16:00.172901    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6qqfz" podUID="9e7df160-d688-4877-895e-edc5f73db16a"
Sep 08 13:16:01.169820 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:01.169781    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mwdn6"
Sep 08 13:16:01.170006 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:16:01.169909    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mwdn6" podUID="0be74f6e-cde6-42fd-8772-4fca7f6bab00"
Sep 08 13:16:02.169382 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:02.169344    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kpscq"
Sep 08 13:16:02.169816 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:16:02.169472    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kpscq" podUID="217a8762-14bb-4990-8a53-3b3bcd6984ac"
Sep 08 13:16:02.169816 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:02.169537    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6qqfz"
Sep 08 13:16:02.169816 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:16:02.169628    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6qqfz" podUID="9e7df160-d688-4877-895e-edc5f73db16a"
Sep 08 13:16:03.169898 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:03.169867    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mwdn6"
Sep 08 13:16:03.170324 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:16:03.170000    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mwdn6" podUID="0be74f6e-cde6-42fd-8772-4fca7f6bab00"
Sep 08 13:16:04.169608 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:04.169569    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6qqfz"
Sep 08 13:16:04.169793 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:04.169731    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kpscq"
Sep 08 13:16:04.169854 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:16:04.169834    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kpscq" podUID="217a8762-14bb-4990-8a53-3b3bcd6984ac"
Sep 08 13:16:04.169912 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:16:04.169722    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6qqfz" podUID="9e7df160-d688-4877-895e-edc5f73db16a"
Sep 08 13:16:05.169308 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:05.169278    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mwdn6"
Sep 08 13:16:05.169823 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:16:05.169391    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mwdn6" podUID="0be74f6e-cde6-42fd-8772-4fca7f6bab00"
Sep 08 13:16:06.170107 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:06.170072    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kpscq"
Sep 08 13:16:06.170572 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:16:06.170260    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kpscq" podUID="217a8762-14bb-4990-8a53-3b3bcd6984ac"
Sep 08 13:16:06.170572 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:06.170072    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6qqfz"
Sep 08 13:16:06.170572 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:16:06.170405    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6qqfz" podUID="9e7df160-d688-4877-895e-edc5f73db16a"
Sep 08 13:16:07.170176 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:07.170144    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mwdn6"
Sep 08 13:16:07.170608 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:16:07.170268    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mwdn6" podUID="0be74f6e-cde6-42fd-8772-4fca7f6bab00"
Sep 08 13:16:07.851099 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:07.851010    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4trk\" (UniqueName: \"kubernetes.io/projected/0be74f6e-cde6-42fd-8772-4fca7f6bab00-kube-api-access-m4trk\") pod \"network-check-target-mwdn6\" (UID: \"0be74f6e-cde6-42fd-8772-4fca7f6bab00\") " pod="openshift-network-diagnostics/network-check-target-mwdn6"
Sep 08 13:16:07.851099 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:07.851050    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/217a8762-14bb-4990-8a53-3b3bcd6984ac-metrics-certs\") pod \"network-metrics-daemon-kpscq\" (UID: \"217a8762-14bb-4990-8a53-3b3bcd6984ac\") " pod="openshift-multus/network-metrics-daemon-kpscq"
Sep 08 13:16:07.851099 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:07.851080    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9e7df160-d688-4877-895e-edc5f73db16a-original-pull-secret\") pod \"global-pull-secret-syncer-6qqfz\" (UID: \"9e7df160-d688-4877-895e-edc5f73db16a\") " pod="kube-system/global-pull-secret-syncer-6qqfz"
Sep 08 13:16:07.851367 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:16:07.851176    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 13:16:07.851367 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:16:07.851192    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 13:16:07.851367 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:16:07.851191    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 13:16:07.851367 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:16:07.851208    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 13:16:07.851367 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:16:07.851223    2573 projected.go:196] Error preparing data for projected volume kube-api-access-m4trk for pod openshift-network-diagnostics/network-check-target-mwdn6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 13:16:07.851367 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:16:07.851234    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e7df160-d688-4877-895e-edc5f73db16a-original-pull-secret podName:9e7df160-d688-4877-895e-edc5f73db16a nodeName:}" failed. No retries permitted until 2026-09-08 13:16:23.851216392 +0000 UTC m=+34.122664315 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9e7df160-d688-4877-895e-edc5f73db16a-original-pull-secret") pod "global-pull-secret-syncer-6qqfz" (UID: "9e7df160-d688-4877-895e-edc5f73db16a") : object "kube-system"/"original-pull-secret" not registered
Sep 08 13:16:07.851367 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:16:07.851259    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/217a8762-14bb-4990-8a53-3b3bcd6984ac-metrics-certs podName:217a8762-14bb-4990-8a53-3b3bcd6984ac nodeName:}" failed. No retries permitted until 2026-09-08 13:16:23.851240449 +0000 UTC m=+34.122688372 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/217a8762-14bb-4990-8a53-3b3bcd6984ac-metrics-certs") pod "network-metrics-daemon-kpscq" (UID: "217a8762-14bb-4990-8a53-3b3bcd6984ac") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 13:16:07.851367 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:16:07.851278    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0be74f6e-cde6-42fd-8772-4fca7f6bab00-kube-api-access-m4trk podName:0be74f6e-cde6-42fd-8772-4fca7f6bab00 nodeName:}" failed. No retries permitted until 2026-09-08 13:16:23.851268317 +0000 UTC m=+34.122716227 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-m4trk" (UniqueName: "kubernetes.io/projected/0be74f6e-cde6-42fd-8772-4fca7f6bab00-kube-api-access-m4trk") pod "network-check-target-mwdn6" (UID: "0be74f6e-cde6-42fd-8772-4fca7f6bab00") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 13:16:08.169652 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:08.169579    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kpscq"
Sep 08 13:16:08.169780 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:16:08.169692    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kpscq" podUID="217a8762-14bb-4990-8a53-3b3bcd6984ac"
Sep 08 13:16:08.169780 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:08.169744    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6qqfz"
Sep 08 13:16:08.169852 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:16:08.169827    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6qqfz" podUID="9e7df160-d688-4877-895e-edc5f73db16a"
Sep 08 13:16:09.169493 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:09.169460    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mwdn6"
Sep 08 13:16:09.169932 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:16:09.169591    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mwdn6" podUID="0be74f6e-cde6-42fd-8772-4fca7f6bab00"
Sep 08 13:16:10.170386 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:10.170356    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kpscq"
Sep 08 13:16:10.170829 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:16:10.170456    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kpscq" podUID="217a8762-14bb-4990-8a53-3b3bcd6984ac"
Sep 08 13:16:10.170829 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:10.170511    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6qqfz"
Sep 08 13:16:10.170829 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:16:10.170617    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6qqfz" podUID="9e7df160-d688-4877-895e-edc5f73db16a"
Sep 08 13:16:11.169528 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:11.169366    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mwdn6"
Sep 08 13:16:11.169618 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:16:11.169601    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mwdn6" podUID="0be74f6e-cde6-42fd-8772-4fca7f6bab00"
Sep 08 13:16:12.169919 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:12.169771    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kpscq"
Sep 08 13:16:12.170350 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:12.169843    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6qqfz"
Sep 08 13:16:12.170350 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:16:12.169990    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kpscq" podUID="217a8762-14bb-4990-8a53-3b3bcd6984ac"
Sep 08 13:16:12.170350 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:16:12.170078    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6qqfz" podUID="9e7df160-d688-4877-895e-edc5f73db16a"
Sep 08 13:16:12.271288 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:12.271215    2573 generic.go:356] "Generic (PLEG): container finished" podID="6a0e339a5ac67687e98c7fee9b0ffa43" containerID="45bb94bab81fda3580950f3fb4efc74ead33df8ca90536d6fb60f358898297d3" exitCode=0
Sep 08 13:16:12.271417 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:12.271301    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-254.ec2.internal" event={"ID":"6a0e339a5ac67687e98c7fee9b0ffa43","Type":"ContainerDied","Data":"45bb94bab81fda3580950f3fb4efc74ead33df8ca90536d6fb60f358898297d3"}
Sep 08 13:16:12.273094 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:12.273068    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hq2vz" event={"ID":"28b8b2ef-528a-4802-816c-c3c1de322861","Type":"ContainerStarted","Data":"02a94da85e14f03344b86a1a9dcadd2f7868586d51053995017020cd45a1c8e3"}
Sep 08 13:16:12.275020 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:12.274986    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-7wrhb" event={"ID":"9c39b80f-3adc-45fd-b1ec-39df3c237b02","Type":"ContainerStarted","Data":"a6e3d2bd396aeb17cce0f18192a69a6173800bdd2666daa59b742fe13f57743d"}
Sep 08 13:16:12.276739 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:12.276716    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-rbnnh" event={"ID":"42aa8a5e-3aba-4581-9f95-f3b978c250f0","Type":"ContainerStarted","Data":"22746cc1b14be65a2723076ce0875d91c579b74bb3341bc248495054838ef996"}
Sep 08 13:16:12.280048 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:12.280031    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8h8cn_b936238e-bdf1-4dac-8bf8-6417c313597c/ovn-acl-logging/0.log"
Sep 08 13:16:12.280375 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:12.280355    2573 generic.go:356] "Generic (PLEG): container finished" podID="b936238e-bdf1-4dac-8bf8-6417c313597c" containerID="a43fd9233ae41985145f8d6d060224db570285a588c6cc478ba027d3589e096c" exitCode=1
Sep 08 13:16:12.280468 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:12.280417    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8h8cn" event={"ID":"b936238e-bdf1-4dac-8bf8-6417c313597c","Type":"ContainerStarted","Data":"d8908200f01f37382ce8cbc96ab8fd3f2b34b8563e05aca0c13c68f1cd79e21a"}
Sep 08 13:16:12.280550 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:12.280473    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8h8cn" event={"ID":"b936238e-bdf1-4dac-8bf8-6417c313597c","Type":"ContainerStarted","Data":"c8295356c846d9eed5cb4f494bd72586034039cc6a36126b1d7a93a6300ea006"}
Sep 08 13:16:12.280550 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:12.280539    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8h8cn" event={"ID":"b936238e-bdf1-4dac-8bf8-6417c313597c","Type":"ContainerStarted","Data":"1dcff88c0a09c37f12a6185606bff4c66611015984242730a836ec4e6a220cbf"}
Sep 08 13:16:12.280648 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:12.280571    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8h8cn" event={"ID":"b936238e-bdf1-4dac-8bf8-6417c313597c","Type":"ContainerStarted","Data":"a9ad726390a2de82ae41964cf6129de6a29b69857cff38ec159e9c3d491af4de"}
Sep 08 13:16:12.280648 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:12.280605    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8h8cn" event={"ID":"b936238e-bdf1-4dac-8bf8-6417c313597c","Type":"ContainerDied","Data":"a43fd9233ae41985145f8d6d060224db570285a588c6cc478ba027d3589e096c"}
Sep 08 13:16:12.280751 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:12.280673    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8h8cn" event={"ID":"b936238e-bdf1-4dac-8bf8-6417c313597c","Type":"ContainerStarted","Data":"d9d0dc98d4e0d350a208497ae68dc4da177567049f99b73dead8a65317611374"}
Sep 08 13:16:12.282051 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:12.282029    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-8ghvz" event={"ID":"c6f2aedb-3459-4350-be14-4a64978cd32a","Type":"ContainerStarted","Data":"b54ea440e5c9f8592d1b81ae52ab0c10778342b02f6eb1c179a98e0524503d33"}
Sep 08 13:16:12.283589 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:12.283566    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-24f78" event={"ID":"137be755-3092-45ba-a6d2-8a025c5e6226","Type":"ContainerStarted","Data":"a2ea4ca0fd3f3d885ec526ba67b7b5c26bcf1e0972cead117f239e7b0a0b5d61"}
Sep 08 13:16:12.285326 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:12.285299    2573 generic.go:356] "Generic (PLEG): container finished" podID="8e45750b-bf27-4d49-8eba-1ba76fadd028" containerID="4be97424dce2191e7e717cc32ce2eb325c11aff40b540f5e6b3385b057b0e4c5" exitCode=0
Sep 08 13:16:12.285424 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:12.285382    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-krqr9" event={"ID":"8e45750b-bf27-4d49-8eba-1ba76fadd028","Type":"ContainerDied","Data":"4be97424dce2191e7e717cc32ce2eb325c11aff40b540f5e6b3385b057b0e4c5"}
Sep 08 13:16:12.287022 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:12.287002    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-kmf95" event={"ID":"8f4ede58-3523-449b-bce0-9ba36f6d919b","Type":"ContainerStarted","Data":"550ed6cf5a0dca727b3bb12fde3fdd4b5be7495ba025f66cdaba8ed2af598e65"}
Sep 08 13:16:12.288807 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:12.288766    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-rlj5t" event={"ID":"a88be331-067a-4b30-bdaa-3b36b7774d2f","Type":"ContainerStarted","Data":"1426a4f49156deb36caef0f2e93350889d6275bda03414b7a5937adbd8003f3d"}
Sep 08 13:16:12.289686 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:12.289639    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-254.ec2.internal" podStartSLOduration=21.289627992 podStartE2EDuration="21.289627992s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 13:15:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:15:54.228809507 +0000 UTC m=+4.500257461" watchObservedRunningTime="2026-09-08 13:16:12.289627992 +0000 UTC m=+22.561075923"
Sep 08 13:16:12.312925 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:12.312894    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-8ghvz" podStartSLOduration=3.773494397 podStartE2EDuration="22.312883737s" totalImagesPullingTime="18.53938934s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 13:15:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:16:12.312680649 +0000 UTC m=+22.584128599" watchObservedRunningTime="2026-09-08 13:16:12.312883737 +0000 UTC m=+22.584331706"
Sep 08 13:16:12.361919 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:12.361874    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-rbnnh" podStartSLOduration=3.890923118 podStartE2EDuration="22.361856114s" totalImagesPullingTime="18.470932996s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 13:15:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:16:12.361110999 +0000 UTC m=+22.632558949" watchObservedRunningTime="2026-09-08 13:16:12.361856114 +0000 UTC m=+22.633304096"
Sep 08 13:16:12.376355 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:12.376240    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-kmf95" podStartSLOduration=3.886088823 podStartE2EDuration="22.376228461s" totalImagesPullingTime="18.490139638s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 13:15:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:16:12.376012667 +0000 UTC m=+22.647460621" watchObservedRunningTime="2026-09-08 13:16:12.376228461 +0000 UTC m=+22.647676393"
Sep 08 13:16:12.391016 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:12.390984    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-7wrhb" podStartSLOduration=3.901171321 podStartE2EDuration="22.390972419s" totalImagesPullingTime="18.489801098s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:15:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:16:12.390756106 +0000 UTC m=+22.662204037" watchObservedRunningTime="2026-09-08 13:16:12.390972419 +0000 UTC m=+22.662420347"
Sep 08 13:16:12.426496 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:12.426462    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-24f78" podStartSLOduration=3.945282029 podStartE2EDuration="22.426448684s" totalImagesPullingTime="18.481166655s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 13:15:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:16:12.426200912 +0000 UTC m=+22.697648844" watchObservedRunningTime="2026-09-08 13:16:12.426448684 +0000 UTC m=+22.697896614"
Sep 08 13:16:12.426617 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:12.426578    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-rlj5t" podStartSLOduration=3.427549278 podStartE2EDuration="22.426568399s" totalImagesPullingTime="18.999019121s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 13:15:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:16:12.411161476 +0000 UTC m=+22.682609407" watchObservedRunningTime="2026-09-08 13:16:12.426568399 +0000 UTC m=+22.698016331"
Sep 08 13:16:12.543944 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:12.543841    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:16:12.619307 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:12.619280    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:16:12.938485 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:12.938344    2573 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 08 13:16:13.017420 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:13.017397    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:16:13.079439 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:13.079359    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:16:13.170020 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:13.170001    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mwdn6"
Sep 08 13:16:13.170493 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:16:13.170099    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mwdn6" podUID="0be74f6e-cde6-42fd-8772-4fca7f6bab00"
Sep 08 13:16:13.173105 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:13.173029    2573 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-08T13:16:12.938481297Z","UUID":"0b709cbc-2cae-40b3-b54c-05a593bf6de2","Handler":null,"Name":"","Endpoint":""}
Sep 08 13:16:13.174583 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:13.174565    2573 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 08 13:16:13.174686 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:13.174598    2573 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 08 13:16:13.293660 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:13.293625    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-254.ec2.internal" event={"ID":"6a0e339a5ac67687e98c7fee9b0ffa43","Type":"ContainerStarted","Data":"fcce664f71c752cf6934a99eb154142217182ca682b931ad3202ae809dd69334"}
Sep 08 13:16:13.296206 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:13.296169    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hq2vz" event={"ID":"28b8b2ef-528a-4802-816c-c3c1de322861","Type":"ContainerStarted","Data":"83231dfb882bcf517f852b9867b8a3a120811f6a670bf7e1e6e53f4a85ee7345"}
Sep 08 13:16:13.309680 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:13.309645    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-254.ec2.internal" podStartSLOduration=22.309632336 podStartE2EDuration="22.309632336s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 13:15:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:16:13.309289803 +0000 UTC m=+23.580737761" watchObservedRunningTime="2026-09-08 13:16:13.309632336 +0000 UTC m=+23.581080267"
Sep 08 13:16:13.418042 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:13.417980    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:16:14.170009 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:14.169979    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6qqfz"
Sep 08 13:16:14.170175 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:14.169987    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kpscq"
Sep 08 13:16:14.170175 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:16:14.170109    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6qqfz" podUID="9e7df160-d688-4877-895e-edc5f73db16a"
Sep 08 13:16:14.170591 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:16:14.170244    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kpscq" podUID="217a8762-14bb-4990-8a53-3b3bcd6984ac"
Sep 08 13:16:14.302901 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:14.302880    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8h8cn_b936238e-bdf1-4dac-8bf8-6417c313597c/ovn-acl-logging/0.log"
Sep 08 13:16:14.303770 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:14.303743    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8h8cn" event={"ID":"b936238e-bdf1-4dac-8bf8-6417c313597c","Type":"ContainerStarted","Data":"1cf099ed693bfe36fa283eefd6b530d2ad2ddf020bee89c9159e746299f66266"}
Sep 08 13:16:14.537826 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:14.537751    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:16:14.640129 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:14.640099    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:16:15.170007 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:15.169979    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mwdn6"
Sep 08 13:16:15.170159 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:16:15.170099    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mwdn6" podUID="0be74f6e-cde6-42fd-8772-4fca7f6bab00"
Sep 08 13:16:15.200256 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:15.200192    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:16:15.308596 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:15.308561    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hq2vz" event={"ID":"28b8b2ef-528a-4802-816c-c3c1de322861","Type":"ContainerStarted","Data":"bf7294da093e58499bc4b505842d2101e0370e093ff638b332ff521a47c9db6a"}
Sep 08 13:16:15.327792 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:15.327741    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hq2vz" podStartSLOduration=4.475217719 podStartE2EDuration="25.327725379s" totalImagesPullingTime="20.85250766s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 13:15:50 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:16:15.327293463 +0000 UTC m=+25.598741396" watchObservedRunningTime="2026-09-08 13:16:15.327725379 +0000 UTC m=+25.599173310"
Sep 08 13:16:16.169608 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:16.169556    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6qqfz"
Sep 08 13:16:16.169764 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:16:16.169681    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6qqfz" podUID="9e7df160-d688-4877-895e-edc5f73db16a"
Sep 08 13:16:16.169764 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:16.169720    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kpscq"
Sep 08 13:16:16.169879 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:16:16.169805    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kpscq" podUID="217a8762-14bb-4990-8a53-3b3bcd6984ac"
Sep 08 13:16:16.623701 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:16.623667    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-7wrhb"
Sep 08 13:16:16.624301 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:16.624274    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-7wrhb"
Sep 08 13:16:17.169644 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:17.169618    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mwdn6"
Sep 08 13:16:17.169852 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:16:17.169726    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mwdn6" podUID="0be74f6e-cde6-42fd-8772-4fca7f6bab00"
Sep 08 13:16:17.316626 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:17.316605    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8h8cn_b936238e-bdf1-4dac-8bf8-6417c313597c/ovn-acl-logging/0.log"
Sep 08 13:16:17.317083 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:17.317035    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8h8cn" event={"ID":"b936238e-bdf1-4dac-8bf8-6417c313597c","Type":"ContainerStarted","Data":"2a8537455378be668a662e37db60d1585625942ac424e278cb476fff8c44cade"}
Sep 08 13:16:17.317382 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:17.317367    2573 scope.go:117] "RemoveContainer" containerID="a43fd9233ae41985145f8d6d060224db570285a588c6cc478ba027d3589e096c"
Sep 08 13:16:18.169347 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:18.169192    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kpscq"
Sep 08 13:16:18.169886 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:18.169244    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6qqfz"
Sep 08 13:16:18.169886 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:16:18.169434    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kpscq" podUID="217a8762-14bb-4990-8a53-3b3bcd6984ac"
Sep 08 13:16:18.169886 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:16:18.169522    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6qqfz" podUID="9e7df160-d688-4877-895e-edc5f73db16a"
Sep 08 13:16:18.321781 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:18.321758    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8h8cn_b936238e-bdf1-4dac-8bf8-6417c313597c/ovn-acl-logging/0.log"
Sep 08 13:16:18.322144 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:18.322120    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8h8cn" event={"ID":"b936238e-bdf1-4dac-8bf8-6417c313597c","Type":"ContainerStarted","Data":"16bffe5539342f88c8cacaa4a58d338de2aaaf9fd7e8c459504977665dd1a788"}
Sep 08 13:16:18.322241 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:18.322187    2573 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 08 13:16:18.322306 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:18.322290    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8h8cn"
Sep 08 13:16:18.322362 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:18.322334    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8h8cn"
Sep 08 13:16:18.324055 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:18.324034    2573 generic.go:356] "Generic (PLEG): container finished" podID="8e45750b-bf27-4d49-8eba-1ba76fadd028" containerID="c607b78fabbc51bff51a6a8019478d15966ef2b3ed28e039ab70db223a3bf727" exitCode=0
Sep 08 13:16:18.324168 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:18.324071    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-krqr9" event={"ID":"8e45750b-bf27-4d49-8eba-1ba76fadd028","Type":"ContainerDied","Data":"c607b78fabbc51bff51a6a8019478d15966ef2b3ed28e039ab70db223a3bf727"}
Sep 08 13:16:18.337455 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:18.337437    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8h8cn"
Sep 08 13:16:18.337730 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:18.337718    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8h8cn"
Sep 08 13:16:18.355031 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:18.354991    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-8h8cn" podStartSLOduration=9.600770964 podStartE2EDuration="28.354980559s" totalImagesPullingTime="18.754209595s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 13:15:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:16:18.353985368 +0000 UTC m=+28.625433299" watchObservedRunningTime="2026-09-08 13:16:18.354980559 +0000 UTC m=+28.626428489"
Sep 08 13:16:18.638878 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:18.638792    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-7wrhb"
Sep 08 13:16:18.639000 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:18.638941    2573 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 08 13:16:18.639482 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:18.639459    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-7wrhb"
Sep 08 13:16:18.707975 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:18.707948    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:16:19.124934 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:19.124896    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-mwdn6"]
Sep 08 13:16:19.125080 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:19.125033    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mwdn6"
Sep 08 13:16:19.125171 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:16:19.125145    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mwdn6" podUID="0be74f6e-cde6-42fd-8772-4fca7f6bab00"
Sep 08 13:16:19.127644 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:19.127622    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-kpscq"]
Sep 08 13:16:19.127733 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:19.127711    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kpscq"
Sep 08 13:16:19.127806 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:16:19.127784    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kpscq" podUID="217a8762-14bb-4990-8a53-3b3bcd6984ac"
Sep 08 13:16:19.140019 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:19.139995    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-6qqfz"]
Sep 08 13:16:19.140104 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:19.140078    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6qqfz"
Sep 08 13:16:19.140159 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:16:19.140147    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6qqfz" podUID="9e7df160-d688-4877-895e-edc5f73db16a"
Sep 08 13:16:19.329026 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:19.328869    2573 generic.go:356] "Generic (PLEG): container finished" podID="8e45750b-bf27-4d49-8eba-1ba76fadd028" containerID="09418c069491534909b81209ff9f602457c1ac51177dcf180bb8eade38a56bc4" exitCode=0
Sep 08 13:16:19.329521 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:19.328930    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-krqr9" event={"ID":"8e45750b-bf27-4d49-8eba-1ba76fadd028","Type":"ContainerDied","Data":"09418c069491534909b81209ff9f602457c1ac51177dcf180bb8eade38a56bc4"}
Sep 08 13:16:19.329521 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:19.329274    2573 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 08 13:16:20.333301 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:20.333272    2573 generic.go:356] "Generic (PLEG): container finished" podID="8e45750b-bf27-4d49-8eba-1ba76fadd028" containerID="a0f100fb03d382f8a4a8e453bf8d3c391bb5636ef6a4b2268a2d78f387e600ee" exitCode=0
Sep 08 13:16:20.333600 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:20.333356    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-krqr9" event={"ID":"8e45750b-bf27-4d49-8eba-1ba76fadd028","Type":"ContainerDied","Data":"a0f100fb03d382f8a4a8e453bf8d3c391bb5636ef6a4b2268a2d78f387e600ee"}
Sep 08 13:16:20.333600 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:20.333418    2573 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 08 13:16:20.385230 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:20.385206    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:16:20.486456 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:20.486389    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:16:21.169324 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:21.169251    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mwdn6"
Sep 08 13:16:21.169518 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:21.169251    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6qqfz"
Sep 08 13:16:21.169518 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:16:21.169368    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mwdn6" podUID="0be74f6e-cde6-42fd-8772-4fca7f6bab00"
Sep 08 13:16:21.169518 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:21.169251    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kpscq"
Sep 08 13:16:21.169518 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:16:21.169424    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6qqfz" podUID="9e7df160-d688-4877-895e-edc5f73db16a"
Sep 08 13:16:21.169730 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:16:21.169517    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kpscq" podUID="217a8762-14bb-4990-8a53-3b3bcd6984ac"
Sep 08 13:16:22.296293 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:22.296255    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8h8cn"
Sep 08 13:16:22.296669 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:22.296491    2573 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 08 13:16:22.307161 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:22.307124    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-8h8cn" podUID="b936238e-bdf1-4dac-8bf8-6417c313597c" containerName="ovnkube-controller" probeResult="failure" output=""
Sep 08 13:16:22.315234 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:22.315208    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-8h8cn" podUID="b936238e-bdf1-4dac-8bf8-6417c313597c" containerName="ovnkube-controller" probeResult="failure" output=""
Sep 08 13:16:23.169398 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:23.169371    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mwdn6"
Sep 08 13:16:23.169568 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:23.169382    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kpscq"
Sep 08 13:16:23.169568 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:16:23.169475    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mwdn6" podUID="0be74f6e-cde6-42fd-8772-4fca7f6bab00"
Sep 08 13:16:23.169568 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:23.169514    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6qqfz"
Sep 08 13:16:23.169710 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:16:23.169620    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kpscq" podUID="217a8762-14bb-4990-8a53-3b3bcd6984ac"
Sep 08 13:16:23.169710 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:16:23.169697    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6qqfz" podUID="9e7df160-d688-4877-895e-edc5f73db16a"
Sep 08 13:16:23.871323 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:23.871290    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4trk\" (UniqueName: \"kubernetes.io/projected/0be74f6e-cde6-42fd-8772-4fca7f6bab00-kube-api-access-m4trk\") pod \"network-check-target-mwdn6\" (UID: \"0be74f6e-cde6-42fd-8772-4fca7f6bab00\") " pod="openshift-network-diagnostics/network-check-target-mwdn6"
Sep 08 13:16:23.872405 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:23.871334    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/217a8762-14bb-4990-8a53-3b3bcd6984ac-metrics-certs\") pod \"network-metrics-daemon-kpscq\" (UID: \"217a8762-14bb-4990-8a53-3b3bcd6984ac\") " pod="openshift-multus/network-metrics-daemon-kpscq"
Sep 08 13:16:23.872405 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:23.871379    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9e7df160-d688-4877-895e-edc5f73db16a-original-pull-secret\") pod \"global-pull-secret-syncer-6qqfz\" (UID: \"9e7df160-d688-4877-895e-edc5f73db16a\") " pod="kube-system/global-pull-secret-syncer-6qqfz"
Sep 08 13:16:23.872405 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:16:23.871441    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 13:16:23.872405 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:16:23.871450    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 13:16:23.872405 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:16:23.871475    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 13:16:23.872405 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:16:23.871520    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/217a8762-14bb-4990-8a53-3b3bcd6984ac-metrics-certs podName:217a8762-14bb-4990-8a53-3b3bcd6984ac nodeName:}" failed. No retries permitted until 2026-09-08 13:16:55.871485797 +0000 UTC m=+66.142933720 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/217a8762-14bb-4990-8a53-3b3bcd6984ac-metrics-certs") pod "network-metrics-daemon-kpscq" (UID: "217a8762-14bb-4990-8a53-3b3bcd6984ac") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 13:16:23.872405 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:16:23.871522    2573 projected.go:196] Error preparing data for projected volume kube-api-access-m4trk for pod openshift-network-diagnostics/network-check-target-mwdn6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 13:16:23.872405 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:16:23.871566    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0be74f6e-cde6-42fd-8772-4fca7f6bab00-kube-api-access-m4trk podName:0be74f6e-cde6-42fd-8772-4fca7f6bab00 nodeName:}" failed. No retries permitted until 2026-09-08 13:16:55.871555976 +0000 UTC m=+66.143003885 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-m4trk" (UniqueName: "kubernetes.io/projected/0be74f6e-cde6-42fd-8772-4fca7f6bab00-kube-api-access-m4trk") pod "network-check-target-mwdn6" (UID: "0be74f6e-cde6-42fd-8772-4fca7f6bab00") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 13:16:23.872405 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:16:23.871491    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 13:16:23.872405 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:16:23.871598    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e7df160-d688-4877-895e-edc5f73db16a-original-pull-secret podName:9e7df160-d688-4877-895e-edc5f73db16a nodeName:}" failed. No retries permitted until 2026-09-08 13:16:55.871590029 +0000 UTC m=+66.143037938 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9e7df160-d688-4877-895e-edc5f73db16a-original-pull-secret") pod "global-pull-secret-syncer-6qqfz" (UID: "9e7df160-d688-4877-895e-edc5f73db16a") : object "kube-system"/"original-pull-secret" not registered
Sep 08 13:16:24.075126 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:24.075098    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-254.ec2.internal" event="NodeReady"
Sep 08 13:16:24.075307 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:24.075255    2573 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 08 13:16:24.137533 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:24.137466    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-knfp2"]
Sep 08 13:16:24.190703 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:24.190679    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-knfp2"
Sep 08 13:16:24.193235 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:24.193215    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 08 13:16:24.193813 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:24.193792    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-8shlb\""
Sep 08 13:16:24.193931 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:24.193839    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 08 13:16:24.194202 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:24.194181    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-qmrbg"]
Sep 08 13:16:24.214967 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:24.214943    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-knfp2"]
Sep 08 13:16:24.215069 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:24.215004    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-qmrbg"]
Sep 08 13:16:24.215131 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:24.215070    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-qmrbg"
Sep 08 13:16:24.215185 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:24.215074    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mwdn6"
Sep 08 13:16:24.220289 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:24.220223    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-x5pb6\""
Sep 08 13:16:24.220546 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:24.220226    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 08 13:16:24.220648 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:24.220342    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 08 13:16:24.220717 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:24.220676    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 08 13:16:24.220717 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:24.220691    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 08 13:16:24.221028 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:24.221008    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 08 13:16:24.221130 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:24.221086    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-8cl4h\""
Sep 08 13:16:24.275148 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:24.275123    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/21ee5269-0851-405f-a9a5-f9d2741e8500-metrics-tls\") pod \"dns-default-knfp2\" (UID: \"21ee5269-0851-405f-a9a5-f9d2741e8500\") " pod="openshift-dns/dns-default-knfp2"
Sep 08 13:16:24.275262 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:24.275163    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21ee5269-0851-405f-a9a5-f9d2741e8500-tmp-dir\") pod \"dns-default-knfp2\" (UID: \"21ee5269-0851-405f-a9a5-f9d2741e8500\") " pod="openshift-dns/dns-default-knfp2"
Sep 08 13:16:24.275262 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:24.275183    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q44ks\" (UniqueName: \"kubernetes.io/projected/21ee5269-0851-405f-a9a5-f9d2741e8500-kube-api-access-q44ks\") pod \"dns-default-knfp2\" (UID: \"21ee5269-0851-405f-a9a5-f9d2741e8500\") " pod="openshift-dns/dns-default-knfp2"
Sep 08 13:16:24.275262 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:24.275205    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/21ee5269-0851-405f-a9a5-f9d2741e8500-config-volume\") pod \"dns-default-knfp2\" (UID: \"21ee5269-0851-405f-a9a5-f9d2741e8500\") " pod="openshift-dns/dns-default-knfp2"
Sep 08 13:16:24.325649 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:24.325622    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:16:24.376389 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:24.376365    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/21ee5269-0851-405f-a9a5-f9d2741e8500-metrics-tls\") pod \"dns-default-knfp2\" (UID: \"21ee5269-0851-405f-a9a5-f9d2741e8500\") " pod="openshift-dns/dns-default-knfp2"
Sep 08 13:16:24.376523 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:24.376401    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21ee5269-0851-405f-a9a5-f9d2741e8500-tmp-dir\") pod \"dns-default-knfp2\" (UID: \"21ee5269-0851-405f-a9a5-f9d2741e8500\") " pod="openshift-dns/dns-default-knfp2"
Sep 08 13:16:24.376523 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:24.376431    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q44ks\" (UniqueName: \"kubernetes.io/projected/21ee5269-0851-405f-a9a5-f9d2741e8500-kube-api-access-q44ks\") pod \"dns-default-knfp2\" (UID: \"21ee5269-0851-405f-a9a5-f9d2741e8500\") " pod="openshift-dns/dns-default-knfp2"
Sep 08 13:16:24.376639 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:16:24.376544    2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 13:16:24.376639 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:16:24.376600    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21ee5269-0851-405f-a9a5-f9d2741e8500-metrics-tls podName:21ee5269-0851-405f-a9a5-f9d2741e8500 nodeName:}" failed. No retries permitted until 2026-09-08 13:16:24.876582073 +0000 UTC m=+35.148029985 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/21ee5269-0851-405f-a9a5-f9d2741e8500-metrics-tls") pod "dns-default-knfp2" (UID: "21ee5269-0851-405f-a9a5-f9d2741e8500") : secret "dns-default-metrics-tls" not found
Sep 08 13:16:24.376721 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:24.376687    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/21ee5269-0851-405f-a9a5-f9d2741e8500-config-volume\") pod \"dns-default-knfp2\" (UID: \"21ee5269-0851-405f-a9a5-f9d2741e8500\") " pod="openshift-dns/dns-default-knfp2"
Sep 08 13:16:24.376769 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:24.376755    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmbjz\" (UniqueName: \"kubernetes.io/projected/831913e7-ea98-4d0b-bbaf-411e16552f73-kube-api-access-xmbjz\") pod \"ingress-canary-qmrbg\" (UID: \"831913e7-ea98-4d0b-bbaf-411e16552f73\") " pod="openshift-ingress-canary/ingress-canary-qmrbg"
Sep 08 13:16:24.376816 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:24.376796    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/831913e7-ea98-4d0b-bbaf-411e16552f73-cert\") pod \"ingress-canary-qmrbg\" (UID: \"831913e7-ea98-4d0b-bbaf-411e16552f73\") " pod="openshift-ingress-canary/ingress-canary-qmrbg"
Sep 08 13:16:24.377117 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:24.377096    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21ee5269-0851-405f-a9a5-f9d2741e8500-tmp-dir\") pod \"dns-default-knfp2\" (UID: \"21ee5269-0851-405f-a9a5-f9d2741e8500\") " pod="openshift-dns/dns-default-knfp2"
Sep 08 13:16:24.377330 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:24.377307    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/21ee5269-0851-405f-a9a5-f9d2741e8500-config-volume\") pod \"dns-default-knfp2\" (UID: \"21ee5269-0851-405f-a9a5-f9d2741e8500\") " pod="openshift-dns/dns-default-knfp2"
Sep 08 13:16:24.387938 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:24.387892    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q44ks\" (UniqueName: \"kubernetes.io/projected/21ee5269-0851-405f-a9a5-f9d2741e8500-kube-api-access-q44ks\") pod \"dns-default-knfp2\" (UID: \"21ee5269-0851-405f-a9a5-f9d2741e8500\") " pod="openshift-dns/dns-default-knfp2"
Sep 08 13:16:24.420533 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:24.420491    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:16:24.477129 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:24.477102    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/831913e7-ea98-4d0b-bbaf-411e16552f73-cert\") pod \"ingress-canary-qmrbg\" (UID: \"831913e7-ea98-4d0b-bbaf-411e16552f73\") " pod="openshift-ingress-canary/ingress-canary-qmrbg"
Sep 08 13:16:24.477283 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:24.477193    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmbjz\" (UniqueName: \"kubernetes.io/projected/831913e7-ea98-4d0b-bbaf-411e16552f73-kube-api-access-xmbjz\") pod \"ingress-canary-qmrbg\" (UID: \"831913e7-ea98-4d0b-bbaf-411e16552f73\") " pod="openshift-ingress-canary/ingress-canary-qmrbg"
Sep 08 13:16:24.480096 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:24.480067    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/831913e7-ea98-4d0b-bbaf-411e16552f73-cert\") pod \"ingress-canary-qmrbg\" (UID: \"831913e7-ea98-4d0b-bbaf-411e16552f73\") " pod="openshift-ingress-canary/ingress-canary-qmrbg"
Sep 08 13:16:24.487606 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:24.487583    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmbjz\" (UniqueName: \"kubernetes.io/projected/831913e7-ea98-4d0b-bbaf-411e16552f73-kube-api-access-xmbjz\") pod \"ingress-canary-qmrbg\" (UID: \"831913e7-ea98-4d0b-bbaf-411e16552f73\") " pod="openshift-ingress-canary/ingress-canary-qmrbg"
Sep 08 13:16:24.526512 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:24.526478    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-qmrbg"
Sep 08 13:16:24.880157 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:24.880126    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/21ee5269-0851-405f-a9a5-f9d2741e8500-metrics-tls\") pod \"dns-default-knfp2\" (UID: \"21ee5269-0851-405f-a9a5-f9d2741e8500\") " pod="openshift-dns/dns-default-knfp2"
Sep 08 13:16:24.880580 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:16:24.880303    2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 13:16:24.880580 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:16:24.880381    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21ee5269-0851-405f-a9a5-f9d2741e8500-metrics-tls podName:21ee5269-0851-405f-a9a5-f9d2741e8500 nodeName:}" failed. No retries permitted until 2026-09-08 13:16:25.880360636 +0000 UTC m=+36.151808551 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/21ee5269-0851-405f-a9a5-f9d2741e8500-metrics-tls") pod "dns-default-knfp2" (UID: "21ee5269-0851-405f-a9a5-f9d2741e8500") : secret "dns-default-metrics-tls" not found
Sep 08 13:16:24.903682 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:24.903653    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-qmrbg"]
Sep 08 13:16:24.908051 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:16:24.908030    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod831913e7_ea98_4d0b_bbaf_411e16552f73.slice/crio-86c25251ef750d07220cc62c207cb57c5e9d17e2fb3a5399ef6faa95c3c21ae0 WatchSource:0}: Error finding container 86c25251ef750d07220cc62c207cb57c5e9d17e2fb3a5399ef6faa95c3c21ae0: Status 404 returned error can't find the container with id 86c25251ef750d07220cc62c207cb57c5e9d17e2fb3a5399ef6faa95c3c21ae0
Sep 08 13:16:25.169509 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:25.169477    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kpscq"
Sep 08 13:16:25.169615 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:25.169476    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6qqfz"
Sep 08 13:16:25.172262 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:25.172243    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 08 13:16:25.172885 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:25.172870    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 08 13:16:25.172951 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:25.172882    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-4rwvw\""
Sep 08 13:16:25.345791 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:25.345715    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-qmrbg" event={"ID":"831913e7-ea98-4d0b-bbaf-411e16552f73","Type":"ContainerStarted","Data":"86c25251ef750d07220cc62c207cb57c5e9d17e2fb3a5399ef6faa95c3c21ae0"}
Sep 08 13:16:25.348443 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:25.348418    2573 generic.go:356] "Generic (PLEG): container finished" podID="8e45750b-bf27-4d49-8eba-1ba76fadd028" containerID="143ee79a43eaa7982d96337622d7ff17673dc4b45ea756bd0b046c4c080df1b7" exitCode=0
Sep 08 13:16:25.348559 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:25.348473    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-krqr9" event={"ID":"8e45750b-bf27-4d49-8eba-1ba76fadd028","Type":"ContainerDied","Data":"143ee79a43eaa7982d96337622d7ff17673dc4b45ea756bd0b046c4c080df1b7"}
Sep 08 13:16:25.888620 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:25.888582    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/21ee5269-0851-405f-a9a5-f9d2741e8500-metrics-tls\") pod \"dns-default-knfp2\" (UID: \"21ee5269-0851-405f-a9a5-f9d2741e8500\") " pod="openshift-dns/dns-default-knfp2"
Sep 08 13:16:25.889340 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:16:25.888746    2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 13:16:25.889340 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:16:25.888820    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21ee5269-0851-405f-a9a5-f9d2741e8500-metrics-tls podName:21ee5269-0851-405f-a9a5-f9d2741e8500 nodeName:}" failed. No retries permitted until 2026-09-08 13:16:27.888802141 +0000 UTC m=+38.160250068 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/21ee5269-0851-405f-a9a5-f9d2741e8500-metrics-tls") pod "dns-default-knfp2" (UID: "21ee5269-0851-405f-a9a5-f9d2741e8500") : secret "dns-default-metrics-tls" not found
Sep 08 13:16:26.355312 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:26.355236    2573 generic.go:356] "Generic (PLEG): container finished" podID="8e45750b-bf27-4d49-8eba-1ba76fadd028" containerID="9f079d42e6169f9e3ddd2b593baac48cfec20b4db653d7cec76a182817fb4fe9" exitCode=0
Sep 08 13:16:26.355312 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:26.355295    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-krqr9" event={"ID":"8e45750b-bf27-4d49-8eba-1ba76fadd028","Type":"ContainerDied","Data":"9f079d42e6169f9e3ddd2b593baac48cfec20b4db653d7cec76a182817fb4fe9"}
Sep 08 13:16:27.360354 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:27.360325    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-krqr9" event={"ID":"8e45750b-bf27-4d49-8eba-1ba76fadd028","Type":"ContainerStarted","Data":"a3cb9399def33c128d6f3e4782ea326e274c0bd068279599a7dc06c45d851408"}
Sep 08 13:16:27.381565 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:27.381495    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-krqr9" podStartSLOduration=7.942047301 podStartE2EDuration="37.38147544s" totalImagesPullingTime="29.439428139s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 13:15:50 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:16:27.380302733 +0000 UTC m=+37.651750664" watchObservedRunningTime="2026-09-08 13:16:27.38147544 +0000 UTC m=+37.652923372"
Sep 08 13:16:27.502642 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:27.502616    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:16:27.609618 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:27.609452    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:16:27.904188 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:27.904153    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/21ee5269-0851-405f-a9a5-f9d2741e8500-metrics-tls\") pod \"dns-default-knfp2\" (UID: \"21ee5269-0851-405f-a9a5-f9d2741e8500\") " pod="openshift-dns/dns-default-knfp2"
Sep 08 13:16:27.904343 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:16:27.904305    2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 13:16:27.904394 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:16:27.904367    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21ee5269-0851-405f-a9a5-f9d2741e8500-metrics-tls podName:21ee5269-0851-405f-a9a5-f9d2741e8500 nodeName:}" failed. No retries permitted until 2026-09-08 13:16:31.904349484 +0000 UTC m=+42.175797412 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/21ee5269-0851-405f-a9a5-f9d2741e8500-metrics-tls") pod "dns-default-knfp2" (UID: "21ee5269-0851-405f-a9a5-f9d2741e8500") : secret "dns-default-metrics-tls" not found
Sep 08 13:16:28.364273 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:28.364232    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-qmrbg" event={"ID":"831913e7-ea98-4d0b-bbaf-411e16552f73","Type":"ContainerStarted","Data":"84acd609f96d1eaff9e5c5e215101f497ebf90b59ec9ef745336760f02754916"}
Sep 08 13:16:28.382569 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:28.382531    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-qmrbg" podStartSLOduration=1.768655595 podStartE2EDuration="4.382515933s" totalImagesPullingTime="2.613860338s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:16:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:16:28.381612854 +0000 UTC m=+38.653060802" watchObservedRunningTime="2026-09-08 13:16:28.382515933 +0000 UTC m=+38.653963855"
Sep 08 13:16:28.961434 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:28.961368    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:16:29.062272 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:29.062240    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:16:30.201596 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:30.201567    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:16:30.285397 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:30.285366    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:16:30.385865 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:30.385841    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:16:31.927977 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:31.927940    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/21ee5269-0851-405f-a9a5-f9d2741e8500-metrics-tls\") pod \"dns-default-knfp2\" (UID: \"21ee5269-0851-405f-a9a5-f9d2741e8500\") " pod="openshift-dns/dns-default-knfp2"
Sep 08 13:16:31.928304 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:16:31.928065    2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 13:16:31.928304 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:16:31.928122    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21ee5269-0851-405f-a9a5-f9d2741e8500-metrics-tls podName:21ee5269-0851-405f-a9a5-f9d2741e8500 nodeName:}" failed. No retries permitted until 2026-09-08 13:16:39.928106002 +0000 UTC m=+50.199553912 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/21ee5269-0851-405f-a9a5-f9d2741e8500-metrics-tls") pod "dns-default-knfp2" (UID: "21ee5269-0851-405f-a9a5-f9d2741e8500") : secret "dns-default-metrics-tls" not found
Sep 08 13:16:32.437352 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:32.437317    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:16:32.520646 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:32.520617    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:16:35.215624 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:35.215586    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:16:35.517661 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:35.517596    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:16:35.604308 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:35.604283    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:16:39.980255 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:39.980222    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/21ee5269-0851-405f-a9a5-f9d2741e8500-metrics-tls\") pod \"dns-default-knfp2\" (UID: \"21ee5269-0851-405f-a9a5-f9d2741e8500\") " pod="openshift-dns/dns-default-knfp2"
Sep 08 13:16:39.980695 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:16:39.980327    2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 13:16:39.980695 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:16:39.980381    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21ee5269-0851-405f-a9a5-f9d2741e8500-metrics-tls podName:21ee5269-0851-405f-a9a5-f9d2741e8500 nodeName:}" failed. No retries permitted until 2026-09-08 13:16:55.980366606 +0000 UTC m=+66.251814516 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/21ee5269-0851-405f-a9a5-f9d2741e8500-metrics-tls") pod "dns-default-knfp2" (UID: "21ee5269-0851-405f-a9a5-f9d2741e8500") : secret "dns-default-metrics-tls" not found
Sep 08 13:16:42.484882 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:42.484849    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:16:42.558964 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:42.558931    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:16:47.318176 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:47.318141    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:16:47.418656 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:47.418628    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:16:47.900102 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:47.900066    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:16:47.980948 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:47.980920    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:16:52.316473 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:52.316446    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8h8cn"
Sep 08 13:16:52.478645 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:52.478602    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:16:52.548021 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:52.547993    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:16:55.886767 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:55.886734    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4trk\" (UniqueName: \"kubernetes.io/projected/0be74f6e-cde6-42fd-8772-4fca7f6bab00-kube-api-access-m4trk\") pod \"network-check-target-mwdn6\" (UID: \"0be74f6e-cde6-42fd-8772-4fca7f6bab00\") " pod="openshift-network-diagnostics/network-check-target-mwdn6"
Sep 08 13:16:55.886767 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:55.886771    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/217a8762-14bb-4990-8a53-3b3bcd6984ac-metrics-certs\") pod \"network-metrics-daemon-kpscq\" (UID: \"217a8762-14bb-4990-8a53-3b3bcd6984ac\") " pod="openshift-multus/network-metrics-daemon-kpscq"
Sep 08 13:16:55.887212 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:55.886802    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9e7df160-d688-4877-895e-edc5f73db16a-original-pull-secret\") pod \"global-pull-secret-syncer-6qqfz\" (UID: \"9e7df160-d688-4877-895e-edc5f73db16a\") " pod="kube-system/global-pull-secret-syncer-6qqfz"
Sep 08 13:16:55.889183 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:55.889164    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 08 13:16:55.889246 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:55.889230    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 08 13:16:55.889292 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:55.889277    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 08 13:16:55.897928 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:16:55.897908    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 08 13:16:55.898022 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:16:55.897973    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/217a8762-14bb-4990-8a53-3b3bcd6984ac-metrics-certs podName:217a8762-14bb-4990-8a53-3b3bcd6984ac nodeName:}" failed. No retries permitted until 2026-09-08 13:17:59.897953717 +0000 UTC m=+130.169401649 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/217a8762-14bb-4990-8a53-3b3bcd6984ac-metrics-certs") pod "network-metrics-daemon-kpscq" (UID: "217a8762-14bb-4990-8a53-3b3bcd6984ac") : secret "metrics-daemon-secret" not found
Sep 08 13:16:55.900270 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:55.900251    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 08 13:16:55.900847 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:55.900832    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9e7df160-d688-4877-895e-edc5f73db16a-original-pull-secret\") pod \"global-pull-secret-syncer-6qqfz\" (UID: \"9e7df160-d688-4877-895e-edc5f73db16a\") " pod="kube-system/global-pull-secret-syncer-6qqfz"
Sep 08 13:16:55.910751 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:55.910730    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4trk\" (UniqueName: \"kubernetes.io/projected/0be74f6e-cde6-42fd-8772-4fca7f6bab00-kube-api-access-m4trk\") pod \"network-check-target-mwdn6\" (UID: \"0be74f6e-cde6-42fd-8772-4fca7f6bab00\") " pod="openshift-network-diagnostics/network-check-target-mwdn6"
Sep 08 13:16:55.987814 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:55.987785    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/21ee5269-0851-405f-a9a5-f9d2741e8500-metrics-tls\") pod \"dns-default-knfp2\" (UID: \"21ee5269-0851-405f-a9a5-f9d2741e8500\") " pod="openshift-dns/dns-default-knfp2"
Sep 08 13:16:55.987951 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:16:55.987896    2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 13:16:55.988011 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:16:55.987958    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21ee5269-0851-405f-a9a5-f9d2741e8500-metrics-tls podName:21ee5269-0851-405f-a9a5-f9d2741e8500 nodeName:}" failed. No retries permitted until 2026-09-08 13:17:27.987941587 +0000 UTC m=+98.259389511 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/21ee5269-0851-405f-a9a5-f9d2741e8500-metrics-tls") pod "dns-default-knfp2" (UID: "21ee5269-0851-405f-a9a5-f9d2741e8500") : secret "dns-default-metrics-tls" not found
Sep 08 13:16:56.034895 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:56.034871    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-x5pb6\""
Sep 08 13:16:56.043039 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:56.043019    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mwdn6"
Sep 08 13:16:56.083487 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:56.083468    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6qqfz"
Sep 08 13:16:56.343385 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:56.343358    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-mwdn6"]
Sep 08 13:16:56.346278 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:16:56.346251    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0be74f6e_cde6_42fd_8772_4fca7f6bab00.slice/crio-5cf7be839caf48d28eab73e2a1b48e923a5433ef2313e0c7a84d13a2f81a972a WatchSource:0}: Error finding container 5cf7be839caf48d28eab73e2a1b48e923a5433ef2313e0c7a84d13a2f81a972a: Status 404 returned error can't find the container with id 5cf7be839caf48d28eab73e2a1b48e923a5433ef2313e0c7a84d13a2f81a972a
Sep 08 13:16:56.376991 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:56.376968    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-6qqfz"]
Sep 08 13:16:56.379314 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:16:56.379289    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e7df160_d688_4877_895e_edc5f73db16a.slice/crio-4c24473d1eb48f5f71ac884352718f54d3413ff99c56bacd3a7ddc385ddb220b WatchSource:0}: Error finding container 4c24473d1eb48f5f71ac884352718f54d3413ff99c56bacd3a7ddc385ddb220b: Status 404 returned error can't find the container with id 4c24473d1eb48f5f71ac884352718f54d3413ff99c56bacd3a7ddc385ddb220b
Sep 08 13:16:56.426877 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:56.426814    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-6qqfz" event={"ID":"9e7df160-d688-4877-895e-edc5f73db16a","Type":"ContainerStarted","Data":"4c24473d1eb48f5f71ac884352718f54d3413ff99c56bacd3a7ddc385ddb220b"}
Sep 08 13:16:56.427836 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:16:56.427818    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-mwdn6" event={"ID":"0be74f6e-cde6-42fd-8772-4fca7f6bab00","Type":"ContainerStarted","Data":"5cf7be839caf48d28eab73e2a1b48e923a5433ef2313e0c7a84d13a2f81a972a"}
Sep 08 13:17:01.442394 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:17:01.442359    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-mwdn6" event={"ID":"0be74f6e-cde6-42fd-8772-4fca7f6bab00","Type":"ContainerStarted","Data":"8a2109a61516532ae7ebc88cb118d57fde95fd0481c02838e9b6f035b485af01"}
Sep 08 13:17:01.442912 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:17:01.442486    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-mwdn6"
Sep 08 13:17:01.443962 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:17:01.443938    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-6qqfz" event={"ID":"9e7df160-d688-4877-895e-edc5f73db16a","Type":"ContainerStarted","Data":"b9acaca47c4144dfad260c00cf1db96d8146ec2d0c8ac959dcc62911cc00a9d0"}
Sep 08 13:17:01.457277 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:17:01.457231    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-mwdn6" podStartSLOduration=66.790944702 podStartE2EDuration="1m11.457218109s" totalImagesPullingTime="4.666273407s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:15:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:17:01.456459292 +0000 UTC m=+71.727907223" watchObservedRunningTime="2026-09-08 13:17:01.457218109 +0000 UTC m=+71.728666041"
Sep 08 13:17:01.470789 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:17:01.470752    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-6qqfz" podStartSLOduration=66.832779912 podStartE2EDuration="1m11.47074439s" totalImagesPullingTime="4.637964478s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 13:15:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:17:01.47060677 +0000 UTC m=+71.742054725" watchObservedRunningTime="2026-09-08 13:17:01.47074439 +0000 UTC m=+71.742192320"
Sep 08 13:17:08.310627 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:17:08.310586    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:17:08.433696 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:17:08.433662    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:17:27.993078 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:17:27.993037    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/21ee5269-0851-405f-a9a5-f9d2741e8500-metrics-tls\") pod \"dns-default-knfp2\" (UID: \"21ee5269-0851-405f-a9a5-f9d2741e8500\") " pod="openshift-dns/dns-default-knfp2"
Sep 08 13:17:27.993567 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:17:27.993164    2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 13:17:27.993567 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:17:27.993216    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21ee5269-0851-405f-a9a5-f9d2741e8500-metrics-tls podName:21ee5269-0851-405f-a9a5-f9d2741e8500 nodeName:}" failed. No retries permitted until 2026-09-08 13:18:31.993202545 +0000 UTC m=+162.264650454 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/21ee5269-0851-405f-a9a5-f9d2741e8500-metrics-tls") pod "dns-default-knfp2" (UID: "21ee5269-0851-405f-a9a5-f9d2741e8500") : secret "dns-default-metrics-tls" not found
Sep 08 13:17:32.448462 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:17:32.448433    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-mwdn6"
Sep 08 13:17:36.080145 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:17:36.080111    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:17:59.995111 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:17:59.995072    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/217a8762-14bb-4990-8a53-3b3bcd6984ac-metrics-certs\") pod \"network-metrics-daemon-kpscq\" (UID: \"217a8762-14bb-4990-8a53-3b3bcd6984ac\") " pod="openshift-multus/network-metrics-daemon-kpscq"
Sep 08 13:17:59.995609 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:17:59.995192    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 08 13:17:59.995609 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:17:59.995247    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/217a8762-14bb-4990-8a53-3b3bcd6984ac-metrics-certs podName:217a8762-14bb-4990-8a53-3b3bcd6984ac nodeName:}" failed. No retries permitted until 2026-09-08 13:20:01.995233133 +0000 UTC m=+252.266681041 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/217a8762-14bb-4990-8a53-3b3bcd6984ac-metrics-certs") pod "network-metrics-daemon-kpscq" (UID: "217a8762-14bb-4990-8a53-3b3bcd6984ac") : secret "metrics-daemon-secret" not found
Sep 08 13:18:00.136003 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:00.135973    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-bqkz9"]
Sep 08 13:18:00.137946 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:00.137928    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-685456655f-bqkz9"
Sep 08 13:18:00.139932 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:00.139903    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-2nr9r\""
Sep 08 13:18:00.144116 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:00.144072    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-69549dc55b-qs44g"]
Sep 08 13:18:00.146058 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:00.146029    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-69549dc55b-qs44g"
Sep 08 13:18:00.146747 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:00.146729    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-bqkz9"]
Sep 08 13:18:00.147943 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:00.147917    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-bchgb\""
Sep 08 13:18:00.148250 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:00.148224    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Sep 08 13:18:00.148334 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:00.148255    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Sep 08 13:18:00.148334 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:00.148259    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Sep 08 13:18:00.148467 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:00.148416    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 08 13:18:00.148534 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:00.148522    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Sep 08 13:18:00.149186 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:00.149162    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 08 13:18:00.166583 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:00.166559    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-69549dc55b-qs44g"]
Sep 08 13:18:00.195952 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:00.195913    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwq56\" (UniqueName: \"kubernetes.io/projected/871377db-9869-404a-b14a-343321527fdb-kube-api-access-dwq56\") pod \"router-default-69549dc55b-qs44g\" (UID: \"871377db-9869-404a-b14a-343321527fdb\") " pod="openshift-ingress/router-default-69549dc55b-qs44g"
Sep 08 13:18:00.195952 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:00.195953    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9sm4k\" (UniqueName: \"kubernetes.io/projected/9d3543a6-4d6f-4454-8ffd-69b8248df0e1-kube-api-access-9sm4k\") pod \"network-check-source-685456655f-bqkz9\" (UID: \"9d3543a6-4d6f-4454-8ffd-69b8248df0e1\") " pod="openshift-network-diagnostics/network-check-source-685456655f-bqkz9"
Sep 08 13:18:00.196135 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:00.195974    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/871377db-9869-404a-b14a-343321527fdb-default-certificate\") pod \"router-default-69549dc55b-qs44g\" (UID: \"871377db-9869-404a-b14a-343321527fdb\") " pod="openshift-ingress/router-default-69549dc55b-qs44g"
Sep 08 13:18:00.196135 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:00.196051    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/871377db-9869-404a-b14a-343321527fdb-metrics-certs\") pod \"router-default-69549dc55b-qs44g\" (UID: \"871377db-9869-404a-b14a-343321527fdb\") " pod="openshift-ingress/router-default-69549dc55b-qs44g"
Sep 08 13:18:00.196135 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:00.196111    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/871377db-9869-404a-b14a-343321527fdb-service-ca-bundle\") pod \"router-default-69549dc55b-qs44g\" (UID: \"871377db-9869-404a-b14a-343321527fdb\") " pod="openshift-ingress/router-default-69549dc55b-qs44g"
Sep 08 13:18:00.196228 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:00.196135    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/871377db-9869-404a-b14a-343321527fdb-stats-auth\") pod \"router-default-69549dc55b-qs44g\" (UID: \"871377db-9869-404a-b14a-343321527fdb\") " pod="openshift-ingress/router-default-69549dc55b-qs44g"
Sep 08 13:18:00.240011 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:00.239984    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-wfwsl"]
Sep 08 13:18:00.242091 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:00.242073    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-wfwsl"
Sep 08 13:18:00.244761 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:00.244744    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Sep 08 13:18:00.246020 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:00.245967    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-bnf42\""
Sep 08 13:18:00.249357 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:00.249342    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Sep 08 13:18:00.253514 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:00.253479    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-wfwsl"]
Sep 08 13:18:00.273466 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:00.273445    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64bfdb6bcc-6lc4p"]
Sep 08 13:18:00.275385 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:00.275371    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64bfdb6bcc-6lc4p"
Sep 08 13:18:00.277539 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:00.277494    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 08 13:18:00.277664 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:00.277572    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 08 13:18:00.277664 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:00.277615    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 08 13:18:00.277826 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:00.277665    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 08 13:18:00.277912 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:00.277883    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 08 13:18:00.278233 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:00.278199    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 08 13:18:00.278233 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:00.278219    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 08 13:18:00.289941 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:00.289921    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64bfdb6bcc-6lc4p"]
Sep 08 13:18:00.297294 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:00.297274    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/871377db-9869-404a-b14a-343321527fdb-service-ca-bundle\") pod \"router-default-69549dc55b-qs44g\" (UID: \"871377db-9869-404a-b14a-343321527fdb\") " pod="openshift-ingress/router-default-69549dc55b-qs44g"
Sep 08 13:18:00.297390 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:00.297301    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/871377db-9869-404a-b14a-343321527fdb-stats-auth\") pod \"router-default-69549dc55b-qs44g\" (UID: \"871377db-9869-404a-b14a-343321527fdb\") " pod="openshift-ingress/router-default-69549dc55b-qs44g"
Sep 08 13:18:00.297390 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:00.297324    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ec967b4f-3738-4934-84bc-2f6ab72f8a24-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-wfwsl\" (UID: \"ec967b4f-3738-4934-84bc-2f6ab72f8a24\") " pod="openshift-network-console/networking-console-plugin-79977f9844-wfwsl"
Sep 08 13:18:00.297390 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:00.297351    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwq56\" (UniqueName: \"kubernetes.io/projected/871377db-9869-404a-b14a-343321527fdb-kube-api-access-dwq56\") pod \"router-default-69549dc55b-qs44g\" (UID: \"871377db-9869-404a-b14a-343321527fdb\") " pod="openshift-ingress/router-default-69549dc55b-qs44g"
Sep 08 13:18:00.297390 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:00.297375    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sm4k\" (UniqueName: \"kubernetes.io/projected/9d3543a6-4d6f-4454-8ffd-69b8248df0e1-kube-api-access-9sm4k\") pod \"network-check-source-685456655f-bqkz9\" (UID: \"9d3543a6-4d6f-4454-8ffd-69b8248df0e1\") " pod="openshift-network-diagnostics/network-check-source-685456655f-bqkz9"
Sep 08 13:18:00.297628 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:00.297422    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/871377db-9869-404a-b14a-343321527fdb-default-certificate\") pod \"router-default-69549dc55b-qs44g\" (UID: \"871377db-9869-404a-b14a-343321527fdb\") " pod="openshift-ingress/router-default-69549dc55b-qs44g"
Sep 08 13:18:00.297628 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:18:00.297450    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/871377db-9869-404a-b14a-343321527fdb-service-ca-bundle podName:871377db-9869-404a-b14a-343321527fdb nodeName:}" failed. No retries permitted until 2026-09-08 13:18:00.797430691 +0000 UTC m=+131.068878603 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/871377db-9869-404a-b14a-343321527fdb-service-ca-bundle") pod "router-default-69549dc55b-qs44g" (UID: "871377db-9869-404a-b14a-343321527fdb") : configmap references non-existent config key: service-ca.crt
Sep 08 13:18:00.297628 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:00.297534    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/871377db-9869-404a-b14a-343321527fdb-metrics-certs\") pod \"router-default-69549dc55b-qs44g\" (UID: \"871377db-9869-404a-b14a-343321527fdb\") " pod="openshift-ingress/router-default-69549dc55b-qs44g"
Sep 08 13:18:00.297628 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:00.297563    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/ec967b4f-3738-4934-84bc-2f6ab72f8a24-nginx-conf\") pod \"networking-console-plugin-79977f9844-wfwsl\" (UID: \"ec967b4f-3738-4934-84bc-2f6ab72f8a24\") " pod="openshift-network-console/networking-console-plugin-79977f9844-wfwsl"
Sep 08 13:18:00.297628 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:18:00.297603    2573 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 08 13:18:00.297843 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:18:00.297675    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/871377db-9869-404a-b14a-343321527fdb-metrics-certs podName:871377db-9869-404a-b14a-343321527fdb nodeName:}" failed. No retries permitted until 2026-09-08 13:18:00.79765508 +0000 UTC m=+131.069102992 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/871377db-9869-404a-b14a-343321527fdb-metrics-certs") pod "router-default-69549dc55b-qs44g" (UID: "871377db-9869-404a-b14a-343321527fdb") : secret "router-metrics-certs-default" not found
Sep 08 13:18:00.299604 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:00.299580    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/871377db-9869-404a-b14a-343321527fdb-stats-auth\") pod \"router-default-69549dc55b-qs44g\" (UID: \"871377db-9869-404a-b14a-343321527fdb\") " pod="openshift-ingress/router-default-69549dc55b-qs44g"
Sep 08 13:18:00.299700 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:00.299664    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/871377db-9869-404a-b14a-343321527fdb-default-certificate\") pod \"router-default-69549dc55b-qs44g\" (UID: \"871377db-9869-404a-b14a-343321527fdb\") " pod="openshift-ingress/router-default-69549dc55b-qs44g"
Sep 08 13:18:00.313679 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:00.313656    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwq56\" (UniqueName: \"kubernetes.io/projected/871377db-9869-404a-b14a-343321527fdb-kube-api-access-dwq56\") pod \"router-default-69549dc55b-qs44g\" (UID: \"871377db-9869-404a-b14a-343321527fdb\") " pod="openshift-ingress/router-default-69549dc55b-qs44g"
Sep 08 13:18:00.314458 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:00.314443    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9sm4k\" (UniqueName: \"kubernetes.io/projected/9d3543a6-4d6f-4454-8ffd-69b8248df0e1-kube-api-access-9sm4k\") pod \"network-check-source-685456655f-bqkz9\" (UID: \"9d3543a6-4d6f-4454-8ffd-69b8248df0e1\") " pod="openshift-network-diagnostics/network-check-source-685456655f-bqkz9"
Sep 08 13:18:00.397868 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:00.397845    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnchl\" (UniqueName: \"kubernetes.io/projected/ffc5700f-1e9f-47a0-a0f0-98fc161b51e5-kube-api-access-cnchl\") pod \"cluster-proxy-proxy-agent-64bfdb6bcc-6lc4p\" (UID: \"ffc5700f-1e9f-47a0-a0f0-98fc161b51e5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64bfdb6bcc-6lc4p"
Sep 08 13:18:00.398019 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:00.397880    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/ec967b4f-3738-4934-84bc-2f6ab72f8a24-nginx-conf\") pod \"networking-console-plugin-79977f9844-wfwsl\" (UID: \"ec967b4f-3738-4934-84bc-2f6ab72f8a24\") " pod="openshift-network-console/networking-console-plugin-79977f9844-wfwsl"
Sep 08 13:18:00.398019 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:00.397912    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffc5700f-1e9f-47a0-a0f0-98fc161b51e5-additional-service-ca\") pod \"cluster-proxy-proxy-agent-64bfdb6bcc-6lc4p\" (UID: \"ffc5700f-1e9f-47a0-a0f0-98fc161b51e5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64bfdb6bcc-6lc4p"
Sep 08 13:18:00.398019 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:00.398007    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/ffc5700f-1e9f-47a0-a0f0-98fc161b51e5-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-64bfdb6bcc-6lc4p\" (UID: \"ffc5700f-1e9f-47a0-a0f0-98fc161b51e5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64bfdb6bcc-6lc4p"
Sep 08 13:18:00.398191 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:00.398043    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ec967b4f-3738-4934-84bc-2f6ab72f8a24-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-wfwsl\" (UID: \"ec967b4f-3738-4934-84bc-2f6ab72f8a24\") " pod="openshift-network-console/networking-console-plugin-79977f9844-wfwsl"
Sep 08 13:18:00.398191 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:00.398095    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/ffc5700f-1e9f-47a0-a0f0-98fc161b51e5-ca\") pod \"cluster-proxy-proxy-agent-64bfdb6bcc-6lc4p\" (UID: \"ffc5700f-1e9f-47a0-a0f0-98fc161b51e5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64bfdb6bcc-6lc4p"
Sep 08 13:18:00.398191 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:00.398130    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/ffc5700f-1e9f-47a0-a0f0-98fc161b51e5-hub\") pod \"cluster-proxy-proxy-agent-64bfdb6bcc-6lc4p\" (UID: \"ffc5700f-1e9f-47a0-a0f0-98fc161b51e5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64bfdb6bcc-6lc4p"
Sep 08 13:18:00.398191 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:18:00.398141    2573 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 08 13:18:00.398191 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:00.398156    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ffc5700f-1e9f-47a0-a0f0-98fc161b51e5-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-64bfdb6bcc-6lc4p\" (UID: \"ffc5700f-1e9f-47a0-a0f0-98fc161b51e5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64bfdb6bcc-6lc4p"
Sep 08 13:18:00.398434 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:18:00.398204    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec967b4f-3738-4934-84bc-2f6ab72f8a24-networking-console-plugin-cert podName:ec967b4f-3738-4934-84bc-2f6ab72f8a24 nodeName:}" failed. No retries permitted until 2026-09-08 13:18:00.898185149 +0000 UTC m=+131.169633069 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/ec967b4f-3738-4934-84bc-2f6ab72f8a24-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-wfwsl" (UID: "ec967b4f-3738-4934-84bc-2f6ab72f8a24") : secret "networking-console-plugin-cert" not found
Sep 08 13:18:00.398667 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:00.398646    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/ec967b4f-3738-4934-84bc-2f6ab72f8a24-nginx-conf\") pod \"networking-console-plugin-79977f9844-wfwsl\" (UID: \"ec967b4f-3738-4934-84bc-2f6ab72f8a24\") " pod="openshift-network-console/networking-console-plugin-79977f9844-wfwsl"
Sep 08 13:18:00.446113 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:00.446085    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-685456655f-bqkz9"
Sep 08 13:18:00.446876 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:00.446853    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:18:00.498619 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:00.498559    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffc5700f-1e9f-47a0-a0f0-98fc161b51e5-additional-service-ca\") pod \"cluster-proxy-proxy-agent-64bfdb6bcc-6lc4p\" (UID: \"ffc5700f-1e9f-47a0-a0f0-98fc161b51e5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64bfdb6bcc-6lc4p"
Sep 08 13:18:00.498619 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:00.498602    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/ffc5700f-1e9f-47a0-a0f0-98fc161b51e5-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-64bfdb6bcc-6lc4p\" (UID: \"ffc5700f-1e9f-47a0-a0f0-98fc161b51e5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64bfdb6bcc-6lc4p"
Sep 08 13:18:00.498844 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:00.498666    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/ffc5700f-1e9f-47a0-a0f0-98fc161b51e5-ca\") pod \"cluster-proxy-proxy-agent-64bfdb6bcc-6lc4p\" (UID: \"ffc5700f-1e9f-47a0-a0f0-98fc161b51e5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64bfdb6bcc-6lc4p"
Sep 08 13:18:00.498844 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:00.498693    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/ffc5700f-1e9f-47a0-a0f0-98fc161b51e5-hub\") pod \"cluster-proxy-proxy-agent-64bfdb6bcc-6lc4p\" (UID: \"ffc5700f-1e9f-47a0-a0f0-98fc161b51e5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64bfdb6bcc-6lc4p"
Sep 08 13:18:00.498844 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:00.498719    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ffc5700f-1e9f-47a0-a0f0-98fc161b51e5-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-64bfdb6bcc-6lc4p\" (UID: \"ffc5700f-1e9f-47a0-a0f0-98fc161b51e5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64bfdb6bcc-6lc4p"
Sep 08 13:18:00.498844 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:00.498796    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnchl\" (UniqueName: \"kubernetes.io/projected/ffc5700f-1e9f-47a0-a0f0-98fc161b51e5-kube-api-access-cnchl\") pod \"cluster-proxy-proxy-agent-64bfdb6bcc-6lc4p\" (UID: \"ffc5700f-1e9f-47a0-a0f0-98fc161b51e5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64bfdb6bcc-6lc4p"
Sep 08 13:18:00.499548 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:00.499366    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffc5700f-1e9f-47a0-a0f0-98fc161b51e5-additional-service-ca\") pod \"cluster-proxy-proxy-agent-64bfdb6bcc-6lc4p\" (UID: \"ffc5700f-1e9f-47a0-a0f0-98fc161b51e5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64bfdb6bcc-6lc4p"
Sep 08 13:18:00.501983 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:00.501863    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/ffc5700f-1e9f-47a0-a0f0-98fc161b51e5-ca\") pod \"cluster-proxy-proxy-agent-64bfdb6bcc-6lc4p\" (UID: \"ffc5700f-1e9f-47a0-a0f0-98fc161b51e5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64bfdb6bcc-6lc4p"
Sep 08 13:18:00.501983 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:00.501940    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/ffc5700f-1e9f-47a0-a0f0-98fc161b51e5-hub\") pod \"cluster-proxy-proxy-agent-64bfdb6bcc-6lc4p\" (UID: \"ffc5700f-1e9f-47a0-a0f0-98fc161b51e5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64bfdb6bcc-6lc4p"
Sep 08 13:18:00.502723 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:00.502676    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ffc5700f-1e9f-47a0-a0f0-98fc161b51e5-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-64bfdb6bcc-6lc4p\" (UID: \"ffc5700f-1e9f-47a0-a0f0-98fc161b51e5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64bfdb6bcc-6lc4p"
Sep 08 13:18:00.503026 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:00.503001    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/ffc5700f-1e9f-47a0-a0f0-98fc161b51e5-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-64bfdb6bcc-6lc4p\" (UID: \"ffc5700f-1e9f-47a0-a0f0-98fc161b51e5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64bfdb6bcc-6lc4p"
Sep 08 13:18:00.508368 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:00.508346    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnchl\" (UniqueName: \"kubernetes.io/projected/ffc5700f-1e9f-47a0-a0f0-98fc161b51e5-kube-api-access-cnchl\") pod \"cluster-proxy-proxy-agent-64bfdb6bcc-6lc4p\" (UID: \"ffc5700f-1e9f-47a0-a0f0-98fc161b51e5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64bfdb6bcc-6lc4p"
Sep 08 13:18:00.547742 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:00.547720    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:18:00.592494 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:00.592473    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64bfdb6bcc-6lc4p"
Sep 08 13:18:00.708833 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:00.708807    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-bqkz9"]
Sep 08 13:18:00.711403 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:18:00.711382    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d3543a6_4d6f_4454_8ffd_69b8248df0e1.slice/crio-6199caf9b2a93732e8f537ad8ad4106e05a10d9ed0989a1bf28e92606c808efe WatchSource:0}: Error finding container 6199caf9b2a93732e8f537ad8ad4106e05a10d9ed0989a1bf28e92606c808efe: Status 404 returned error can't find the container with id 6199caf9b2a93732e8f537ad8ad4106e05a10d9ed0989a1bf28e92606c808efe
Sep 08 13:18:00.801320 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:00.801292    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/871377db-9869-404a-b14a-343321527fdb-service-ca-bundle\") pod \"router-default-69549dc55b-qs44g\" (UID: \"871377db-9869-404a-b14a-343321527fdb\") " pod="openshift-ingress/router-default-69549dc55b-qs44g"
Sep 08 13:18:00.801456 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:00.801434    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/871377db-9869-404a-b14a-343321527fdb-metrics-certs\") pod \"router-default-69549dc55b-qs44g\" (UID: \"871377db-9869-404a-b14a-343321527fdb\") " pod="openshift-ingress/router-default-69549dc55b-qs44g"
Sep 08 13:18:00.801542 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:18:00.801470    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/871377db-9869-404a-b14a-343321527fdb-service-ca-bundle podName:871377db-9869-404a-b14a-343321527fdb nodeName:}" failed. No retries permitted until 2026-09-08 13:18:01.801445969 +0000 UTC m=+132.072893893 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/871377db-9869-404a-b14a-343321527fdb-service-ca-bundle") pod "router-default-69549dc55b-qs44g" (UID: "871377db-9869-404a-b14a-343321527fdb") : configmap references non-existent config key: service-ca.crt
Sep 08 13:18:00.801542 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:18:00.801525    2573 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 08 13:18:00.801620 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:18:00.801577    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/871377db-9869-404a-b14a-343321527fdb-metrics-certs podName:871377db-9869-404a-b14a-343321527fdb nodeName:}" failed. No retries permitted until 2026-09-08 13:18:01.801562054 +0000 UTC m=+132.073009965 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/871377db-9869-404a-b14a-343321527fdb-metrics-certs") pod "router-default-69549dc55b-qs44g" (UID: "871377db-9869-404a-b14a-343321527fdb") : secret "router-metrics-certs-default" not found
Sep 08 13:18:00.859135 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:00.859108    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64bfdb6bcc-6lc4p"]
Sep 08 13:18:00.862211 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:18:00.862188    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podffc5700f_1e9f_47a0_a0f0_98fc161b51e5.slice/crio-83f844a4727f37be351488e306997cd210f9b292b9e0bb837755de233acc79af WatchSource:0}: Error finding container 83f844a4727f37be351488e306997cd210f9b292b9e0bb837755de233acc79af: Status 404 returned error can't find the container with id 83f844a4727f37be351488e306997cd210f9b292b9e0bb837755de233acc79af
Sep 08 13:18:00.902624 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:00.902601    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ec967b4f-3738-4934-84bc-2f6ab72f8a24-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-wfwsl\" (UID: \"ec967b4f-3738-4934-84bc-2f6ab72f8a24\") " pod="openshift-network-console/networking-console-plugin-79977f9844-wfwsl"
Sep 08 13:18:00.902754 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:18:00.902707    2573 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 08 13:18:00.902796 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:18:00.902754    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec967b4f-3738-4934-84bc-2f6ab72f8a24-networking-console-plugin-cert podName:ec967b4f-3738-4934-84bc-2f6ab72f8a24 nodeName:}" failed. No retries permitted until 2026-09-08 13:18:01.902739069 +0000 UTC m=+132.174186998 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/ec967b4f-3738-4934-84bc-2f6ab72f8a24-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-wfwsl" (UID: "ec967b4f-3738-4934-84bc-2f6ab72f8a24") : secret "networking-console-plugin-cert" not found
Sep 08 13:18:01.584287 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:01.584249    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64bfdb6bcc-6lc4p" event={"ID":"ffc5700f-1e9f-47a0-a0f0-98fc161b51e5","Type":"ContainerStarted","Data":"83f844a4727f37be351488e306997cd210f9b292b9e0bb837755de233acc79af"}
Sep 08 13:18:01.585635 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:01.585608    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-685456655f-bqkz9" event={"ID":"9d3543a6-4d6f-4454-8ffd-69b8248df0e1","Type":"ContainerStarted","Data":"456a541e91fe69a6a519f78dad2639787b92ebd05aecaa0eb5590bda45dfd5d5"}
Sep 08 13:18:01.585748 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:01.585654    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-685456655f-bqkz9" event={"ID":"9d3543a6-4d6f-4454-8ffd-69b8248df0e1","Type":"ContainerStarted","Data":"6199caf9b2a93732e8f537ad8ad4106e05a10d9ed0989a1bf28e92606c808efe"}
Sep 08 13:18:01.601674 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:01.601632    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-685456655f-bqkz9" podStartSLOduration=1.601617104 podStartE2EDuration="1.601617104s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:18:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:18:01.600091949 +0000 UTC m=+131.871539880" watchObservedRunningTime="2026-09-08 13:18:01.601617104 +0000 UTC m=+131.873065034"
Sep 08 13:18:01.808554 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:01.808483    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/871377db-9869-404a-b14a-343321527fdb-metrics-certs\") pod \"router-default-69549dc55b-qs44g\" (UID: \"871377db-9869-404a-b14a-343321527fdb\") " pod="openshift-ingress/router-default-69549dc55b-qs44g"
Sep 08 13:18:01.808554 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:01.808557    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/871377db-9869-404a-b14a-343321527fdb-service-ca-bundle\") pod \"router-default-69549dc55b-qs44g\" (UID: \"871377db-9869-404a-b14a-343321527fdb\") " pod="openshift-ingress/router-default-69549dc55b-qs44g"
Sep 08 13:18:01.808753 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:18:01.808655    2573 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 08 13:18:01.808753 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:18:01.808710    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/871377db-9869-404a-b14a-343321527fdb-service-ca-bundle podName:871377db-9869-404a-b14a-343321527fdb nodeName:}" failed. No retries permitted until 2026-09-08 13:18:03.808691921 +0000 UTC m=+134.080139844 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/871377db-9869-404a-b14a-343321527fdb-service-ca-bundle") pod "router-default-69549dc55b-qs44g" (UID: "871377db-9869-404a-b14a-343321527fdb") : configmap references non-existent config key: service-ca.crt
Sep 08 13:18:01.808753 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:18:01.808735    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/871377db-9869-404a-b14a-343321527fdb-metrics-certs podName:871377db-9869-404a-b14a-343321527fdb nodeName:}" failed. No retries permitted until 2026-09-08 13:18:03.808723217 +0000 UTC m=+134.080171127 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/871377db-9869-404a-b14a-343321527fdb-metrics-certs") pod "router-default-69549dc55b-qs44g" (UID: "871377db-9869-404a-b14a-343321527fdb") : secret "router-metrics-certs-default" not found
Sep 08 13:18:01.911511 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:01.911468    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ec967b4f-3738-4934-84bc-2f6ab72f8a24-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-wfwsl\" (UID: \"ec967b4f-3738-4934-84bc-2f6ab72f8a24\") " pod="openshift-network-console/networking-console-plugin-79977f9844-wfwsl"
Sep 08 13:18:01.911695 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:18:01.911647    2573 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 08 13:18:01.911771 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:18:01.911740    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec967b4f-3738-4934-84bc-2f6ab72f8a24-networking-console-plugin-cert podName:ec967b4f-3738-4934-84bc-2f6ab72f8a24 nodeName:}" failed. No retries permitted until 2026-09-08 13:18:03.911716787 +0000 UTC m=+134.183164708 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/ec967b4f-3738-4934-84bc-2f6ab72f8a24-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-wfwsl" (UID: "ec967b4f-3738-4934-84bc-2f6ab72f8a24") : secret "networking-console-plugin-cert" not found
Sep 08 13:18:03.463786 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:03.463752    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:18:03.543033 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:03.542976    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:18:03.827646 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:03.827560    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/871377db-9869-404a-b14a-343321527fdb-metrics-certs\") pod \"router-default-69549dc55b-qs44g\" (UID: \"871377db-9869-404a-b14a-343321527fdb\") " pod="openshift-ingress/router-default-69549dc55b-qs44g"
Sep 08 13:18:03.827646 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:03.827622    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/871377db-9869-404a-b14a-343321527fdb-service-ca-bundle\") pod \"router-default-69549dc55b-qs44g\" (UID: \"871377db-9869-404a-b14a-343321527fdb\") " pod="openshift-ingress/router-default-69549dc55b-qs44g"
Sep 08 13:18:03.827866 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:18:03.827727    2573 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 08 13:18:03.827866 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:18:03.827785    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/871377db-9869-404a-b14a-343321527fdb-service-ca-bundle podName:871377db-9869-404a-b14a-343321527fdb nodeName:}" failed. No retries permitted until 2026-09-08 13:18:07.827763281 +0000 UTC m=+138.099211206 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/871377db-9869-404a-b14a-343321527fdb-service-ca-bundle") pod "router-default-69549dc55b-qs44g" (UID: "871377db-9869-404a-b14a-343321527fdb") : configmap references non-existent config key: service-ca.crt
Sep 08 13:18:03.827866 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:18:03.827805    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/871377db-9869-404a-b14a-343321527fdb-metrics-certs podName:871377db-9869-404a-b14a-343321527fdb nodeName:}" failed. No retries permitted until 2026-09-08 13:18:07.827798214 +0000 UTC m=+138.099246126 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/871377db-9869-404a-b14a-343321527fdb-metrics-certs") pod "router-default-69549dc55b-qs44g" (UID: "871377db-9869-404a-b14a-343321527fdb") : secret "router-metrics-certs-default" not found
Sep 08 13:18:03.928744 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:03.928706    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ec967b4f-3738-4934-84bc-2f6ab72f8a24-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-wfwsl\" (UID: \"ec967b4f-3738-4934-84bc-2f6ab72f8a24\") " pod="openshift-network-console/networking-console-plugin-79977f9844-wfwsl"
Sep 08 13:18:03.928902 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:18:03.928882    2573 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 08 13:18:03.928980 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:18:03.928969    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec967b4f-3738-4934-84bc-2f6ab72f8a24-networking-console-plugin-cert podName:ec967b4f-3738-4934-84bc-2f6ab72f8a24 nodeName:}" failed. No retries permitted until 2026-09-08 13:18:07.92895144 +0000 UTC m=+138.200399359 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/ec967b4f-3738-4934-84bc-2f6ab72f8a24-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-wfwsl" (UID: "ec967b4f-3738-4934-84bc-2f6ab72f8a24") : secret "networking-console-plugin-cert" not found
Sep 08 13:18:05.564293 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:05.564260    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-kgz9w"]
Sep 08 13:18:05.566938 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:05.566916    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-kgz9w"
Sep 08 13:18:05.569738 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:05.569718    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 08 13:18:05.570485 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:05.570463    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 08 13:18:05.571141 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:05.571124    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 08 13:18:05.572089 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:05.572070    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-x5h6q\""
Sep 08 13:18:05.572169 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:05.572104    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 08 13:18:05.573869 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:05.573849    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:18:05.581525 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:05.581491    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-kgz9w"]
Sep 08 13:18:05.597316 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:05.597293    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64bfdb6bcc-6lc4p" event={"ID":"ffc5700f-1e9f-47a0-a0f0-98fc161b51e5","Type":"ContainerStarted","Data":"aa45d25b4ba890d745b15e2d2479167623cc0989f7af0eaa12c20de58b98cc3f"}
Sep 08 13:18:05.597415 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:05.597334    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64bfdb6bcc-6lc4p" event={"ID":"ffc5700f-1e9f-47a0-a0f0-98fc161b51e5","Type":"ContainerStarted","Data":"841bc07c0b5e4080ba2565930fc1ca98ef6a6bbe3d60e4d424e639376536b19a"}
Sep 08 13:18:05.597415 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:05.597352    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64bfdb6bcc-6lc4p" event={"ID":"ffc5700f-1e9f-47a0-a0f0-98fc161b51e5","Type":"ContainerStarted","Data":"28720affbd9c10869534876bf4bbce482110b9f0ea74ba18ebe6a0102212980a"}
Sep 08 13:18:05.620839 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:05.620798    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64bfdb6bcc-6lc4p" podStartSLOduration=1.934337849 podStartE2EDuration="5.620784723s" totalImagesPullingTime="3.686446874s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:18:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:18:05.620030223 +0000 UTC m=+135.891478155" watchObservedRunningTime="2026-09-08 13:18:05.620784723 +0000 UTC m=+135.892232653"
Sep 08 13:18:05.641060 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:05.641039    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6934e1da-3606-4819-a625-60e175d0c1ce-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kgz9w\" (UID: \"6934e1da-3606-4819-a625-60e175d0c1ce\") " pod="openshift-insights/insights-runtime-extractor-kgz9w"
Sep 08 13:18:05.641126 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:05.641069    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9z7kh\" (UniqueName: \"kubernetes.io/projected/6934e1da-3606-4819-a625-60e175d0c1ce-kube-api-access-9z7kh\") pod \"insights-runtime-extractor-kgz9w\" (UID: \"6934e1da-3606-4819-a625-60e175d0c1ce\") " pod="openshift-insights/insights-runtime-extractor-kgz9w"
Sep 08 13:18:05.641126 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:05.641103    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6934e1da-3606-4819-a625-60e175d0c1ce-crio-socket\") pod \"insights-runtime-extractor-kgz9w\" (UID: \"6934e1da-3606-4819-a625-60e175d0c1ce\") " pod="openshift-insights/insights-runtime-extractor-kgz9w"
Sep 08 13:18:05.641222 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:05.641161    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6934e1da-3606-4819-a625-60e175d0c1ce-data-volume\") pod \"insights-runtime-extractor-kgz9w\" (UID: \"6934e1da-3606-4819-a625-60e175d0c1ce\") " pod="openshift-insights/insights-runtime-extractor-kgz9w"
Sep 08 13:18:05.641266 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:05.641229    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6934e1da-3606-4819-a625-60e175d0c1ce-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kgz9w\" (UID: \"6934e1da-3606-4819-a625-60e175d0c1ce\") " pod="openshift-insights/insights-runtime-extractor-kgz9w"
Sep 08 13:18:05.686290 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:05.686268    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:18:05.742409 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:05.742384    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6934e1da-3606-4819-a625-60e175d0c1ce-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kgz9w\" (UID: \"6934e1da-3606-4819-a625-60e175d0c1ce\") " pod="openshift-insights/insights-runtime-extractor-kgz9w"
Sep 08 13:18:05.742494 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:05.742446    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9z7kh\" (UniqueName: \"kubernetes.io/projected/6934e1da-3606-4819-a625-60e175d0c1ce-kube-api-access-9z7kh\") pod \"insights-runtime-extractor-kgz9w\" (UID: \"6934e1da-3606-4819-a625-60e175d0c1ce\") " pod="openshift-insights/insights-runtime-extractor-kgz9w"
Sep 08 13:18:05.742601 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:05.742548    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6934e1da-3606-4819-a625-60e175d0c1ce-crio-socket\") pod \"insights-runtime-extractor-kgz9w\" (UID: \"6934e1da-3606-4819-a625-60e175d0c1ce\") " pod="openshift-insights/insights-runtime-extractor-kgz9w"
Sep 08 13:18:05.742601 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:05.742577    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6934e1da-3606-4819-a625-60e175d0c1ce-data-volume\") pod \"insights-runtime-extractor-kgz9w\" (UID: \"6934e1da-3606-4819-a625-60e175d0c1ce\") " pod="openshift-insights/insights-runtime-extractor-kgz9w"
Sep 08 13:18:05.742709 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:05.742641    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6934e1da-3606-4819-a625-60e175d0c1ce-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kgz9w\" (UID: \"6934e1da-3606-4819-a625-60e175d0c1ce\") " pod="openshift-insights/insights-runtime-extractor-kgz9w"
Sep 08 13:18:05.742709 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:05.742674    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6934e1da-3606-4819-a625-60e175d0c1ce-crio-socket\") pod \"insights-runtime-extractor-kgz9w\" (UID: \"6934e1da-3606-4819-a625-60e175d0c1ce\") " pod="openshift-insights/insights-runtime-extractor-kgz9w"
Sep 08 13:18:05.742815 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:18:05.742745    2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 08 13:18:05.742815 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:18:05.742799    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6934e1da-3606-4819-a625-60e175d0c1ce-insights-runtime-extractor-tls podName:6934e1da-3606-4819-a625-60e175d0c1ce nodeName:}" failed. No retries permitted until 2026-09-08 13:18:06.242782179 +0000 UTC m=+136.514230091 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6934e1da-3606-4819-a625-60e175d0c1ce-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kgz9w" (UID: "6934e1da-3606-4819-a625-60e175d0c1ce") : secret "insights-runtime-extractor-tls" not found
Sep 08 13:18:05.742894 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:18:05.742852    2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 08 13:18:05.742935 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:18:05.742909    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6934e1da-3606-4819-a625-60e175d0c1ce-kube-rbac-proxy-cm podName:6934e1da-3606-4819-a625-60e175d0c1ce nodeName:}" failed. No retries permitted until 2026-09-08 13:18:06.242892287 +0000 UTC m=+136.514340220 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6934e1da-3606-4819-a625-60e175d0c1ce-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kgz9w" (UID: "6934e1da-3606-4819-a625-60e175d0c1ce") : configmap "kube-rbac-proxy" not found
Sep 08 13:18:05.742935 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:05.742917    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6934e1da-3606-4819-a625-60e175d0c1ce-data-volume\") pod \"insights-runtime-extractor-kgz9w\" (UID: \"6934e1da-3606-4819-a625-60e175d0c1ce\") " pod="openshift-insights/insights-runtime-extractor-kgz9w"
Sep 08 13:18:05.750103 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:05.750078    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9z7kh\" (UniqueName: \"kubernetes.io/projected/6934e1da-3606-4819-a625-60e175d0c1ce-kube-api-access-9z7kh\") pod \"insights-runtime-extractor-kgz9w\" (UID: \"6934e1da-3606-4819-a625-60e175d0c1ce\") " pod="openshift-insights/insights-runtime-extractor-kgz9w"
Sep 08 13:18:06.246135 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:06.246101    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6934e1da-3606-4819-a625-60e175d0c1ce-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kgz9w\" (UID: \"6934e1da-3606-4819-a625-60e175d0c1ce\") " pod="openshift-insights/insights-runtime-extractor-kgz9w"
Sep 08 13:18:06.246311 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:06.246173    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6934e1da-3606-4819-a625-60e175d0c1ce-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kgz9w\" (UID: \"6934e1da-3606-4819-a625-60e175d0c1ce\") " pod="openshift-insights/insights-runtime-extractor-kgz9w"
Sep 08 13:18:06.246311 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:18:06.246239    2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 08 13:18:06.246311 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:18:06.246294    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6934e1da-3606-4819-a625-60e175d0c1ce-kube-rbac-proxy-cm podName:6934e1da-3606-4819-a625-60e175d0c1ce nodeName:}" failed. No retries permitted until 2026-09-08 13:18:07.246280298 +0000 UTC m=+137.517728207 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6934e1da-3606-4819-a625-60e175d0c1ce-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kgz9w" (UID: "6934e1da-3606-4819-a625-60e175d0c1ce") : configmap "kube-rbac-proxy" not found
Sep 08 13:18:06.246311 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:18:06.246243    2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 08 13:18:06.246493 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:18:06.246378    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6934e1da-3606-4819-a625-60e175d0c1ce-insights-runtime-extractor-tls podName:6934e1da-3606-4819-a625-60e175d0c1ce nodeName:}" failed. No retries permitted until 2026-09-08 13:18:07.246353914 +0000 UTC m=+137.517801837 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6934e1da-3606-4819-a625-60e175d0c1ce-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kgz9w" (UID: "6934e1da-3606-4819-a625-60e175d0c1ce") : secret "insights-runtime-extractor-tls" not found
Sep 08 13:18:06.643302 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:06.643267    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:18:06.733886 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:06.733848    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:18:06.833681 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:06.833657    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:18:07.253920 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:07.253892    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6934e1da-3606-4819-a625-60e175d0c1ce-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kgz9w\" (UID: \"6934e1da-3606-4819-a625-60e175d0c1ce\") " pod="openshift-insights/insights-runtime-extractor-kgz9w"
Sep 08 13:18:07.254059 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:07.253954    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6934e1da-3606-4819-a625-60e175d0c1ce-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kgz9w\" (UID: \"6934e1da-3606-4819-a625-60e175d0c1ce\") " pod="openshift-insights/insights-runtime-extractor-kgz9w"
Sep 08 13:18:07.254059 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:18:07.254016    2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 08 13:18:07.254131 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:18:07.254066    2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 08 13:18:07.254131 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:18:07.254102    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6934e1da-3606-4819-a625-60e175d0c1ce-kube-rbac-proxy-cm podName:6934e1da-3606-4819-a625-60e175d0c1ce nodeName:}" failed. No retries permitted until 2026-09-08 13:18:09.254084504 +0000 UTC m=+139.525532416 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6934e1da-3606-4819-a625-60e175d0c1ce-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kgz9w" (UID: "6934e1da-3606-4819-a625-60e175d0c1ce") : configmap "kube-rbac-proxy" not found
Sep 08 13:18:07.254131 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:18:07.254120    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6934e1da-3606-4819-a625-60e175d0c1ce-insights-runtime-extractor-tls podName:6934e1da-3606-4819-a625-60e175d0c1ce nodeName:}" failed. No retries permitted until 2026-09-08 13:18:09.254111694 +0000 UTC m=+139.525559602 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6934e1da-3606-4819-a625-60e175d0c1ce-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kgz9w" (UID: "6934e1da-3606-4819-a625-60e175d0c1ce") : secret "insights-runtime-extractor-tls" not found
Sep 08 13:18:07.292461 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:07.292443    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-rbnnh_42aa8a5e-3aba-4581-9f95-f3b978c250f0/dns-node-resolver/0.log"
Sep 08 13:18:07.858178 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:07.858138    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/871377db-9869-404a-b14a-343321527fdb-service-ca-bundle\") pod \"router-default-69549dc55b-qs44g\" (UID: \"871377db-9869-404a-b14a-343321527fdb\") " pod="openshift-ingress/router-default-69549dc55b-qs44g"
Sep 08 13:18:07.858635 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:07.858255    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/871377db-9869-404a-b14a-343321527fdb-metrics-certs\") pod \"router-default-69549dc55b-qs44g\" (UID: \"871377db-9869-404a-b14a-343321527fdb\") " pod="openshift-ingress/router-default-69549dc55b-qs44g"
Sep 08 13:18:07.858635 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:18:07.858292    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/871377db-9869-404a-b14a-343321527fdb-service-ca-bundle podName:871377db-9869-404a-b14a-343321527fdb nodeName:}" failed. No retries permitted until 2026-09-08 13:18:15.858276204 +0000 UTC m=+146.129724113 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/871377db-9869-404a-b14a-343321527fdb-service-ca-bundle") pod "router-default-69549dc55b-qs44g" (UID: "871377db-9869-404a-b14a-343321527fdb") : configmap references non-existent config key: service-ca.crt
Sep 08 13:18:07.858635 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:18:07.858366    2573 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 08 13:18:07.858635 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:18:07.858423    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/871377db-9869-404a-b14a-343321527fdb-metrics-certs podName:871377db-9869-404a-b14a-343321527fdb nodeName:}" failed. No retries permitted until 2026-09-08 13:18:15.858405809 +0000 UTC m=+146.129853737 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/871377db-9869-404a-b14a-343321527fdb-metrics-certs") pod "router-default-69549dc55b-qs44g" (UID: "871377db-9869-404a-b14a-343321527fdb") : secret "router-metrics-certs-default" not found
Sep 08 13:18:07.884179 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:07.884154    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:18:07.959272 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:07.959246    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ec967b4f-3738-4934-84bc-2f6ab72f8a24-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-wfwsl\" (UID: \"ec967b4f-3738-4934-84bc-2f6ab72f8a24\") " pod="openshift-network-console/networking-console-plugin-79977f9844-wfwsl"
Sep 08 13:18:07.959400 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:18:07.959386    2573 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 08 13:18:07.959445 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:18:07.959439    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec967b4f-3738-4934-84bc-2f6ab72f8a24-networking-console-plugin-cert podName:ec967b4f-3738-4934-84bc-2f6ab72f8a24 nodeName:}" failed. No retries permitted until 2026-09-08 13:18:15.959424316 +0000 UTC m=+146.230872239 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/ec967b4f-3738-4934-84bc-2f6ab72f8a24-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-wfwsl" (UID: "ec967b4f-3738-4934-84bc-2f6ab72f8a24") : secret "networking-console-plugin-cert" not found
Sep 08 13:18:08.293077 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:08.293050    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-24f78_137be755-3092-45ba-a6d2-8a025c5e6226/node-ca/0.log"
Sep 08 13:18:09.267797 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:09.267763    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6934e1da-3606-4819-a625-60e175d0c1ce-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kgz9w\" (UID: \"6934e1da-3606-4819-a625-60e175d0c1ce\") " pod="openshift-insights/insights-runtime-extractor-kgz9w"
Sep 08 13:18:09.268242 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:09.267846    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6934e1da-3606-4819-a625-60e175d0c1ce-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kgz9w\" (UID: \"6934e1da-3606-4819-a625-60e175d0c1ce\") " pod="openshift-insights/insights-runtime-extractor-kgz9w"
Sep 08 13:18:09.268242 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:18:09.267890    2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 08 13:18:09.268242 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:18:09.267965    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6934e1da-3606-4819-a625-60e175d0c1ce-kube-rbac-proxy-cm podName:6934e1da-3606-4819-a625-60e175d0c1ce nodeName:}" failed. No retries permitted until 2026-09-08 13:18:13.267948447 +0000 UTC m=+143.539396361 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6934e1da-3606-4819-a625-60e175d0c1ce-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kgz9w" (UID: "6934e1da-3606-4819-a625-60e175d0c1ce") : configmap "kube-rbac-proxy" not found
Sep 08 13:18:09.268242 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:18:09.268006    2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 08 13:18:09.268242 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:18:09.268080    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6934e1da-3606-4819-a625-60e175d0c1ce-insights-runtime-extractor-tls podName:6934e1da-3606-4819-a625-60e175d0c1ce nodeName:}" failed. No retries permitted until 2026-09-08 13:18:13.268060157 +0000 UTC m=+143.539508086 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6934e1da-3606-4819-a625-60e175d0c1ce-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kgz9w" (UID: "6934e1da-3606-4819-a625-60e175d0c1ce") : secret "insights-runtime-extractor-tls" not found
Sep 08 13:18:09.493185 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:09.493134    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-qmrbg_831913e7-ea98-4d0b-bbaf-411e16552f73/serve-healthcheck-canary/0.log"
Sep 08 13:18:13.296943 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:13.296909    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6934e1da-3606-4819-a625-60e175d0c1ce-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kgz9w\" (UID: \"6934e1da-3606-4819-a625-60e175d0c1ce\") " pod="openshift-insights/insights-runtime-extractor-kgz9w"
Sep 08 13:18:13.297358 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:13.296981    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6934e1da-3606-4819-a625-60e175d0c1ce-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kgz9w\" (UID: \"6934e1da-3606-4819-a625-60e175d0c1ce\") " pod="openshift-insights/insights-runtime-extractor-kgz9w"
Sep 08 13:18:13.297358 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:18:13.297052    2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 08 13:18:13.297358 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:18:13.297069    2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 08 13:18:13.297358 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:18:13.297111    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6934e1da-3606-4819-a625-60e175d0c1ce-kube-rbac-proxy-cm podName:6934e1da-3606-4819-a625-60e175d0c1ce nodeName:}" failed. No retries permitted until 2026-09-08 13:18:21.297096454 +0000 UTC m=+151.568544362 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6934e1da-3606-4819-a625-60e175d0c1ce-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kgz9w" (UID: "6934e1da-3606-4819-a625-60e175d0c1ce") : configmap "kube-rbac-proxy" not found
Sep 08 13:18:13.297358 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:18:13.297125    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6934e1da-3606-4819-a625-60e175d0c1ce-insights-runtime-extractor-tls podName:6934e1da-3606-4819-a625-60e175d0c1ce nodeName:}" failed. No retries permitted until 2026-09-08 13:18:21.297118046 +0000 UTC m=+151.568565956 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6934e1da-3606-4819-a625-60e175d0c1ce-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kgz9w" (UID: "6934e1da-3606-4819-a625-60e175d0c1ce") : secret "insights-runtime-extractor-tls" not found
Sep 08 13:18:13.302820 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:13.302779    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:18:13.856798 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:13.856762    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:18:14.110799 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:14.110723    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:18:14.509896 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:14.509849    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:18:14.971973 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:14.971938    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:18:15.411268 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:15.411233    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:18:15.511910 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:15.511881    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:18:15.918588 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:15.918558    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/871377db-9869-404a-b14a-343321527fdb-metrics-certs\") pod \"router-default-69549dc55b-qs44g\" (UID: \"871377db-9869-404a-b14a-343321527fdb\") " pod="openshift-ingress/router-default-69549dc55b-qs44g"
Sep 08 13:18:15.918736 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:15.918609    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/871377db-9869-404a-b14a-343321527fdb-service-ca-bundle\") pod \"router-default-69549dc55b-qs44g\" (UID: \"871377db-9869-404a-b14a-343321527fdb\") " pod="openshift-ingress/router-default-69549dc55b-qs44g"
Sep 08 13:18:15.918736 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:18:15.918697    2573 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 08 13:18:15.918806 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:18:15.918747    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/871377db-9869-404a-b14a-343321527fdb-service-ca-bundle podName:871377db-9869-404a-b14a-343321527fdb nodeName:}" failed. No retries permitted until 2026-09-08 13:18:31.918732795 +0000 UTC m=+162.190180723 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/871377db-9869-404a-b14a-343321527fdb-service-ca-bundle") pod "router-default-69549dc55b-qs44g" (UID: "871377db-9869-404a-b14a-343321527fdb") : configmap references non-existent config key: service-ca.crt
Sep 08 13:18:15.918806 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:18:15.918762    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/871377db-9869-404a-b14a-343321527fdb-metrics-certs podName:871377db-9869-404a-b14a-343321527fdb nodeName:}" failed. No retries permitted until 2026-09-08 13:18:31.918756636 +0000 UTC m=+162.190204544 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/871377db-9869-404a-b14a-343321527fdb-metrics-certs") pod "router-default-69549dc55b-qs44g" (UID: "871377db-9869-404a-b14a-343321527fdb") : secret "router-metrics-certs-default" not found
Sep 08 13:18:16.019585 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:16.019549    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ec967b4f-3738-4934-84bc-2f6ab72f8a24-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-wfwsl\" (UID: \"ec967b4f-3738-4934-84bc-2f6ab72f8a24\") " pod="openshift-network-console/networking-console-plugin-79977f9844-wfwsl"
Sep 08 13:18:16.019734 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:18:16.019704    2573 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 08 13:18:16.019774 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:18:16.019765    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec967b4f-3738-4934-84bc-2f6ab72f8a24-networking-console-plugin-cert podName:ec967b4f-3738-4934-84bc-2f6ab72f8a24 nodeName:}" failed. No retries permitted until 2026-09-08 13:18:32.019748489 +0000 UTC m=+162.291196415 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/ec967b4f-3738-4934-84bc-2f6ab72f8a24-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-wfwsl" (UID: "ec967b4f-3738-4934-84bc-2f6ab72f8a24") : secret "networking-console-plugin-cert" not found
Sep 08 13:18:16.104106 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:16.104058    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:18:16.511972 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:16.511924    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:18:16.612690 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:16.612647    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:18:16.713161 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:16.713128    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:18:17.111961 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:17.111925    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:18:17.193254 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:17.193226    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:18:18.350689 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:18.350662    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:18:18.416013 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:18.415955    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:18:18.504281 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:18.504253    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:18:18.710535 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:18.710483    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:18:18.813149 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:18.813115    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:18:19.103387 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:19.103311    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:18:19.161894 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:19.161873    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:18:19.919387 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:19.919353    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:18:20.308397 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:20.308322    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:18:20.517641 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:20.517609    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:18:20.711386 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:20.711353    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:18:21.114046 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:21.113973    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:18:21.363307 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:21.363262    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6934e1da-3606-4819-a625-60e175d0c1ce-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kgz9w\" (UID: \"6934e1da-3606-4819-a625-60e175d0c1ce\") " pod="openshift-insights/insights-runtime-extractor-kgz9w"
Sep 08 13:18:21.363467 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:21.363359    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6934e1da-3606-4819-a625-60e175d0c1ce-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kgz9w\" (UID: \"6934e1da-3606-4819-a625-60e175d0c1ce\") " pod="openshift-insights/insights-runtime-extractor-kgz9w"
Sep 08 13:18:21.363530 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:18:21.363481    2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 08 13:18:21.363569 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:18:21.363564    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6934e1da-3606-4819-a625-60e175d0c1ce-insights-runtime-extractor-tls podName:6934e1da-3606-4819-a625-60e175d0c1ce nodeName:}" failed. No retries permitted until 2026-09-08 13:18:37.363547071 +0000 UTC m=+167.634995000 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6934e1da-3606-4819-a625-60e175d0c1ce-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kgz9w" (UID: "6934e1da-3606-4819-a625-60e175d0c1ce") : secret "insights-runtime-extractor-tls" not found
Sep 08 13:18:21.363864 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:21.363839    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6934e1da-3606-4819-a625-60e175d0c1ce-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kgz9w\" (UID: \"6934e1da-3606-4819-a625-60e175d0c1ce\") " pod="openshift-insights/insights-runtime-extractor-kgz9w"
Sep 08 13:18:21.715392 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:21.715364    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:18:22.184418 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:22.184389    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:18:22.309861 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:22.309837    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:18:22.711136 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:22.711105    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:18:23.080127 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:23.080045    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:18:23.176538 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:23.176489    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:18:23.666372 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:23.666342    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:18:24.131512 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:24.131478    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:18:24.227933 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:24.227907    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:18:24.624097 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:24.624067    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:18:25.172788 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:25.172759    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:18:27.200946 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:18:27.200912    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-knfp2" podUID="21ee5269-0851-405f-a9a5-f9d2741e8500"
Sep 08 13:18:27.661080 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:27.661055    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-knfp2"
Sep 08 13:18:28.179477 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:18:28.179442    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-kpscq" podUID="217a8762-14bb-4990-8a53-3b3bcd6984ac"
Sep 08 13:18:28.653216 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:28.653191    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:18:28.909870 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:28.909804    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:18:31.936141 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:31.936102    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/871377db-9869-404a-b14a-343321527fdb-service-ca-bundle\") pod \"router-default-69549dc55b-qs44g\" (UID: \"871377db-9869-404a-b14a-343321527fdb\") " pod="openshift-ingress/router-default-69549dc55b-qs44g"
Sep 08 13:18:31.936544 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:31.936244    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/871377db-9869-404a-b14a-343321527fdb-metrics-certs\") pod \"router-default-69549dc55b-qs44g\" (UID: \"871377db-9869-404a-b14a-343321527fdb\") " pod="openshift-ingress/router-default-69549dc55b-qs44g"
Sep 08 13:18:31.936723 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:31.936702    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/871377db-9869-404a-b14a-343321527fdb-service-ca-bundle\") pod \"router-default-69549dc55b-qs44g\" (UID: \"871377db-9869-404a-b14a-343321527fdb\") " pod="openshift-ingress/router-default-69549dc55b-qs44g"
Sep 08 13:18:31.938389 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:31.938371    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/871377db-9869-404a-b14a-343321527fdb-metrics-certs\") pod \"router-default-69549dc55b-qs44g\" (UID: \"871377db-9869-404a-b14a-343321527fdb\") " pod="openshift-ingress/router-default-69549dc55b-qs44g"
Sep 08 13:18:31.954443 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:31.954412    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-69549dc55b-qs44g"
Sep 08 13:18:32.037597 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:32.037559    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/21ee5269-0851-405f-a9a5-f9d2741e8500-metrics-tls\") pod \"dns-default-knfp2\" (UID: \"21ee5269-0851-405f-a9a5-f9d2741e8500\") " pod="openshift-dns/dns-default-knfp2"
Sep 08 13:18:32.037744 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:32.037642    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ec967b4f-3738-4934-84bc-2f6ab72f8a24-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-wfwsl\" (UID: \"ec967b4f-3738-4934-84bc-2f6ab72f8a24\") " pod="openshift-network-console/networking-console-plugin-79977f9844-wfwsl"
Sep 08 13:18:32.039712 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:32.039684    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/21ee5269-0851-405f-a9a5-f9d2741e8500-metrics-tls\") pod \"dns-default-knfp2\" (UID: \"21ee5269-0851-405f-a9a5-f9d2741e8500\") " pod="openshift-dns/dns-default-knfp2"
Sep 08 13:18:32.039908 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:32.039879    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ec967b4f-3738-4934-84bc-2f6ab72f8a24-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-wfwsl\" (UID: \"ec967b4f-3738-4934-84bc-2f6ab72f8a24\") " pod="openshift-network-console/networking-console-plugin-79977f9844-wfwsl"
Sep 08 13:18:32.050848 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:32.050824    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-wfwsl"
Sep 08 13:18:32.163975 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:32.163945    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-8shlb\""
Sep 08 13:18:32.172001 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:32.171987    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-knfp2"
Sep 08 13:18:32.227484 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:18:32.227456    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod871377db_9869_404a_b14a_343321527fdb.slice/crio-cc402dfd6e65556f23d75f9f53e6fccfa4852be2e3ac4126da96e9c956eb581b WatchSource:0}: Error finding container cc402dfd6e65556f23d75f9f53e6fccfa4852be2e3ac4126da96e9c956eb581b: Status 404 returned error can't find the container with id cc402dfd6e65556f23d75f9f53e6fccfa4852be2e3ac4126da96e9c956eb581b
Sep 08 13:18:32.233353 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:32.233294    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-69549dc55b-qs44g"]
Sep 08 13:18:32.306291 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:32.306260    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:18:32.314677 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:32.314655    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-wfwsl"]
Sep 08 13:18:32.317026 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:18:32.316999    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec967b4f_3738_4934_84bc_2f6ab72f8a24.slice/crio-e66a0a78f7aae85e9a50b6a7115c87570ae29cf0e7dcee34ab5935eb5b85fed0 WatchSource:0}: Error finding container e66a0a78f7aae85e9a50b6a7115c87570ae29cf0e7dcee34ab5935eb5b85fed0: Status 404 returned error can't find the container with id e66a0a78f7aae85e9a50b6a7115c87570ae29cf0e7dcee34ab5935eb5b85fed0
Sep 08 13:18:32.406895 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:32.406866    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:18:32.437687 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:32.437655    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-knfp2"]
Sep 08 13:18:32.442275 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:18:32.442251    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21ee5269_0851_405f_a9a5_f9d2741e8500.slice/crio-2ef92211a35354e0712c9b05e44ad906594f3220f1e6a741ac9775c09c354d3e WatchSource:0}: Error finding container 2ef92211a35354e0712c9b05e44ad906594f3220f1e6a741ac9775c09c354d3e: Status 404 returned error can't find the container with id 2ef92211a35354e0712c9b05e44ad906594f3220f1e6a741ac9775c09c354d3e
Sep 08 13:18:32.674547 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:32.674519    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-knfp2" event={"ID":"21ee5269-0851-405f-a9a5-f9d2741e8500","Type":"ContainerStarted","Data":"2ef92211a35354e0712c9b05e44ad906594f3220f1e6a741ac9775c09c354d3e"}
Sep 08 13:18:32.675543 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:32.675524    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-79977f9844-wfwsl" event={"ID":"ec967b4f-3738-4934-84bc-2f6ab72f8a24","Type":"ContainerStarted","Data":"e66a0a78f7aae85e9a50b6a7115c87570ae29cf0e7dcee34ab5935eb5b85fed0"}
Sep 08 13:18:32.676955 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:32.676936    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-69549dc55b-qs44g" event={"ID":"871377db-9869-404a-b14a-343321527fdb","Type":"ContainerStarted","Data":"11ddbbb4a9ea73969079acd66c9fedac373fc94dfddc0b657268f299d7d8c02e"}
Sep 08 13:18:32.677045 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:32.676971    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-69549dc55b-qs44g" event={"ID":"871377db-9869-404a-b14a-343321527fdb","Type":"ContainerStarted","Data":"cc402dfd6e65556f23d75f9f53e6fccfa4852be2e3ac4126da96e9c956eb581b"}
Sep 08 13:18:32.698636 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:32.698594    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-69549dc55b-qs44g" podStartSLOduration=32.698581447 podStartE2EDuration="32.698581447s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:18:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:18:32.697839191 +0000 UTC m=+162.969287132" watchObservedRunningTime="2026-09-08 13:18:32.698581447 +0000 UTC m=+162.970029377"
Sep 08 13:18:32.955633 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:32.955562    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-69549dc55b-qs44g"
Sep 08 13:18:32.957905 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:32.957883    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-69549dc55b-qs44g"
Sep 08 13:18:33.680974 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:33.680944    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-69549dc55b-qs44g"
Sep 08 13:18:33.682462 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:33.682432    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-69549dc55b-qs44g"
Sep 08 13:18:34.685694 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:34.685660    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-knfp2" event={"ID":"21ee5269-0851-405f-a9a5-f9d2741e8500","Type":"ContainerStarted","Data":"e2ea1a5c9d611a484be724a3e19a31e0117224e8d6f84789714358f0493268c0"}
Sep 08 13:18:34.686122 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:34.685715    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-knfp2" event={"ID":"21ee5269-0851-405f-a9a5-f9d2741e8500","Type":"ContainerStarted","Data":"28ac3fde42fc01fedac8e8bef5270c895b8b190ef57c7d494788ac0c80ab6a26"}
Sep 08 13:18:34.686122 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:34.685774    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-knfp2"
Sep 08 13:18:34.687147 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:34.687124    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-79977f9844-wfwsl" event={"ID":"ec967b4f-3738-4934-84bc-2f6ab72f8a24","Type":"ContainerStarted","Data":"cb4d6dfef0201227a0af8c3381bbb17f538edde66aa9ee0e63181cc619e82c4b"}
Sep 08 13:18:34.715013 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:34.714966    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-knfp2" podStartSLOduration=128.880240795 podStartE2EDuration="2m10.714948219s" totalImagesPullingTime="1.834707424s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:16:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:18:34.714919258 +0000 UTC m=+164.986367190" watchObservedRunningTime="2026-09-08 13:18:34.714948219 +0000 UTC m=+164.986396157"
Sep 08 13:18:34.734516 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:34.734467    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-79977f9844-wfwsl" podStartSLOduration=32.779141502 podStartE2EDuration="34.734453234s" totalImagesPullingTime="1.955311732s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:18:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:18:34.733334938 +0000 UTC m=+165.004782869" watchObservedRunningTime="2026-09-08 13:18:34.734453234 +0000 UTC m=+165.005901165"
Sep 08 13:18:34.994847 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:34.994772    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:18:35.422000 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:35.421968    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:18:36.411948 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:36.411912    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:18:36.500113 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:36.500078    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:18:37.374733 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:37.374684    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6934e1da-3606-4819-a625-60e175d0c1ce-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kgz9w\" (UID: \"6934e1da-3606-4819-a625-60e175d0c1ce\") " pod="openshift-insights/insights-runtime-extractor-kgz9w"
Sep 08 13:18:37.376913 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:37.376886    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6934e1da-3606-4819-a625-60e175d0c1ce-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kgz9w\" (UID: \"6934e1da-3606-4819-a625-60e175d0c1ce\") " pod="openshift-insights/insights-runtime-extractor-kgz9w"
Sep 08 13:18:37.675559 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:37.675452    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-kgz9w"
Sep 08 13:18:37.949820 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:37.949751    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-kgz9w"]
Sep 08 13:18:37.952836 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:18:37.952804    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6934e1da_3606_4819_a625_60e175d0c1ce.slice/crio-979053e08cf0353a0cfed7f5e37a03b86d5b10ac34e8e87c1a15b342e9eb5449 WatchSource:0}: Error finding container 979053e08cf0353a0cfed7f5e37a03b86d5b10ac34e8e87c1a15b342e9eb5449: Status 404 returned error can't find the container with id 979053e08cf0353a0cfed7f5e37a03b86d5b10ac34e8e87c1a15b342e9eb5449
Sep 08 13:18:38.701661 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:38.701624    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kgz9w" event={"ID":"6934e1da-3606-4819-a625-60e175d0c1ce","Type":"ContainerStarted","Data":"7322ab09e646d72da954f35bb58a949f058601f8aea64ecb1c1027bee18b6c12"}
Sep 08 13:18:38.701999 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:38.701679    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kgz9w" event={"ID":"6934e1da-3606-4819-a625-60e175d0c1ce","Type":"ContainerStarted","Data":"979053e08cf0353a0cfed7f5e37a03b86d5b10ac34e8e87c1a15b342e9eb5449"}
Sep 08 13:18:39.707464 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:39.707431    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kgz9w" event={"ID":"6934e1da-3606-4819-a625-60e175d0c1ce","Type":"ContainerStarted","Data":"638cae140e188c5d1e2a24f4956b8b9e8e9a9bae458384609988a9fb92736790"}
Sep 08 13:18:39.759132 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:39.759107    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:18:39.868627 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:39.868603    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:18:39.963944 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:39.963883    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:18:40.204101 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:40.204072    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:18:40.305778 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:40.305714    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:18:41.715476 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:41.715438    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kgz9w" event={"ID":"6934e1da-3606-4819-a625-60e175d0c1ce","Type":"ContainerStarted","Data":"9d69a91d1cea58ba6a39bee3172384d8b201559e24d009c75388024929347aea"}
Sep 08 13:18:41.734051 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:41.734007    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-kgz9w" podStartSLOduration=33.569411601 podStartE2EDuration="36.733993598s" totalImagesPullingTime="3.164581997s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 13:18:05 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:18:41.733000962 +0000 UTC m=+172.004448904" watchObservedRunningTime="2026-09-08 13:18:41.733993598 +0000 UTC m=+172.005441550"
Sep 08 13:18:43.169595 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:43.169493    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kpscq"
Sep 08 13:18:43.402958 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:43.402926    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:18:43.470113 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:43.470040    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:18:44.695359 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:44.695333    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-knfp2"
Sep 08 13:18:48.003483 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:48.003450    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:18:48.105485 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:48.105458    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:18:50.987460 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:50.987427    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-qnzxl"]
Sep 08 13:18:50.990587 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:50.990571    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-6f4946db56-qnzxl"
Sep 08 13:18:50.992892 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:50.992862    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 08 13:18:50.993009 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:50.992952    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 08 13:18:50.993382 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:50.993363    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 08 13:18:50.993708 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:50.993692    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 08 13:18:50.994184 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:50.994169    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-6jx4h\""
Sep 08 13:18:50.994458 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:50.994442    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 08 13:18:50.999571 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:50.999549    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-qnzxl"]
Sep 08 13:18:51.062881 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:51.062859    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4k9dp\" (UniqueName: \"kubernetes.io/projected/b3d87f50-8434-482b-b645-5118e2960fda-kube-api-access-4k9dp\") pod \"prometheus-operator-6f4946db56-qnzxl\" (UID: \"b3d87f50-8434-482b-b645-5118e2960fda\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-qnzxl"
Sep 08 13:18:51.062965 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:51.062891    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b3d87f50-8434-482b-b645-5118e2960fda-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-qnzxl\" (UID: \"b3d87f50-8434-482b-b645-5118e2960fda\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-qnzxl"
Sep 08 13:18:51.062965 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:51.062911    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b3d87f50-8434-482b-b645-5118e2960fda-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-qnzxl\" (UID: \"b3d87f50-8434-482b-b645-5118e2960fda\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-qnzxl"
Sep 08 13:18:51.063044 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:51.062986    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/b3d87f50-8434-482b-b645-5118e2960fda-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-qnzxl\" (UID: \"b3d87f50-8434-482b-b645-5118e2960fda\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-qnzxl"
Sep 08 13:18:51.163267 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:51.163233    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/b3d87f50-8434-482b-b645-5118e2960fda-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-qnzxl\" (UID: \"b3d87f50-8434-482b-b645-5118e2960fda\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-qnzxl"
Sep 08 13:18:51.163357 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:51.163284    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4k9dp\" (UniqueName: \"kubernetes.io/projected/b3d87f50-8434-482b-b645-5118e2960fda-kube-api-access-4k9dp\") pod \"prometheus-operator-6f4946db56-qnzxl\" (UID: \"b3d87f50-8434-482b-b645-5118e2960fda\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-qnzxl"
Sep 08 13:18:51.163357 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:51.163310    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b3d87f50-8434-482b-b645-5118e2960fda-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-qnzxl\" (UID: \"b3d87f50-8434-482b-b645-5118e2960fda\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-qnzxl"
Sep 08 13:18:51.163357 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:51.163343    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b3d87f50-8434-482b-b645-5118e2960fda-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-qnzxl\" (UID: \"b3d87f50-8434-482b-b645-5118e2960fda\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-qnzxl"
Sep 08 13:18:51.163452 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:18:51.163380    2573 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Sep 08 13:18:51.163485 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:18:51.163459    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3d87f50-8434-482b-b645-5118e2960fda-prometheus-operator-tls podName:b3d87f50-8434-482b-b645-5118e2960fda nodeName:}" failed. No retries permitted until 2026-09-08 13:18:51.663438506 +0000 UTC m=+181.934886422 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/b3d87f50-8434-482b-b645-5118e2960fda-prometheus-operator-tls") pod "prometheus-operator-6f4946db56-qnzxl" (UID: "b3d87f50-8434-482b-b645-5118e2960fda") : secret "prometheus-operator-tls" not found
Sep 08 13:18:51.164003 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:51.163984    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b3d87f50-8434-482b-b645-5118e2960fda-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-qnzxl\" (UID: \"b3d87f50-8434-482b-b645-5118e2960fda\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-qnzxl"
Sep 08 13:18:51.165600 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:51.165580    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b3d87f50-8434-482b-b645-5118e2960fda-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-qnzxl\" (UID: \"b3d87f50-8434-482b-b645-5118e2960fda\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-qnzxl"
Sep 08 13:18:51.174041 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:51.174022    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4k9dp\" (UniqueName: \"kubernetes.io/projected/b3d87f50-8434-482b-b645-5118e2960fda-kube-api-access-4k9dp\") pod \"prometheus-operator-6f4946db56-qnzxl\" (UID: \"b3d87f50-8434-482b-b645-5118e2960fda\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-qnzxl"
Sep 08 13:18:51.667299 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:51.667261    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/b3d87f50-8434-482b-b645-5118e2960fda-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-qnzxl\" (UID: \"b3d87f50-8434-482b-b645-5118e2960fda\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-qnzxl"
Sep 08 13:18:51.669488 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:51.669467    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/b3d87f50-8434-482b-b645-5118e2960fda-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-qnzxl\" (UID: \"b3d87f50-8434-482b-b645-5118e2960fda\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-qnzxl"
Sep 08 13:18:51.898986 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:51.898943    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-6f4946db56-qnzxl"
Sep 08 13:18:52.167561 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:52.167531    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-qnzxl"]
Sep 08 13:18:52.170530 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:18:52.170492    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3d87f50_8434_482b_b645_5118e2960fda.slice/crio-aaca5e914e2a23ee9109c6487de134196e4405b6204f3b64a206db3ddabe0186 WatchSource:0}: Error finding container aaca5e914e2a23ee9109c6487de134196e4405b6204f3b64a206db3ddabe0186: Status 404 returned error can't find the container with id aaca5e914e2a23ee9109c6487de134196e4405b6204f3b64a206db3ddabe0186
Sep 08 13:18:52.753873 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:52.753841    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-qnzxl" event={"ID":"b3d87f50-8434-482b-b645-5118e2960fda","Type":"ContainerStarted","Data":"aaca5e914e2a23ee9109c6487de134196e4405b6204f3b64a206db3ddabe0186"}
Sep 08 13:18:53.611888 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:53.611848    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:18:53.710941 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:53.710911    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:18:53.808450 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:53.808423    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:18:54.413855 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:54.413765    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:18:54.761921 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:54.761840    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-qnzxl" event={"ID":"b3d87f50-8434-482b-b645-5118e2960fda","Type":"ContainerStarted","Data":"9873ae8bcc5ed09613ddf40b9035be687c27c5e4ca5220a8aba18f53ec955892"}
Sep 08 13:18:54.761921 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:54.761894    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-qnzxl" event={"ID":"b3d87f50-8434-482b-b645-5118e2960fda","Type":"ContainerStarted","Data":"adfb26dcdb3568b2f1c556257ab01c7166d4da01503d2c967393c348a3c66603"}
Sep 08 13:18:54.779869 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:54.779823    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-6f4946db56-qnzxl" podStartSLOduration=2.814759197 podStartE2EDuration="4.779808688s" totalImagesPullingTime="1.965049491s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:18:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:18:54.778281265 +0000 UTC m=+185.049729206" watchObservedRunningTime="2026-09-08 13:18:54.779808688 +0000 UTC m=+185.051256619"
Sep 08 13:18:55.753545 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:55.753491    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:18:56.377742 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:56.377713    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-nv6pk"]
Sep 08 13:18:56.380800 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:56.380781    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-nv6pk"
Sep 08 13:18:56.384280 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:56.384263    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 08 13:18:56.384668 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:56.384653    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-7sv8r\""
Sep 08 13:18:56.384870 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:56.384857    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 08 13:18:56.385054 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:56.385039    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 08 13:18:56.402192 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:56.402174    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/07f9b84f-8894-43de-95df-ee95e733be31-metrics-client-ca\") pod \"node-exporter-nv6pk\" (UID: \"07f9b84f-8894-43de-95df-ee95e733be31\") " pod="openshift-monitoring/node-exporter-nv6pk"
Sep 08 13:18:56.402267 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:56.402199    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/07f9b84f-8894-43de-95df-ee95e733be31-node-exporter-accelerators-collector-config\") pod \"node-exporter-nv6pk\" (UID: \"07f9b84f-8894-43de-95df-ee95e733be31\") " pod="openshift-monitoring/node-exporter-nv6pk"
Sep 08 13:18:56.402267 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:56.402221    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkbs7\" (UniqueName: \"kubernetes.io/projected/07f9b84f-8894-43de-95df-ee95e733be31-kube-api-access-tkbs7\") pod \"node-exporter-nv6pk\" (UID: \"07f9b84f-8894-43de-95df-ee95e733be31\") " pod="openshift-monitoring/node-exporter-nv6pk"
Sep 08 13:18:56.402348 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:56.402291    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/07f9b84f-8894-43de-95df-ee95e733be31-node-exporter-textfile\") pod \"node-exporter-nv6pk\" (UID: \"07f9b84f-8894-43de-95df-ee95e733be31\") " pod="openshift-monitoring/node-exporter-nv6pk"
Sep 08 13:18:56.402348 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:56.402334    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/07f9b84f-8894-43de-95df-ee95e733be31-root\") pod \"node-exporter-nv6pk\" (UID: \"07f9b84f-8894-43de-95df-ee95e733be31\") " pod="openshift-monitoring/node-exporter-nv6pk"
Sep 08 13:18:56.402407 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:56.402358    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/07f9b84f-8894-43de-95df-ee95e733be31-sys\") pod \"node-exporter-nv6pk\" (UID: \"07f9b84f-8894-43de-95df-ee95e733be31\") " pod="openshift-monitoring/node-exporter-nv6pk"
Sep 08 13:18:56.402440 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:56.402428    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/07f9b84f-8894-43de-95df-ee95e733be31-node-exporter-tls\") pod \"node-exporter-nv6pk\" (UID: \"07f9b84f-8894-43de-95df-ee95e733be31\") " pod="openshift-monitoring/node-exporter-nv6pk"
Sep 08 13:18:56.402475 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:56.402450    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/07f9b84f-8894-43de-95df-ee95e733be31-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nv6pk\" (UID: \"07f9b84f-8894-43de-95df-ee95e733be31\") " pod="openshift-monitoring/node-exporter-nv6pk"
Sep 08 13:18:56.402528 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:56.402474    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/07f9b84f-8894-43de-95df-ee95e733be31-node-exporter-wtmp\") pod \"node-exporter-nv6pk\" (UID: \"07f9b84f-8894-43de-95df-ee95e733be31\") " pod="openshift-monitoring/node-exporter-nv6pk"
Sep 08 13:18:56.503332 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:56.503312    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/07f9b84f-8894-43de-95df-ee95e733be31-sys\") pod \"node-exporter-nv6pk\" (UID: \"07f9b84f-8894-43de-95df-ee95e733be31\") " pod="openshift-monitoring/node-exporter-nv6pk"
Sep 08 13:18:56.503438 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:56.503342    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/07f9b84f-8894-43de-95df-ee95e733be31-node-exporter-tls\") pod \"node-exporter-nv6pk\" (UID: \"07f9b84f-8894-43de-95df-ee95e733be31\") " pod="openshift-monitoring/node-exporter-nv6pk"
Sep 08 13:18:56.503438 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:56.503359    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/07f9b84f-8894-43de-95df-ee95e733be31-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nv6pk\" (UID: \"07f9b84f-8894-43de-95df-ee95e733be31\") " pod="openshift-monitoring/node-exporter-nv6pk"
Sep 08 13:18:56.503438 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:56.503375    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/07f9b84f-8894-43de-95df-ee95e733be31-node-exporter-wtmp\") pod \"node-exporter-nv6pk\" (UID: \"07f9b84f-8894-43de-95df-ee95e733be31\") " pod="openshift-monitoring/node-exporter-nv6pk"
Sep 08 13:18:56.503438 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:56.503390    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/07f9b84f-8894-43de-95df-ee95e733be31-metrics-client-ca\") pod \"node-exporter-nv6pk\" (UID: \"07f9b84f-8894-43de-95df-ee95e733be31\") " pod="openshift-monitoring/node-exporter-nv6pk"
Sep 08 13:18:56.503438 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:56.503405    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/07f9b84f-8894-43de-95df-ee95e733be31-node-exporter-accelerators-collector-config\") pod \"node-exporter-nv6pk\" (UID: \"07f9b84f-8894-43de-95df-ee95e733be31\") " pod="openshift-monitoring/node-exporter-nv6pk"
Sep 08 13:18:56.503438 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:56.503420    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkbs7\" (UniqueName: \"kubernetes.io/projected/07f9b84f-8894-43de-95df-ee95e733be31-kube-api-access-tkbs7\") pod \"node-exporter-nv6pk\" (UID: \"07f9b84f-8894-43de-95df-ee95e733be31\") " pod="openshift-monitoring/node-exporter-nv6pk"
Sep 08 13:18:56.503768 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:56.503461    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/07f9b84f-8894-43de-95df-ee95e733be31-sys\") pod \"node-exporter-nv6pk\" (UID: \"07f9b84f-8894-43de-95df-ee95e733be31\") " pod="openshift-monitoring/node-exporter-nv6pk"
Sep 08 13:18:56.503768 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:56.503529    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/07f9b84f-8894-43de-95df-ee95e733be31-node-exporter-textfile\") pod \"node-exporter-nv6pk\" (UID: \"07f9b84f-8894-43de-95df-ee95e733be31\") " pod="openshift-monitoring/node-exporter-nv6pk"
Sep 08 13:18:56.503768 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:56.503546    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/07f9b84f-8894-43de-95df-ee95e733be31-node-exporter-wtmp\") pod \"node-exporter-nv6pk\" (UID: \"07f9b84f-8894-43de-95df-ee95e733be31\") " pod="openshift-monitoring/node-exporter-nv6pk"
Sep 08 13:18:56.503768 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:56.503582    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/07f9b84f-8894-43de-95df-ee95e733be31-root\") pod \"node-exporter-nv6pk\" (UID: \"07f9b84f-8894-43de-95df-ee95e733be31\") " pod="openshift-monitoring/node-exporter-nv6pk"
Sep 08 13:18:56.503768 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:56.503654    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/07f9b84f-8894-43de-95df-ee95e733be31-root\") pod \"node-exporter-nv6pk\" (UID: \"07f9b84f-8894-43de-95df-ee95e733be31\") " pod="openshift-monitoring/node-exporter-nv6pk"
Sep 08 13:18:56.504022 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:56.503883    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/07f9b84f-8894-43de-95df-ee95e733be31-node-exporter-textfile\") pod \"node-exporter-nv6pk\" (UID: \"07f9b84f-8894-43de-95df-ee95e733be31\") " pod="openshift-monitoring/node-exporter-nv6pk"
Sep 08 13:18:56.504101 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:56.504083    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/07f9b84f-8894-43de-95df-ee95e733be31-metrics-client-ca\") pod \"node-exporter-nv6pk\" (UID: \"07f9b84f-8894-43de-95df-ee95e733be31\") " pod="openshift-monitoring/node-exporter-nv6pk"
Sep 08 13:18:56.504156 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:56.504084    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/07f9b84f-8894-43de-95df-ee95e733be31-node-exporter-accelerators-collector-config\") pod \"node-exporter-nv6pk\" (UID: \"07f9b84f-8894-43de-95df-ee95e733be31\") " pod="openshift-monitoring/node-exporter-nv6pk"
Sep 08 13:18:56.505613 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:56.505594    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/07f9b84f-8894-43de-95df-ee95e733be31-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nv6pk\" (UID: \"07f9b84f-8894-43de-95df-ee95e733be31\") " pod="openshift-monitoring/node-exporter-nv6pk"
Sep 08 13:18:56.505716 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:56.505700    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/07f9b84f-8894-43de-95df-ee95e733be31-node-exporter-tls\") pod \"node-exporter-nv6pk\" (UID: \"07f9b84f-8894-43de-95df-ee95e733be31\") " pod="openshift-monitoring/node-exporter-nv6pk"
Sep 08 13:18:56.511730 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:56.511711    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkbs7\" (UniqueName: \"kubernetes.io/projected/07f9b84f-8894-43de-95df-ee95e733be31-kube-api-access-tkbs7\") pod \"node-exporter-nv6pk\" (UID: \"07f9b84f-8894-43de-95df-ee95e733be31\") " pod="openshift-monitoring/node-exporter-nv6pk"
Sep 08 13:18:56.689832 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:56.689759    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-nv6pk"
Sep 08 13:18:56.700248 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:18:56.700224    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07f9b84f_8894_43de_95df_ee95e733be31.slice/crio-a53823299702c08c56ac7befb2c1b41074fc58cd3dccc944c171c35ae9920d04 WatchSource:0}: Error finding container a53823299702c08c56ac7befb2c1b41074fc58cd3dccc944c171c35ae9920d04: Status 404 returned error can't find the container with id a53823299702c08c56ac7befb2c1b41074fc58cd3dccc944c171c35ae9920d04
Sep 08 13:18:56.769350 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:56.769320    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nv6pk" event={"ID":"07f9b84f-8894-43de-95df-ee95e733be31","Type":"ContainerStarted","Data":"a53823299702c08c56ac7befb2c1b41074fc58cd3dccc944c171c35ae9920d04"}
Sep 08 13:18:58.911742 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:58.911709    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:18:59.017955 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:59.017892    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:18:59.372007 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:59.371973    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:18:59.494720 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:59.494688    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:18:59.610292 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:59.610261    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:18:59.723545 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:59.723461    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:18:59.781546 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:59.781522    2573 generic.go:356] "Generic (PLEG): container finished" podID="07f9b84f-8894-43de-95df-ee95e733be31" containerID="4ed571a62e85390185cc18dfd5047d1bf5ee73ffae336ab6da2da3b369cdb22d" exitCode=0
Sep 08 13:18:59.781687 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:18:59.781599    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nv6pk" event={"ID":"07f9b84f-8894-43de-95df-ee95e733be31","Type":"ContainerDied","Data":"4ed571a62e85390185cc18dfd5047d1bf5ee73ffae336ab6da2da3b369cdb22d"}
Sep 08 13:19:00.694460 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:19:00.694425    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-64b7f4f884-ds5dg"]
Sep 08 13:19:00.697510 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:19:00.697487    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-64b7f4f884-ds5dg"
Sep 08 13:19:00.702878 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:19:00.702848    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-tkj47\""
Sep 08 13:19:00.703021 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:19:00.702881    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 08 13:19:00.703021 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:19:00.702922    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 08 13:19:00.703147 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:19:00.703042    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-12d2hed2faek5\""
Sep 08 13:19:00.703212 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:19:00.703172    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 08 13:19:00.703212 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:19:00.703194    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 08 13:19:00.729844 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:19:00.729820    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-64b7f4f884-ds5dg"]
Sep 08 13:19:00.737015 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:19:00.736991    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/0a3e2a72-2d91-42e4-b48e-ecaccc6cafd8-metrics-server-audit-profiles\") pod \"metrics-server-64b7f4f884-ds5dg\" (UID: \"0a3e2a72-2d91-42e4-b48e-ecaccc6cafd8\") " pod="openshift-monitoring/metrics-server-64b7f4f884-ds5dg"
Sep 08 13:19:00.737116 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:19:00.737031    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a3e2a72-2d91-42e4-b48e-ecaccc6cafd8-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-64b7f4f884-ds5dg\" (UID: \"0a3e2a72-2d91-42e4-b48e-ecaccc6cafd8\") " pod="openshift-monitoring/metrics-server-64b7f4f884-ds5dg"
Sep 08 13:19:00.737116 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:19:00.737061    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/0a3e2a72-2d91-42e4-b48e-ecaccc6cafd8-audit-log\") pod \"metrics-server-64b7f4f884-ds5dg\" (UID: \"0a3e2a72-2d91-42e4-b48e-ecaccc6cafd8\") " pod="openshift-monitoring/metrics-server-64b7f4f884-ds5dg"
Sep 08 13:19:00.737116 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:19:00.737100    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/0a3e2a72-2d91-42e4-b48e-ecaccc6cafd8-secret-metrics-server-tls\") pod \"metrics-server-64b7f4f884-ds5dg\" (UID: \"0a3e2a72-2d91-42e4-b48e-ecaccc6cafd8\") " pod="openshift-monitoring/metrics-server-64b7f4f884-ds5dg"
Sep 08 13:19:00.737277 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:19:00.737140    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/0a3e2a72-2d91-42e4-b48e-ecaccc6cafd8-client-ca-bundle\") pod \"metrics-server-64b7f4f884-ds5dg\" (UID: \"0a3e2a72-2d91-42e4-b48e-ecaccc6cafd8\") " pod="openshift-monitoring/metrics-server-64b7f4f884-ds5dg"
Sep 08 13:19:00.737277 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:19:00.737166    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xk2dr\" (UniqueName: \"kubernetes.io/projected/0a3e2a72-2d91-42e4-b48e-ecaccc6cafd8-kube-api-access-xk2dr\") pod \"metrics-server-64b7f4f884-ds5dg\" (UID: \"0a3e2a72-2d91-42e4-b48e-ecaccc6cafd8\") " pod="openshift-monitoring/metrics-server-64b7f4f884-ds5dg"
Sep 08 13:19:00.737277 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:19:00.737186    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/0a3e2a72-2d91-42e4-b48e-ecaccc6cafd8-secret-metrics-server-client-certs\") pod \"metrics-server-64b7f4f884-ds5dg\" (UID: \"0a3e2a72-2d91-42e4-b48e-ecaccc6cafd8\") " pod="openshift-monitoring/metrics-server-64b7f4f884-ds5dg"
Sep 08 13:19:00.787177 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:19:00.787155    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nv6pk" event={"ID":"07f9b84f-8894-43de-95df-ee95e733be31","Type":"ContainerStarted","Data":"c608e8b88f1ce04f692326a9e7aca3a7fc621438d8b65eb1771cf94cf6d27d54"}
Sep 08 13:19:00.787253 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:19:00.787193    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nv6pk" event={"ID":"07f9b84f-8894-43de-95df-ee95e733be31","Type":"ContainerStarted","Data":"0dff98228331eb3134b6c55feb29295966c596bc37f1491e7b8cb0c62cd72941"}
Sep 08 13:19:00.805557 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:19:00.805497    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-nv6pk" podStartSLOduration=2.720569299 podStartE2EDuration="4.805485467s" totalImagesPullingTime="2.025026045s" totalInitContainerRuntime="59.890123ms" isStatefulPod=true podCreationTimestamp="2026-09-08 13:18:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:19:00.804772192 +0000 UTC m=+191.076220124" watchObservedRunningTime="2026-09-08 13:19:00.805485467 +0000 UTC m=+191.076933398"
Sep 08 13:19:00.837564 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:19:00.837541    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/0a3e2a72-2d91-42e4-b48e-ecaccc6cafd8-client-ca-bundle\") pod \"metrics-server-64b7f4f884-ds5dg\" (UID: \"0a3e2a72-2d91-42e4-b48e-ecaccc6cafd8\") " pod="openshift-monitoring/metrics-server-64b7f4f884-ds5dg"
Sep 08 13:19:00.837633 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:19:00.837568    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xk2dr\" (UniqueName: \"kubernetes.io/projected/0a3e2a72-2d91-42e4-b48e-ecaccc6cafd8-kube-api-access-xk2dr\") pod \"metrics-server-64b7f4f884-ds5dg\" (UID: \"0a3e2a72-2d91-42e4-b48e-ecaccc6cafd8\") " pod="openshift-monitoring/metrics-server-64b7f4f884-ds5dg"
Sep 08 13:19:00.837633 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:19:00.837588    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/0a3e2a72-2d91-42e4-b48e-ecaccc6cafd8-secret-metrics-server-client-certs\") pod \"metrics-server-64b7f4f884-ds5dg\" (UID: \"0a3e2a72-2d91-42e4-b48e-ecaccc6cafd8\") " pod="openshift-monitoring/metrics-server-64b7f4f884-ds5dg"
Sep 08 13:19:00.837633 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:19:00.837619    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/0a3e2a72-2d91-42e4-b48e-ecaccc6cafd8-metrics-server-audit-profiles\") pod \"metrics-server-64b7f4f884-ds5dg\" (UID: \"0a3e2a72-2d91-42e4-b48e-ecaccc6cafd8\") " pod="openshift-monitoring/metrics-server-64b7f4f884-ds5dg"
Sep 08 13:19:00.837751 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:19:00.837694    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a3e2a72-2d91-42e4-b48e-ecaccc6cafd8-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-64b7f4f884-ds5dg\" (UID: \"0a3e2a72-2d91-42e4-b48e-ecaccc6cafd8\") " pod="openshift-monitoring/metrics-server-64b7f4f884-ds5dg"
Sep 08 13:19:00.837798 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:19:00.837764    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/0a3e2a72-2d91-42e4-b48e-ecaccc6cafd8-audit-log\") pod \"metrics-server-64b7f4f884-ds5dg\" (UID: \"0a3e2a72-2d91-42e4-b48e-ecaccc6cafd8\") " pod="openshift-monitoring/metrics-server-64b7f4f884-ds5dg"
Sep 08 13:19:00.837858 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:19:00.837819    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/0a3e2a72-2d91-42e4-b48e-ecaccc6cafd8-secret-metrics-server-tls\") pod \"metrics-server-64b7f4f884-ds5dg\" (UID: \"0a3e2a72-2d91-42e4-b48e-ecaccc6cafd8\") " pod="openshift-monitoring/metrics-server-64b7f4f884-ds5dg"
Sep 08 13:19:00.838262 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:19:00.838232    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/0a3e2a72-2d91-42e4-b48e-ecaccc6cafd8-audit-log\") pod \"metrics-server-64b7f4f884-ds5dg\" (UID: \"0a3e2a72-2d91-42e4-b48e-ecaccc6cafd8\") " pod="openshift-monitoring/metrics-server-64b7f4f884-ds5dg"
Sep 08 13:19:00.838480 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:19:00.838460    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a3e2a72-2d91-42e4-b48e-ecaccc6cafd8-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-64b7f4f884-ds5dg\" (UID: \"0a3e2a72-2d91-42e4-b48e-ecaccc6cafd8\") " pod="openshift-monitoring/metrics-server-64b7f4f884-ds5dg"
Sep 08 13:19:00.838652 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:19:00.838626    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/0a3e2a72-2d91-42e4-b48e-ecaccc6cafd8-metrics-server-audit-profiles\") pod \"metrics-server-64b7f4f884-ds5dg\" (UID: \"0a3e2a72-2d91-42e4-b48e-ecaccc6cafd8\") " pod="openshift-monitoring/metrics-server-64b7f4f884-ds5dg"
Sep 08 13:19:00.840000 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:19:00.839973    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/0a3e2a72-2d91-42e4-b48e-ecaccc6cafd8-client-ca-bundle\") pod \"metrics-server-64b7f4f884-ds5dg\" (UID: \"0a3e2a72-2d91-42e4-b48e-ecaccc6cafd8\") " pod="openshift-monitoring/metrics-server-64b7f4f884-ds5dg"
Sep 08 13:19:00.840162 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:19:00.840145    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/0a3e2a72-2d91-42e4-b48e-ecaccc6cafd8-secret-metrics-server-client-certs\") pod \"metrics-server-64b7f4f884-ds5dg\" (UID: \"0a3e2a72-2d91-42e4-b48e-ecaccc6cafd8\") " pod="openshift-monitoring/metrics-server-64b7f4f884-ds5dg"
Sep 08 13:19:00.840222 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:19:00.840205    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/0a3e2a72-2d91-42e4-b48e-ecaccc6cafd8-secret-metrics-server-tls\") pod \"metrics-server-64b7f4f884-ds5dg\" (UID: \"0a3e2a72-2d91-42e4-b48e-ecaccc6cafd8\") " pod="openshift-monitoring/metrics-server-64b7f4f884-ds5dg"
Sep 08 13:19:00.846483 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:19:00.846462    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xk2dr\" (UniqueName: \"kubernetes.io/projected/0a3e2a72-2d91-42e4-b48e-ecaccc6cafd8-kube-api-access-xk2dr\") pod \"metrics-server-64b7f4f884-ds5dg\" (UID: \"0a3e2a72-2d91-42e4-b48e-ecaccc6cafd8\") " pod="openshift-monitoring/metrics-server-64b7f4f884-ds5dg"
Sep 08 13:19:01.007025 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:19:01.006964    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-64b7f4f884-ds5dg"
Sep 08 13:19:01.166036 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:19:01.165996    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:01.257553 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:19:01.257465    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:01.280278 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:19:01.280248    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-64b7f4f884-ds5dg"]
Sep 08 13:19:01.282937 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:19:01.282911    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a3e2a72_2d91_42e4_b48e_ecaccc6cafd8.slice/crio-09e9c07c781ce44173a6929647f5f55b3b446e4c5f077c1e96c77ceb3b36d831 WatchSource:0}: Error finding container 09e9c07c781ce44173a6929647f5f55b3b446e4c5f077c1e96c77ceb3b36d831: Status 404 returned error can't find the container with id 09e9c07c781ce44173a6929647f5f55b3b446e4c5f077c1e96c77ceb3b36d831
Sep 08 13:19:01.538396 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:19:01.538307    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:01.791975 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:19:01.791890    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-64b7f4f884-ds5dg" event={"ID":"0a3e2a72-2d91-42e4-b48e-ecaccc6cafd8","Type":"ContainerStarted","Data":"09e9c07c781ce44173a6929647f5f55b3b446e4c5f077c1e96c77ceb3b36d831"}
Sep 08 13:19:02.273194 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:19:02.273158    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:02.631665 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:19:02.631630    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:02.640575 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:19:02.640550    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 08 13:19:02.645314 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:19:02.645286    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:19:02.650677 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:19:02.650650    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 08 13:19:02.651355 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:19:02.651337    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 08 13:19:02.651595 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:19:02.651569    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 08 13:19:02.651915 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:19:02.651889    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 08 13:19:02.652109 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:19:02.652091    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 08 13:19:02.652294 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:19:02.652275    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 08 13:19:02.652765 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:19:02.652737    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 08 13:19:02.652857 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:19:02.652821    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 08 13:19:02.652928 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:19:02.652751    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-bbm9m\""
Sep 08 13:19:02.653081 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:19:02.653058    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 08 13:19:02.653669 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:19:02.653115    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 08 13:19:02.653790 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:19:02.653115    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 08 13:19:02.653790 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:19:02.653446    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-14vno50in3h6s\""
Sep 08 13:19:02.659403 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:19:02.659380    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 08 13:19:02.666905 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:19:02.666881    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 08 13:19:02.669308 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:19:02.669291    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 08 13:19:02.680459 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:19:02.680437    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 08 13:19:02.750780 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:19:02.750750    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/89edf18f-ede2-4d2b-9ad4-8fcdaa4d3cff-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"89edf18f-ede2-4d2b-9ad4-8fcdaa4d3cff\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:19:02.750907 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:19:02.750789    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/89edf18f-ede2-4d2b-9ad4-8fcdaa4d3cff-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"89edf18f-ede2-4d2b-9ad4-8fcdaa4d3cff\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:19:02.750907 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:19:02.750816    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/89edf18f-ede2-4d2b-9ad4-8fcdaa4d3cff-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"89edf18f-ede2-4d2b-9ad4-8fcdaa4d3cff\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:19:02.750907 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:19:02.750855    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/89edf18f-ede2-4d2b-9ad4-8fcdaa4d3cff-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"89edf18f-ede2-4d2b-9ad4-8fcdaa4d3cff\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:19:02.751036 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:19:02.750930    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvhjr\" (UniqueName: \"kubernetes.io/projected/89edf18f-ede2-4d2b-9ad4-8fcdaa4d3cff-kube-api-access-jvhjr\") pod \"prometheus-k8s-0\" (UID: \"89edf18f-ede2-4d2b-9ad4-8fcdaa4d3cff\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:19:02.751036 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:19:02.750983    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/89edf18f-ede2-4d2b-9ad4-8fcdaa4d3cff-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"89edf18f-ede2-4d2b-9ad4-8fcdaa4d3cff\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:19:02.751036 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:19:02.751023    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/89edf18f-ede2-4d2b-9ad4-8fcdaa4d3cff-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"89edf18f-ede2-4d2b-9ad4-8fcdaa4d3cff\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:19:02.751173 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:19:02.751038    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/89edf18f-ede2-4d2b-9ad4-8fcdaa4d3cff-web-config\") pod \"prometheus-k8s-0\" (UID: \"89edf18f-ede2-4d2b-9ad4-8fcdaa4d3cff\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:19:02.751173 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:19:02.751061    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/89edf18f-ede2-4d2b-9ad4-8fcdaa4d3cff-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"89edf18f-ede2-4d2b-9ad4-8fcdaa4d3cff\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:19:02.751173 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:19:02.751154    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/89edf18f-ede2-4d2b-9ad4-8fcdaa4d3cff-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"89edf18f-ede2-4d2b-9ad4-8fcdaa4d3cff\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:19:02.751328 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:19:02.751190    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/89edf18f-ede2-4d2b-9ad4-8fcdaa4d3cff-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"89edf18f-ede2-4d2b-9ad4-8fcdaa4d3cff\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:19:02.751328 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:19:02.751226    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/89edf18f-ede2-4d2b-9ad4-8fcdaa4d3cff-config\") pod \"prometheus-k8s-0\" (UID: \"89edf18f-ede2-4d2b-9ad4-8fcdaa4d3cff\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:19:02.751328 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:19:02.751260    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/89edf18f-ede2-4d2b-9ad4-8fcdaa4d3cff-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"89edf18f-ede2-4d2b-9ad4-8fcdaa4d3cff\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:19:02.751328 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:19:02.751289    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/89edf18f-ede2-4d2b-9ad4-8fcdaa4d3cff-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"89edf18f-ede2-4d2b-9ad4-8fcdaa4d3cff\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:19:02.751546 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:19:02.751359    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/89edf18f-ede2-4d2b-9ad4-8fcdaa4d3cff-config-out\") pod \"prometheus-k8s-0\" (UID: \"89edf18f-ede2-4d2b-9ad4-8fcdaa4d3cff\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:19:02.751546 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:19:02.751392    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/89edf18f-ede2-4d2b-9ad4-8fcdaa4d3cff-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"89edf18f-ede2-4d2b-9ad4-8fcdaa4d3cff\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:19:02.751546 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:19:02.751427    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/89edf18f-ede2-4d2b-9ad4-8fcdaa4d3cff-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"89edf18f-ede2-4d2b-9ad4-8fcdaa4d3cff\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:19:02.751546 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:19:02.751456    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/89edf18f-ede2-4d2b-9ad4-8fcdaa4d3cff-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"89edf18f-ede2-4d2b-9ad4-8fcdaa4d3cff\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:19:02.751546 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:19:02.751487    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/89edf18f-ede2-4d2b-9ad4-8fcdaa4d3cff-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"89edf18f-ede2-4d2b-9ad4-8fcdaa4d3cff\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:19:02.751546 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:19:02.751543    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/89edf18f-ede2-4d2b-9ad4-8fcdaa4d3cff-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"89edf18f-ede2-4d2b-9ad4-8fcdaa4d3cff\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:19:02.852168 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:19:02.852132    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/89edf18f-ede2-4d2b-9ad4-8fcdaa4d3cff-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"89edf18f-ede2-4d2b-9ad4-8fcdaa4d3cff\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:19:02.852673 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:19:02.852652    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/89edf18f-ede2-4d2b-9ad4-8fcdaa4d3cff-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"89edf18f-ede2-4d2b-9ad4-8fcdaa4d3cff\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:19:02.852828 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:19:02.852813    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/89edf18f-ede2-4d2b-9ad4-8fcdaa4d3cff-config-out\") pod \"prometheus-k8s-0\" (UID: \"89edf18f-ede2-4d2b-9ad4-8fcdaa4d3cff\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:19:02.852922 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:19:02.852909    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/89edf18f-ede2-4d2b-9ad4-8fcdaa4d3cff-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"89edf18f-ede2-4d2b-9ad4-8fcdaa4d3cff\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:19:02.853025 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:19:02.853014    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/89edf18f-ede2-4d2b-9ad4-8fcdaa4d3cff-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"89edf18f-ede2-4d2b-9ad4-8fcdaa4d3cff\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:19:02.853138 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:19:02.853125    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/89edf18f-ede2-4d2b-9ad4-8fcdaa4d3cff-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"89edf18f-ede2-4d2b-9ad4-8fcdaa4d3cff\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:19:02.853226 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:19:02.853212    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/89edf18f-ede2-4d2b-9ad4-8fcdaa4d3cff-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"89edf18f-ede2-4d2b-9ad4-8fcdaa4d3cff\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:19:02.853319 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:19:02.853306    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/89edf18f-ede2-4d2b-9ad4-8fcdaa4d3cff-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"89edf18f-ede2-4d2b-9ad4-8fcdaa4d3cff\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:19:02.853414 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:19:02.853403    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/89edf18f-ede2-4d2b-9ad4-8fcdaa4d3cff-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"89edf18f-ede2-4d2b-9ad4-8fcdaa4d3cff\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:19:02.853493 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:19:02.853481    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/89edf18f-ede2-4d2b-9ad4-8fcdaa4d3cff-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"89edf18f-ede2-4d2b-9ad4-8fcdaa4d3cff\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:19:02.853854 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:19:02.853663    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/89edf18f-ede2-4d2b-9ad4-8fcdaa4d3cff-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"89edf18f-ede2-4d2b-9ad4-8fcdaa4d3cff\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:19:02.853854 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:19:02.853695    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/89edf18f-ede2-4d2b-9ad4-8fcdaa4d3cff-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"89edf18f-ede2-4d2b-9ad4-8fcdaa4d3cff\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:19:02.853854 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:19:02.853717    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/89edf18f-ede2-4d2b-9ad4-8fcdaa4d3cff-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"89edf18f-ede2-4d2b-9ad4-8fcdaa4d3cff\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:19:02.853854 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:19:02.853745    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/89edf18f-ede2-4d2b-9ad4-8fcdaa4d3cff-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"89edf18f-ede2-4d2b-9ad4-8fcdaa4d3cff\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:19:02.853854 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:19:02.853772    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvhjr\" (UniqueName: \"kubernetes.io/projected/89edf18f-ede2-4d2b-9ad4-8fcdaa4d3cff-kube-api-access-jvhjr\") pod \"prometheus-k8s-0\" (UID: \"89edf18f-ede2-4d2b-9ad4-8fcdaa4d3cff\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:19:02.853854 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:19:02.853801    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/89edf18f-ede2-4d2b-9ad4-8fcdaa4d3cff-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"89edf18f-ede2-4d2b-9ad4-8fcdaa4d3cff\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:19:02.853854 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:19:02.853857    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/89edf18f-ede2-4d2b-9ad4-8fcdaa4d3cff-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"89edf18f-ede2-4d2b-9ad4-8fcdaa4d3cff\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:19:02.854228 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:19:02.853881    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/89edf18f-ede2-4d2b-9ad4-8fcdaa4d3cff-web-config\") pod \"prometheus-k8s-0\" (UID: \"89edf18f-ede2-4d2b-9ad4-8fcdaa4d3cff\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:19:02.854228 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:19:02.853904    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/89edf18f-ede2-4d2b-9ad4-8fcdaa4d3cff-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"89edf18f-ede2-4d2b-9ad4-8fcdaa4d3cff\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:19:02.854228 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:19:02.853937    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/89edf18f-ede2-4d2b-9ad4-8fcdaa4d3cff-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"89edf18f-ede2-4d2b-9ad4-8fcdaa4d3cff\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:19:02.854228 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:19:02.853960    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/89edf18f-ede2-4d2b-9ad4-8fcdaa4d3cff-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"89edf18f-ede2-4d2b-9ad4-8fcdaa4d3cff\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:19:02.854228 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:19:02.853991    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/89edf18f-ede2-4d2b-9ad4-8fcdaa4d3cff-config\") pod \"prometheus-k8s-0\" (UID: \"89edf18f-ede2-4d2b-9ad4-8fcdaa4d3cff\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:19:02.854478 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:19:02.852923    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/89edf18f-ede2-4d2b-9ad4-8fcdaa4d3cff-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"89edf18f-ede2-4d2b-9ad4-8fcdaa4d3cff\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:19:02.855897 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:19:02.854881    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/89edf18f-ede2-4d2b-9ad4-8fcdaa4d3cff-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"89edf18f-ede2-4d2b-9ad4-8fcdaa4d3cff\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:19:02.855897 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:19:02.854978    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/89edf18f-ede2-4d2b-9ad4-8fcdaa4d3cff-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"89edf18f-ede2-4d2b-9ad4-8fcdaa4d3cff\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:19:02.855897 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:19:02.855170    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/89edf18f-ede2-4d2b-9ad4-8fcdaa4d3cff-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"89edf18f-ede2-4d2b-9ad4-8fcdaa4d3cff\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:19:02.855897 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:19:02.855604    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/89edf18f-ede2-4d2b-9ad4-8fcdaa4d3cff-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"89edf18f-ede2-4d2b-9ad4-8fcdaa4d3cff\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:19:02.857339 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:19:02.857319    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/89edf18f-ede2-4d2b-9ad4-8fcdaa4d3cff-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"89edf18f-ede2-4d2b-9ad4-8fcdaa4d3cff\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:19:02.859054 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:19:02.859027    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/89edf18f-ede2-4d2b-9ad4-8fcdaa4d3cff-config\") pod \"prometheus-k8s-0\" (UID: \"89edf18f-ede2-4d2b-9ad4-8fcdaa4d3cff\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:19:02.859142 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:19:02.859032    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/89edf18f-ede2-4d2b-9ad4-8fcdaa4d3cff-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"89edf18f-ede2-4d2b-9ad4-8fcdaa4d3cff\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:19:02.859212 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:19:02.859144    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/89edf18f-ede2-4d2b-9ad4-8fcdaa4d3cff-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"89edf18f-ede2-4d2b-9ad4-8fcdaa4d3cff\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:19:02.859469 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:19:02.859448    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/89edf18f-ede2-4d2b-9ad4-8fcdaa4d3cff-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"89edf18f-ede2-4d2b-9ad4-8fcdaa4d3cff\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:19:02.859544 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:19:02.859477    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/89edf18f-ede2-4d2b-9ad4-8fcdaa4d3cff-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"89edf18f-ede2-4d2b-9ad4-8fcdaa4d3cff\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:19:02.859677 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:19:02.859639    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/89edf18f-ede2-4d2b-9ad4-8fcdaa4d3cff-config-out\") pod \"prometheus-k8s-0\" (UID: \"89edf18f-ede2-4d2b-9ad4-8fcdaa4d3cff\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:19:02.859776 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:19:02.859674    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/89edf18f-ede2-4d2b-9ad4-8fcdaa4d3cff-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"89edf18f-ede2-4d2b-9ad4-8fcdaa4d3cff\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:19:02.859957 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:19:02.859915    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/89edf18f-ede2-4d2b-9ad4-8fcdaa4d3cff-web-config\") pod \"prometheus-k8s-0\" (UID: \"89edf18f-ede2-4d2b-9ad4-8fcdaa4d3cff\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:19:02.860746 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:19:02.860721    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/89edf18f-ede2-4d2b-9ad4-8fcdaa4d3cff-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"89edf18f-ede2-4d2b-9ad4-8fcdaa4d3cff\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:19:02.860855 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:19:02.860720    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/89edf18f-ede2-4d2b-9ad4-8fcdaa4d3cff-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"89edf18f-ede2-4d2b-9ad4-8fcdaa4d3cff\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:19:02.861205 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:19:02.861182    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/89edf18f-ede2-4d2b-9ad4-8fcdaa4d3cff-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"89edf18f-ede2-4d2b-9ad4-8fcdaa4d3cff\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:19:02.865447 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:19:02.865427    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvhjr\" (UniqueName: \"kubernetes.io/projected/89edf18f-ede2-4d2b-9ad4-8fcdaa4d3cff-kube-api-access-jvhjr\") pod \"prometheus-k8s-0\" (UID: \"89edf18f-ede2-4d2b-9ad4-8fcdaa4d3cff\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:19:02.958711 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:19:02.958630    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:19:03.128075 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:19:03.128042    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:03.228900 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:19:03.228834    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:03.252015 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:19:03.251987    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 08 13:19:03.254597 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:19:03.254570    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89edf18f_ede2_4d2b_9ad4_8fcdaa4d3cff.slice/crio-c30166ffad10fb588227b86cd73dd1ee53473779c9a917deb0ca5a3673ae468b WatchSource:0}: Error finding container c30166ffad10fb588227b86cd73dd1ee53473779c9a917deb0ca5a3673ae468b: Status 404 returned error can't find the container with id c30166ffad10fb588227b86cd73dd1ee53473779c9a917deb0ca5a3673ae468b
Sep 08 13:19:03.800884 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:19:03.800850    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"89edf18f-ede2-4d2b-9ad4-8fcdaa4d3cff","Type":"ContainerStarted","Data":"c30166ffad10fb588227b86cd73dd1ee53473779c9a917deb0ca5a3673ae468b"}
Sep 08 13:19:06.812682 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:19:06.812653    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-64b7f4f884-ds5dg" event={"ID":"0a3e2a72-2d91-42e4-b48e-ecaccc6cafd8","Type":"ContainerStarted","Data":"b7b25d066d5d6273e21e08fb6295f0c1384bfc30c0f1405de44968855dbc199f"}
Sep 08 13:19:06.833088 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:19:06.833041    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-64b7f4f884-ds5dg" podStartSLOduration=1.376119032 podStartE2EDuration="6.833027159s" totalImagesPullingTime="5.456908127s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:19:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:19:06.832144705 +0000 UTC m=+197.103592663" watchObservedRunningTime="2026-09-08 13:19:06.833027159 +0000 UTC m=+197.104475090"
Sep 08 13:19:07.819074 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:19:07.818894    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"89edf18f-ede2-4d2b-9ad4-8fcdaa4d3cff","Type":"ContainerStarted","Data":"beb9375e07b4ce0af985662fbf4c6d6ad921cb701814214c56e87372c0beedbc"}
Sep 08 13:19:08.823205 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:19:08.823167    2573 generic.go:356] "Generic (PLEG): container finished" podID="89edf18f-ede2-4d2b-9ad4-8fcdaa4d3cff" containerID="beb9375e07b4ce0af985662fbf4c6d6ad921cb701814214c56e87372c0beedbc" exitCode=0
Sep 08 13:19:08.823608 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:19:08.823245    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"89edf18f-ede2-4d2b-9ad4-8fcdaa4d3cff","Type":"ContainerDied","Data":"beb9375e07b4ce0af985662fbf4c6d6ad921cb701814214c56e87372c0beedbc"}
Sep 08 13:19:12.776836 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:19:12.776802    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:12.843091 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:19:12.843061    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"89edf18f-ede2-4d2b-9ad4-8fcdaa4d3cff","Type":"ContainerStarted","Data":"703265dfc3fa8566a2721c263d942a07dc8216a4e977b6a9705921acc1e2ae93"}
Sep 08 13:19:12.843226 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:19:12.843148    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"89edf18f-ede2-4d2b-9ad4-8fcdaa4d3cff","Type":"ContainerStarted","Data":"0391265716e66d7e67e3e65d5a1015fb62904eeda3c32ceec73b869730da3567"}
Sep 08 13:19:12.848211 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:19:12.848183    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:13.965314 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:19:13.965282    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:14.058293 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:19:14.058265    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:14.857521 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:19:14.856889    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"89edf18f-ede2-4d2b-9ad4-8fcdaa4d3cff","Type":"ContainerStarted","Data":"552795d1eccc522e3bb7491b13c8891d8dce6b0f75984b21a5df66aae628f1a2"}
Sep 08 13:19:14.857521 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:19:14.856969    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"89edf18f-ede2-4d2b-9ad4-8fcdaa4d3cff","Type":"ContainerStarted","Data":"0b36cefe9488f85364a4b2f9b1dd3f957c98732c3b4865633176144dc206eb56"}
Sep 08 13:19:14.857521 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:19:14.857008    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"89edf18f-ede2-4d2b-9ad4-8fcdaa4d3cff","Type":"ContainerStarted","Data":"859a411f01e60a98ced32a92742ec42bb4a75fd7f29ac22e79ffacaca71349e6"}
Sep 08 13:19:14.857521 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:19:14.857044    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"89edf18f-ede2-4d2b-9ad4-8fcdaa4d3cff","Type":"ContainerStarted","Data":"72259579a577c71ebbefe62ad782484d74db4df6f7beaafc9c14b6436b6d10f2"}
Sep 08 13:19:14.890751 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:19:14.890694    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.755250511 podStartE2EDuration="12.890674654s" totalImagesPullingTime="10.111438011s" totalInitContainerRuntime="23.986132ms" isStatefulPod=false podCreationTimestamp="2026-09-08 13:19:02 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:19:14.888960736 +0000 UTC m=+205.160408705" watchObservedRunningTime="2026-09-08 13:19:14.890674654 +0000 UTC m=+205.162122606"
Sep 08 13:19:17.959002 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:19:17.958969    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:19:20.019686 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:19:20.019656    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:20.120279 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:19:20.120248    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:21.007599 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:19:21.007570    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-64b7f4f884-ds5dg"
Sep 08 13:19:21.007740 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:19:21.007628    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-64b7f4f884-ds5dg"
Sep 08 13:19:21.147166 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:19:21.147134    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:28.947110 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:19:28.947034    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:29.044084 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:19:29.044056    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:30.594590 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:19:30.594548    2573 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64bfdb6bcc-6lc4p" podUID="ffc5700f-1e9f-47a0-a0f0-98fc161b51e5" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 08 13:19:40.594515 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:19:40.594459    2573 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64bfdb6bcc-6lc4p" podUID="ffc5700f-1e9f-47a0-a0f0-98fc161b51e5" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 08 13:19:41.012726 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:19:41.012691    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-64b7f4f884-ds5dg"
Sep 08 13:19:41.017878 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:19:41.017849    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-64b7f4f884-ds5dg"
Sep 08 13:19:50.593723 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:19:50.593686    2573 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64bfdb6bcc-6lc4p" podUID="ffc5700f-1e9f-47a0-a0f0-98fc161b51e5" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 08 13:19:50.594083 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:19:50.593752    2573 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64bfdb6bcc-6lc4p"
Sep 08 13:19:50.594232 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:19:50.594201    2573 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"aa45d25b4ba890d745b15e2d2479167623cc0989f7af0eaa12c20de58b98cc3f"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64bfdb6bcc-6lc4p" containerMessage="Container service-proxy failed liveness probe, will be restarted"
Sep 08 13:19:50.594270 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:19:50.594254    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64bfdb6bcc-6lc4p" podUID="ffc5700f-1e9f-47a0-a0f0-98fc161b51e5" containerName="service-proxy" containerID="cri-o://aa45d25b4ba890d745b15e2d2479167623cc0989f7af0eaa12c20de58b98cc3f" gracePeriod=30
Sep 08 13:19:50.992497 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:19:50.992468    2573 generic.go:356] "Generic (PLEG): container finished" podID="ffc5700f-1e9f-47a0-a0f0-98fc161b51e5" containerID="aa45d25b4ba890d745b15e2d2479167623cc0989f7af0eaa12c20de58b98cc3f" exitCode=2
Sep 08 13:19:50.992647 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:19:50.992535    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64bfdb6bcc-6lc4p" event={"ID":"ffc5700f-1e9f-47a0-a0f0-98fc161b51e5","Type":"ContainerDied","Data":"aa45d25b4ba890d745b15e2d2479167623cc0989f7af0eaa12c20de58b98cc3f"}
Sep 08 13:19:50.992647 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:19:50.992582    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64bfdb6bcc-6lc4p" event={"ID":"ffc5700f-1e9f-47a0-a0f0-98fc161b51e5","Type":"ContainerStarted","Data":"832522c903b2c3b5eb71eb76968402907f271e33b05e16c865e606383643e17b"}
Sep 08 13:19:51.108952 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:19:51.108924    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:51.187133 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:19:51.187109    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:20:02.010032 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:20:02.009995    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/217a8762-14bb-4990-8a53-3b3bcd6984ac-metrics-certs\") pod \"network-metrics-daemon-kpscq\" (UID: \"217a8762-14bb-4990-8a53-3b3bcd6984ac\") " pod="openshift-multus/network-metrics-daemon-kpscq"
Sep 08 13:20:02.012193 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:20:02.012173    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/217a8762-14bb-4990-8a53-3b3bcd6984ac-metrics-certs\") pod \"network-metrics-daemon-kpscq\" (UID: \"217a8762-14bb-4990-8a53-3b3bcd6984ac\") " pod="openshift-multus/network-metrics-daemon-kpscq"
Sep 08 13:20:02.073046 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:20:02.073019    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-4rwvw\""
Sep 08 13:20:02.080496 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:20:02.080478    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kpscq"
Sep 08 13:20:02.349019 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:20:02.348944    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-kpscq"]
Sep 08 13:20:02.351861 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:20:02.351832    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod217a8762_14bb_4990_8a53_3b3bcd6984ac.slice/crio-4f5081f69c99a9c680061b7a108b42b953c03bc7029e2aea45893594105622a0 WatchSource:0}: Error finding container 4f5081f69c99a9c680061b7a108b42b953c03bc7029e2aea45893594105622a0: Status 404 returned error can't find the container with id 4f5081f69c99a9c680061b7a108b42b953c03bc7029e2aea45893594105622a0
Sep 08 13:20:02.958980 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:20:02.958945    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:20:02.974015 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:20:02.973991    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:20:03.035225 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:20:03.035197    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-kpscq" event={"ID":"217a8762-14bb-4990-8a53-3b3bcd6984ac","Type":"ContainerStarted","Data":"4f5081f69c99a9c680061b7a108b42b953c03bc7029e2aea45893594105622a0"}
Sep 08 13:20:03.049032 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:20:03.049012    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:20:05.045564 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:20:05.045524    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-kpscq" event={"ID":"217a8762-14bb-4990-8a53-3b3bcd6984ac","Type":"ContainerStarted","Data":"eccfa65f0ab1efe85886195df21953a81823007462cfa9ccc5e58f6643e345b0"}
Sep 08 13:20:05.045954 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:20:05.045580    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-kpscq" event={"ID":"217a8762-14bb-4990-8a53-3b3bcd6984ac","Type":"ContainerStarted","Data":"08879c3dbf879b7dceb5f170cabbfc206efd7c252ff9bbc9cda6dc0355d0a253"}
Sep 08 13:20:05.064601 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:20:05.064551    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-kpscq" podStartSLOduration=253.407125288 podStartE2EDuration="4m15.064535527s" totalImagesPullingTime="1.657410239s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:15:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:20:05.063435587 +0000 UTC m=+255.334883542" watchObservedRunningTime="2026-09-08 13:20:05.064535527 +0000 UTC m=+255.335983477"
Sep 08 13:20:05.172463 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:20:05.172438    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:20:05.252740 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:20:05.252715    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:20:12.567985 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:20:12.567954    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:20:12.667376 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:20:12.667351    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:20:13.059846 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:20:13.059797    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:20:13.127770 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:20:13.127743    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:20:13.191939 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:20:13.191911    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:20:13.276113 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:20:13.276092    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:20:13.799886 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:20:13.799858    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:20:17.098835 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:20:17.098801    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:20:23.088184 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:20:23.088144    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:20:23.190926 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:20:23.190897    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:20:25.174228 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:20:25.174196    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:20:25.880281 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:20:25.880251    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:20:28.964138 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:20:28.964058    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:20:29.061588 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:20:29.061557    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:20:38.098449 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:20:38.098419    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:20:49.009750 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:20:49.009720    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:20:50.113553 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:20:50.113522    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8h8cn_b936238e-bdf1-4dac-8bf8-6417c313597c/ovn-acl-logging/0.log"
Sep 08 13:20:50.117839 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:20:50.117814    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8h8cn_b936238e-bdf1-4dac-8bf8-6417c313597c/ovn-acl-logging/0.log"
Sep 08 13:20:50.119394 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:20:50.119369    2573 kubelet.go:1648] "Image garbage collection succeeded"
Sep 08 13:20:59.408370 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:20:59.408334    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:21:00.801940 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:21:00.801902    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:21:00.874490 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:21:00.874457    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:22:06.901488 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:22:06.901413    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:22:07.248691 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:22:07.248604    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:22:07.347677 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:22:07.347647    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:22:07.448945 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:22:07.448915    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:22:29.743597 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:22:29.743568    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:22:29.834663 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:22:29.834633    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:22:34.107086 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:22:34.107047    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:22:34.224832 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:22:34.224801    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:22:35.749723 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:22:35.749688    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:22:36.057283 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:22:36.057204    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:22:36.228524 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:22:36.228465    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:22:36.444448 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:22:36.444414    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:22:36.764962 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:22:36.764880    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:22:37.016421 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:22:37.016331    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:22:37.116795 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:22:37.116756    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:22:37.216523 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:22:37.216479    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:22:37.613492 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:22:37.613456    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:22:37.855007 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:22:37.854963    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:22:38.017456 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:22:38.017391    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:22:38.136010 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:22:38.135978    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:23:28.452704 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:23:28.452670    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:23:33.260207 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:23:33.260175    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:23:39.834309 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:23:39.834273    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:23:40.840366 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:23:40.840333    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:23:40.935537 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:23:40.935495    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:23:41.065562 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:23:41.065530    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:23:41.240807 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:23:41.240774    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:23:41.604971 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:23:41.604893    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:23:41.778850 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:23:41.778820    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:23:43.656148 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:23:43.656118    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:23:43.759892 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:23:43.759862    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:23:44.887369 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:23:44.887325    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:23:44.988857 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:23:44.988827    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:23:47.403813 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:23:47.403776    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:23:47.604838 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:23:47.604805    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:23:49.962570 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:23:49.962536    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:23:50.081713 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:23:50.081681    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:23:52.061619 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:23:52.061591    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:23:52.161721 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:23:52.161683    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:23:54.374378 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:23:54.374345    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:23:54.482403 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:23:54.482372    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:23:54.574189 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:23:54.574156    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:24:00.740377 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:24:00.740346    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:24:00.822291 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:24:00.822260    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:24:13.682450 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:24:13.682416    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:24:13.779948 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:24:13.779919    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:24:23.791849 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:24:23.791814    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:24:28.106592 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:24:28.106557    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:24:28.193731 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:24:28.193705    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-66c568d8-gsld6"]
Sep 08 13:24:28.199799 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:24:28.199771    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-66c568d8-gsld6"
Sep 08 13:24:28.207540 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:24:28.207519    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 08 13:24:28.207647 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:24:28.207545    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 08 13:24:28.209690 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:24:28.209668    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-hl9k8\""
Sep 08 13:24:28.209786 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:24:28.209712    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 08 13:24:28.213004 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:24:28.212986    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 08 13:24:28.213803 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:24:28.213790    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 08 13:24:28.218120 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:24:28.218100    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-66c568d8-gsld6"]
Sep 08 13:24:28.299687 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:24:28.299664    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/edb013d6-1adf-4545-b602-1d3617f9fc25-cert\") pod \"lws-controller-manager-66c568d8-gsld6\" (UID: \"edb013d6-1adf-4545-b602-1d3617f9fc25\") " pod="openshift-lws-operator/lws-controller-manager-66c568d8-gsld6"
Sep 08 13:24:28.299785 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:24:28.299690    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-st4wh\" (UniqueName: \"kubernetes.io/projected/edb013d6-1adf-4545-b602-1d3617f9fc25-kube-api-access-st4wh\") pod \"lws-controller-manager-66c568d8-gsld6\" (UID: \"edb013d6-1adf-4545-b602-1d3617f9fc25\") " pod="openshift-lws-operator/lws-controller-manager-66c568d8-gsld6"
Sep 08 13:24:28.299785 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:24:28.299712    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/edb013d6-1adf-4545-b602-1d3617f9fc25-manager-config\") pod \"lws-controller-manager-66c568d8-gsld6\" (UID: \"edb013d6-1adf-4545-b602-1d3617f9fc25\") " pod="openshift-lws-operator/lws-controller-manager-66c568d8-gsld6"
Sep 08 13:24:28.299785 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:24:28.299730    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/edb013d6-1adf-4545-b602-1d3617f9fc25-metrics-cert\") pod \"lws-controller-manager-66c568d8-gsld6\" (UID: \"edb013d6-1adf-4545-b602-1d3617f9fc25\") " pod="openshift-lws-operator/lws-controller-manager-66c568d8-gsld6"
Sep 08 13:24:28.337014 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:24:28.336989    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:24:28.400316 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:24:28.400296    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/edb013d6-1adf-4545-b602-1d3617f9fc25-cert\") pod \"lws-controller-manager-66c568d8-gsld6\" (UID: \"edb013d6-1adf-4545-b602-1d3617f9fc25\") " pod="openshift-lws-operator/lws-controller-manager-66c568d8-gsld6"
Sep 08 13:24:28.400455 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:24:28.400325    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-st4wh\" (UniqueName: \"kubernetes.io/projected/edb013d6-1adf-4545-b602-1d3617f9fc25-kube-api-access-st4wh\") pod \"lws-controller-manager-66c568d8-gsld6\" (UID: \"edb013d6-1adf-4545-b602-1d3617f9fc25\") " pod="openshift-lws-operator/lws-controller-manager-66c568d8-gsld6"
Sep 08 13:24:28.400455 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:24:28.400348    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/edb013d6-1adf-4545-b602-1d3617f9fc25-manager-config\") pod \"lws-controller-manager-66c568d8-gsld6\" (UID: \"edb013d6-1adf-4545-b602-1d3617f9fc25\") " pod="openshift-lws-operator/lws-controller-manager-66c568d8-gsld6"
Sep 08 13:24:28.400455 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:24:28.400369    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/edb013d6-1adf-4545-b602-1d3617f9fc25-metrics-cert\") pod \"lws-controller-manager-66c568d8-gsld6\" (UID: \"edb013d6-1adf-4545-b602-1d3617f9fc25\") " pod="openshift-lws-operator/lws-controller-manager-66c568d8-gsld6"
Sep 08 13:24:28.400979 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:24:28.400957    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/edb013d6-1adf-4545-b602-1d3617f9fc25-manager-config\") pod \"lws-controller-manager-66c568d8-gsld6\" (UID: \"edb013d6-1adf-4545-b602-1d3617f9fc25\") " pod="openshift-lws-operator/lws-controller-manager-66c568d8-gsld6"
Sep 08 13:24:28.402672 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:24:28.402654    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/edb013d6-1adf-4545-b602-1d3617f9fc25-cert\") pod \"lws-controller-manager-66c568d8-gsld6\" (UID: \"edb013d6-1adf-4545-b602-1d3617f9fc25\") " pod="openshift-lws-operator/lws-controller-manager-66c568d8-gsld6"
Sep 08 13:24:28.402830 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:24:28.402811    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/edb013d6-1adf-4545-b602-1d3617f9fc25-metrics-cert\") pod \"lws-controller-manager-66c568d8-gsld6\" (UID: \"edb013d6-1adf-4545-b602-1d3617f9fc25\") " pod="openshift-lws-operator/lws-controller-manager-66c568d8-gsld6"
Sep 08 13:24:28.463022 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:24:28.462997    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-st4wh\" (UniqueName: \"kubernetes.io/projected/edb013d6-1adf-4545-b602-1d3617f9fc25-kube-api-access-st4wh\") pod \"lws-controller-manager-66c568d8-gsld6\" (UID: \"edb013d6-1adf-4545-b602-1d3617f9fc25\") " pod="openshift-lws-operator/lws-controller-manager-66c568d8-gsld6"
Sep 08 13:24:28.508477 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:24:28.508454    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-66c568d8-gsld6"
Sep 08 13:24:28.789577 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:24:28.789402    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-66c568d8-gsld6"]
Sep 08 13:24:28.792117 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:24:28.792090    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedb013d6_1adf_4545_b602_1d3617f9fc25.slice/crio-ea1021399881a526efcf4f04716fe2ed46806c3a31f5348efe655ba28674d862 WatchSource:0}: Error finding container ea1021399881a526efcf4f04716fe2ed46806c3a31f5348efe655ba28674d862: Status 404 returned error can't find the container with id ea1021399881a526efcf4f04716fe2ed46806c3a31f5348efe655ba28674d862
Sep 08 13:24:28.793917 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:24:28.793903    2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 08 13:24:29.004514 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:24:29.004455    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-66c568d8-gsld6" event={"ID":"edb013d6-1adf-4545-b602-1d3617f9fc25","Type":"ContainerStarted","Data":"ea1021399881a526efcf4f04716fe2ed46806c3a31f5348efe655ba28674d862"}
Sep 08 13:24:30.465445 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:24:30.465415    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:24:30.639471 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:24:30.639435    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:24:30.805602 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:24:30.805527    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:24:32.017916 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:24:32.017882    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-66c568d8-gsld6" event={"ID":"edb013d6-1adf-4545-b602-1d3617f9fc25","Type":"ContainerStarted","Data":"fbd3fbebebcae18112e4def9283a026b8a2b0eb54f62c94fdfc793184fddf4d8"}
Sep 08 13:24:32.018433 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:24:32.018008    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-66c568d8-gsld6"
Sep 08 13:24:32.054694 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:24:32.054653    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-66c568d8-gsld6" podStartSLOduration=1.5075504130000001 podStartE2EDuration="4.054636495s" totalImagesPullingTime="2.547086082s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:24:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:24:32.05276867 +0000 UTC m=+522.324216612" watchObservedRunningTime="2026-09-08 13:24:32.054636495 +0000 UTC m=+522.326084427"
Sep 08 13:24:32.667582 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:24:32.667547    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:24:34.898950 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:24:34.898916    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:24:35.017275 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:24:35.017245    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:24:37.505549 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:24:37.505516    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:24:37.606348 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:24:37.606315    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:24:40.258678 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:24:40.258643    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:24:40.746527 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:24:40.746472    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:24:42.970913 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:24:42.970833    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:24:43.024303 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:24:43.024274    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-66c568d8-gsld6"
Sep 08 13:24:43.086540 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:24:43.086495    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:25:06.144785 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:25:06.144750    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:25:07.996742 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:25:07.996712    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-ccbdf686-ls9ww"]
Sep 08 13:25:07.999782 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:25:07.999767    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-ccbdf686-ls9ww"
Sep 08 13:25:08.002814 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:25:08.002790    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-lcbkh\""
Sep 08 13:25:08.002814 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:25:08.002803    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Sep 08 13:25:08.002980 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:25:08.002830    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 08 13:25:08.003458 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:25:08.003439    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 08 13:25:08.012406 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:25:08.012388    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-ccbdf686-ls9ww"]
Sep 08 13:25:08.078754 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:25:08.078729    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-br8lx\" (UniqueName: \"kubernetes.io/projected/d422a1d5-7838-4d16-919a-c6c81d36b614-kube-api-access-br8lx\") pod \"dns-operator-controller-manager-ccbdf686-ls9ww\" (UID: \"d422a1d5-7838-4d16-919a-c6c81d36b614\") " pod="kuadrant-system/dns-operator-controller-manager-ccbdf686-ls9ww"
Sep 08 13:25:08.179927 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:25:08.179901    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-br8lx\" (UniqueName: \"kubernetes.io/projected/d422a1d5-7838-4d16-919a-c6c81d36b614-kube-api-access-br8lx\") pod \"dns-operator-controller-manager-ccbdf686-ls9ww\" (UID: \"d422a1d5-7838-4d16-919a-c6c81d36b614\") " pod="kuadrant-system/dns-operator-controller-manager-ccbdf686-ls9ww"
Sep 08 13:25:08.197942 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:25:08.197894    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-br8lx\" (UniqueName: \"kubernetes.io/projected/d422a1d5-7838-4d16-919a-c6c81d36b614-kube-api-access-br8lx\") pod \"dns-operator-controller-manager-ccbdf686-ls9ww\" (UID: \"d422a1d5-7838-4d16-919a-c6c81d36b614\") " pod="kuadrant-system/dns-operator-controller-manager-ccbdf686-ls9ww"
Sep 08 13:25:08.309900 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:25:08.309844    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-ccbdf686-ls9ww"
Sep 08 13:25:08.605586 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:25:08.605514    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-ccbdf686-ls9ww"]
Sep 08 13:25:08.608833 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:25:08.608813    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd422a1d5_7838_4d16_919a_c6c81d36b614.slice/crio-c153ef1109ace4aaf67fe379aa274a2a7c44074bab005c5ddeede9ed6a0f581f WatchSource:0}: Error finding container c153ef1109ace4aaf67fe379aa274a2a7c44074bab005c5ddeede9ed6a0f581f: Status 404 returned error can't find the container with id c153ef1109ace4aaf67fe379aa274a2a7c44074bab005c5ddeede9ed6a0f581f
Sep 08 13:25:08.622066 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:25:08.622047    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:25:09.154959 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:25:09.154926    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-ccbdf686-ls9ww" event={"ID":"d422a1d5-7838-4d16-919a-c6c81d36b614","Type":"ContainerStarted","Data":"c153ef1109ace4aaf67fe379aa274a2a7c44074bab005c5ddeede9ed6a0f581f"}
Sep 08 13:25:11.099460 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:25:11.099427    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:25:11.202065 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:25:11.202040    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:25:12.061346 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:25:12.061318    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:25:12.167860 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:25:12.167827    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-ccbdf686-ls9ww" event={"ID":"d422a1d5-7838-4d16-919a-c6c81d36b614","Type":"ContainerStarted","Data":"e16fa2197c15cbfb4f1662d09cb64e2dc6d38dfdad6d6d7ea2f7f46d75e5c4bd"}
Sep 08 13:25:12.168288 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:25:12.167957    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-ccbdf686-ls9ww"
Sep 08 13:25:12.220703 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:25:12.220646    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-ccbdf686-ls9ww" podStartSLOduration=1.778340287 podStartE2EDuration="5.220629609s" totalImagesPullingTime="3.442289322s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:25:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:25:12.212604936 +0000 UTC m=+562.484052878" watchObservedRunningTime="2026-09-08 13:25:12.220629609 +0000 UTC m=+562.492077542"
Sep 08 13:25:12.221037 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:25:12.221015    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:25:13.050769 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:25:13.050740    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:25:13.345697 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:25:13.345605    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:25:13.520900 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:25:13.520872    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:25:13.781447 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:25:13.781414    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:25:15.971914 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:25:15.971884    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:25:17.409040 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:25:17.409011    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:25:17.591383 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:25:17.591346    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-774d94476f-mmlq6"]
Sep 08 13:25:17.594470 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:25:17.594455    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-774d94476f-mmlq6"
Sep 08 13:25:17.597678 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:25:17.597659    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-k26ml\""
Sep 08 13:25:17.605553 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:25:17.605530    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-774d94476f-mmlq6"]
Sep 08 13:25:17.760381 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:25:17.760318    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhkgn\" (UniqueName: \"kubernetes.io/projected/c099226c-7892-45ed-81c4-e3a353c6c340-kube-api-access-lhkgn\") pod \"limitador-operator-controller-manager-774d94476f-mmlq6\" (UID: \"c099226c-7892-45ed-81c4-e3a353c6c340\") " pod="kuadrant-system/limitador-operator-controller-manager-774d94476f-mmlq6"
Sep 08 13:25:17.861094 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:25:17.861065    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhkgn\" (UniqueName: \"kubernetes.io/projected/c099226c-7892-45ed-81c4-e3a353c6c340-kube-api-access-lhkgn\") pod \"limitador-operator-controller-manager-774d94476f-mmlq6\" (UID: \"c099226c-7892-45ed-81c4-e3a353c6c340\") " pod="kuadrant-system/limitador-operator-controller-manager-774d94476f-mmlq6"
Sep 08 13:25:17.877382 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:25:17.877355    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhkgn\" (UniqueName: \"kubernetes.io/projected/c099226c-7892-45ed-81c4-e3a353c6c340-kube-api-access-lhkgn\") pod \"limitador-operator-controller-manager-774d94476f-mmlq6\" (UID: \"c099226c-7892-45ed-81c4-e3a353c6c340\") " pod="kuadrant-system/limitador-operator-controller-manager-774d94476f-mmlq6"
Sep 08 13:25:17.904135 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:25:17.904113    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-774d94476f-mmlq6"
Sep 08 13:25:18.194763 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:25:18.194736    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-774d94476f-mmlq6"]
Sep 08 13:25:18.199224 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:25:18.199197    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc099226c_7892_45ed_81c4_e3a353c6c340.slice/crio-d0600bcf4c1bc4d7cf28b0cb31ad72b26b57ad59d5aac1be987d041b191a8aff WatchSource:0}: Error finding container d0600bcf4c1bc4d7cf28b0cb31ad72b26b57ad59d5aac1be987d041b191a8aff: Status 404 returned error can't find the container with id d0600bcf4c1bc4d7cf28b0cb31ad72b26b57ad59d5aac1be987d041b191a8aff
Sep 08 13:25:18.983180 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:25:18.983148    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:25:19.097996 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:25:19.097965    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:25:19.193370 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:25:19.193342    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-774d94476f-mmlq6" event={"ID":"c099226c-7892-45ed-81c4-e3a353c6c340","Type":"ContainerStarted","Data":"d0600bcf4c1bc4d7cf28b0cb31ad72b26b57ad59d5aac1be987d041b191a8aff"}
Sep 08 13:25:21.418636 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:25:21.418603    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:25:21.519640 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:25:21.519604    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:25:23.173840 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:25:23.173810    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-ccbdf686-ls9ww"
Sep 08 13:25:23.720872 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:25:23.720830    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:25:23.871980 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:25:23.871952    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:25:24.011816 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:25:24.011736    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:25:24.212377 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:25:24.212343    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-774d94476f-mmlq6" event={"ID":"c099226c-7892-45ed-81c4-e3a353c6c340","Type":"ContainerStarted","Data":"cac9a87715dab15a9d2562cecb5152d0e02f99ecb57374fc4fb02a36038a642d"}
Sep 08 13:25:24.212773 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:25:24.212481    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-774d94476f-mmlq6"
Sep 08 13:25:24.269266 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:25:24.269205    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:25:24.274998 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:25:24.274953    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-774d94476f-mmlq6" podStartSLOduration=2.223617788 podStartE2EDuration="7.274939261s" totalImagesPullingTime="5.051321473s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:25:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:25:24.269852609 +0000 UTC m=+574.541300540" watchObservedRunningTime="2026-09-08 13:25:24.274939261 +0000 UTC m=+574.546387192"
Sep 08 13:25:24.489051 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:25:24.489014    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:25:28.758919 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:25:28.758888    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:25:32.527916 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:25:32.527885    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:25:32.673880 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:25:32.673849    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:25:32.762409 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:25:32.762383    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:25:35.224123 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:25:35.224090    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-774d94476f-mmlq6"
Sep 08 13:25:35.441879 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:25:35.441825    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:25:35.549485 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:25:35.549412    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:25:38.038483 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:25:38.038443    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:25:38.162821 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:25:38.162791    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:25:43.440950 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:25:43.440919    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:25:43.505903 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:25:43.505875    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:25:43.836570 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:25:43.836467    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:25:43.938093 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:25:43.938058    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:25:44.238525 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:25:44.238474    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:25:44.306153 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:25:44.306127    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:25:50.141008 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:25:50.140981    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8h8cn_b936238e-bdf1-4dac-8bf8-6417c313597c/ovn-acl-logging/0.log"
Sep 08 13:25:50.142080 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:25:50.142053    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8h8cn_b936238e-bdf1-4dac-8bf8-6417c313597c/ovn-acl-logging/0.log"
Sep 08 13:25:52.252202 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:25:52.252133    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:25:52.352892 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:25:52.352860    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:25:55.244978 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:25:55.244944    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:25:55.391307 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:25:55.391270    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:25:58.169373 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:25:58.169342    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:26:05.909903 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:26:05.909868    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:26:06.060767 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:26:06.060719    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:26:06.163593 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:26:06.163493    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:26:06.185376 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:26:06.185349    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-r4wjf"]
Sep 08 13:26:06.188329 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:26:06.188313    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-85d9db4d5d-r4wjf"
Sep 08 13:26:06.190379 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:26:06.190358    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 08 13:26:06.190478 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:26:06.190449    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-nlm5f\""
Sep 08 13:26:06.197855 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:26:06.197835    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-r4wjf"]
Sep 08 13:26:06.229641 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:26:06.229616    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-r4wjf"]
Sep 08 13:26:06.304292 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:26:06.304260    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvk55\" (UniqueName: \"kubernetes.io/projected/05b740c9-d688-487a-8d60-cc248a226a1c-kube-api-access-bvk55\") pod \"limitador-limitador-85d9db4d5d-r4wjf\" (UID: \"05b740c9-d688-487a-8d60-cc248a226a1c\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-r4wjf"
Sep 08 13:26:06.304813 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:26:06.304781    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/05b740c9-d688-487a-8d60-cc248a226a1c-config-file\") pod \"limitador-limitador-85d9db4d5d-r4wjf\" (UID: \"05b740c9-d688-487a-8d60-cc248a226a1c\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-r4wjf"
Sep 08 13:26:06.405633 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:26:06.405602    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvk55\" (UniqueName: \"kubernetes.io/projected/05b740c9-d688-487a-8d60-cc248a226a1c-kube-api-access-bvk55\") pod \"limitador-limitador-85d9db4d5d-r4wjf\" (UID: \"05b740c9-d688-487a-8d60-cc248a226a1c\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-r4wjf"
Sep 08 13:26:06.405753 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:26:06.405651    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/05b740c9-d688-487a-8d60-cc248a226a1c-config-file\") pod \"limitador-limitador-85d9db4d5d-r4wjf\" (UID: \"05b740c9-d688-487a-8d60-cc248a226a1c\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-r4wjf"
Sep 08 13:26:06.406190 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:26:06.406173    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/05b740c9-d688-487a-8d60-cc248a226a1c-config-file\") pod \"limitador-limitador-85d9db4d5d-r4wjf\" (UID: \"05b740c9-d688-487a-8d60-cc248a226a1c\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-r4wjf"
Sep 08 13:26:06.414013 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:26:06.413969    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvk55\" (UniqueName: \"kubernetes.io/projected/05b740c9-d688-487a-8d60-cc248a226a1c-kube-api-access-bvk55\") pod \"limitador-limitador-85d9db4d5d-r4wjf\" (UID: \"05b740c9-d688-487a-8d60-cc248a226a1c\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-r4wjf"
Sep 08 13:26:06.499062 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:26:06.499029    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-85d9db4d5d-r4wjf"
Sep 08 13:26:06.738833 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:26:06.738761    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:26:06.767636 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:26:06.767607    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-r4wjf"]
Sep 08 13:26:06.769970 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:26:06.769941    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05b740c9_d688_487a_8d60_cc248a226a1c.slice/crio-3557ab98f6c6e04f0fda2365fcc9ce2dfc6314f407833ab1a11b2aa4dd3863fb WatchSource:0}: Error finding container 3557ab98f6c6e04f0fda2365fcc9ce2dfc6314f407833ab1a11b2aa4dd3863fb: Status 404 returned error can't find the container with id 3557ab98f6c6e04f0fda2365fcc9ce2dfc6314f407833ab1a11b2aa4dd3863fb
Sep 08 13:26:06.824634 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:26:06.824612    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:26:07.377438 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:26:07.377403    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-85d9db4d5d-r4wjf" event={"ID":"05b740c9-d688-487a-8d60-cc248a226a1c","Type":"ContainerStarted","Data":"3557ab98f6c6e04f0fda2365fcc9ce2dfc6314f407833ab1a11b2aa4dd3863fb"}
Sep 08 13:26:12.397894 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:26:12.397858    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-85d9db4d5d-r4wjf" event={"ID":"05b740c9-d688-487a-8d60-cc248a226a1c","Type":"ContainerStarted","Data":"f425da8bbc1d3b0eb38084351e28eea70e3516dca81ff720cc858b6e28f76622"}
Sep 08 13:26:12.398307 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:26:12.397984    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-85d9db4d5d-r4wjf"
Sep 08 13:26:12.413841 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:26:12.413793    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-85d9db4d5d-r4wjf" podStartSLOduration=1.6400699699999999 podStartE2EDuration="6.413779227s" totalImagesPullingTime="4.773709257s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:26:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:26:12.413442784 +0000 UTC m=+622.684890737" watchObservedRunningTime="2026-09-08 13:26:12.413779227 +0000 UTC m=+622.685227166"
Sep 08 13:26:19.305052 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:26:19.305019    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:26:19.464414 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:26:19.464379    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:26:23.402818 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:26:23.402789    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-85d9db4d5d-r4wjf"
Sep 08 13:26:29.459949 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:26:29.459916    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:26:29.560579 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:26:29.560550    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:26:37.422602 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:26:37.422565    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:26:37.524471 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:26:37.524442    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:27:03.107962 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:27:03.107920    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:27:03.194822 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:27:03.194786    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:27:03.292155 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:27:03.292126    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:27:03.389981 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:27:03.389953    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:27:03.731091 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:27:03.731002    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:27:03.816867 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:27:03.816830    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:28:21.547164 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:28:21.547108    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:28:21.686402 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:28:21.686367    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:28:21.802610 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:28:21.802541    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:28:21.903147 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:28:21.903115    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:28:23.052650 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:28:23.052617    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:28:23.144415 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:28:23.144382    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:29:06.659447 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:29:06.659409    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:29:08.090906 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:29:08.090872    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:29:14.854262 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:29:14.854179    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:29:57.135901 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:29:57.135869    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:29:57.233039 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:29:57.233010    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:30:09.631657 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:30:09.631620    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:30:09.725646 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:30:09.725620    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:30:17.606357 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:30:17.606323    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:30:24.909346 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:30:24.909312    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-6gnmx"]
Sep 08 13:30:24.912412 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:30:24.912394    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-6gnmx"
Sep 08 13:30:24.915814 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:30:24.915789    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"default-dockercfg-kq4l4\""
Sep 08 13:30:24.915814 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:30:24.915808    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"kube-root-ca.crt\""
Sep 08 13:30:24.915992 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:30:24.915791    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"fc-smoke-test-kserve-self-signed-certs\""
Sep 08 13:30:24.915992 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:30:24.915955    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"openshift-service-ca.crt\""
Sep 08 13:30:24.927520 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:30:24.927483    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-6gnmx"]
Sep 08 13:30:24.991332 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:30:24.991305    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1410313b-9079-4416-babf-eb62a87b7e7f-model-cache\") pod \"fc-smoke-test-kserve-7f4687897-6gnmx\" (UID: \"1410313b-9079-4416-babf-eb62a87b7e7f\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-6gnmx"
Sep 08 13:30:24.991461 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:30:24.991334    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1410313b-9079-4416-babf-eb62a87b7e7f-tmp-dir\") pod \"fc-smoke-test-kserve-7f4687897-6gnmx\" (UID: \"1410313b-9079-4416-babf-eb62a87b7e7f\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-6gnmx"
Sep 08 13:30:24.991461 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:30:24.991434    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1410313b-9079-4416-babf-eb62a87b7e7f-home\") pod \"fc-smoke-test-kserve-7f4687897-6gnmx\" (UID: \"1410313b-9079-4416-babf-eb62a87b7e7f\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-6gnmx"
Sep 08 13:30:24.991599 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:30:24.991463    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1410313b-9079-4416-babf-eb62a87b7e7f-dshm\") pod \"fc-smoke-test-kserve-7f4687897-6gnmx\" (UID: \"1410313b-9079-4416-babf-eb62a87b7e7f\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-6gnmx"
Sep 08 13:30:24.991599 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:30:24.991493    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1410313b-9079-4416-babf-eb62a87b7e7f-tls-certs\") pod \"fc-smoke-test-kserve-7f4687897-6gnmx\" (UID: \"1410313b-9079-4416-babf-eb62a87b7e7f\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-6gnmx"
Sep 08 13:30:24.991599 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:30:24.991541    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrgfv\" (UniqueName: \"kubernetes.io/projected/1410313b-9079-4416-babf-eb62a87b7e7f-kube-api-access-wrgfv\") pod \"fc-smoke-test-kserve-7f4687897-6gnmx\" (UID: \"1410313b-9079-4416-babf-eb62a87b7e7f\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-6gnmx"
Sep 08 13:30:25.003291 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:30:25.003259    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:30:25.091891 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:30:25.091864    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1410313b-9079-4416-babf-eb62a87b7e7f-home\") pod \"fc-smoke-test-kserve-7f4687897-6gnmx\" (UID: \"1410313b-9079-4416-babf-eb62a87b7e7f\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-6gnmx"
Sep 08 13:30:25.092018 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:30:25.091896    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1410313b-9079-4416-babf-eb62a87b7e7f-dshm\") pod \"fc-smoke-test-kserve-7f4687897-6gnmx\" (UID: \"1410313b-9079-4416-babf-eb62a87b7e7f\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-6gnmx"
Sep 08 13:30:25.092018 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:30:25.091917    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1410313b-9079-4416-babf-eb62a87b7e7f-tls-certs\") pod \"fc-smoke-test-kserve-7f4687897-6gnmx\" (UID: \"1410313b-9079-4416-babf-eb62a87b7e7f\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-6gnmx"
Sep 08 13:30:25.092018 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:30:25.091948    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrgfv\" (UniqueName: \"kubernetes.io/projected/1410313b-9079-4416-babf-eb62a87b7e7f-kube-api-access-wrgfv\") pod \"fc-smoke-test-kserve-7f4687897-6gnmx\" (UID: \"1410313b-9079-4416-babf-eb62a87b7e7f\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-6gnmx"
Sep 08 13:30:25.092184 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:30:25.092019    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1410313b-9079-4416-babf-eb62a87b7e7f-model-cache\") pod \"fc-smoke-test-kserve-7f4687897-6gnmx\" (UID: \"1410313b-9079-4416-babf-eb62a87b7e7f\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-6gnmx"
Sep 08 13:30:25.092184 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:30:25.092042    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1410313b-9079-4416-babf-eb62a87b7e7f-tmp-dir\") pod \"fc-smoke-test-kserve-7f4687897-6gnmx\" (UID: \"1410313b-9079-4416-babf-eb62a87b7e7f\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-6gnmx"
Sep 08 13:30:25.092338 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:30:25.092317    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1410313b-9079-4416-babf-eb62a87b7e7f-home\") pod \"fc-smoke-test-kserve-7f4687897-6gnmx\" (UID: \"1410313b-9079-4416-babf-eb62a87b7e7f\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-6gnmx"
Sep 08 13:30:25.092403 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:30:25.092380    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1410313b-9079-4416-babf-eb62a87b7e7f-model-cache\") pod \"fc-smoke-test-kserve-7f4687897-6gnmx\" (UID: \"1410313b-9079-4416-babf-eb62a87b7e7f\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-6gnmx"
Sep 08 13:30:25.092440 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:30:25.092414    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1410313b-9079-4416-babf-eb62a87b7e7f-tmp-dir\") pod \"fc-smoke-test-kserve-7f4687897-6gnmx\" (UID: \"1410313b-9079-4416-babf-eb62a87b7e7f\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-6gnmx"
Sep 08 13:30:25.094080 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:30:25.094060    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1410313b-9079-4416-babf-eb62a87b7e7f-dshm\") pod \"fc-smoke-test-kserve-7f4687897-6gnmx\" (UID: \"1410313b-9079-4416-babf-eb62a87b7e7f\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-6gnmx"
Sep 08 13:30:25.094383 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:30:25.094369    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1410313b-9079-4416-babf-eb62a87b7e7f-tls-certs\") pod \"fc-smoke-test-kserve-7f4687897-6gnmx\" (UID: \"1410313b-9079-4416-babf-eb62a87b7e7f\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-6gnmx"
Sep 08 13:30:25.101179 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:30:25.101159    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrgfv\" (UniqueName: \"kubernetes.io/projected/1410313b-9079-4416-babf-eb62a87b7e7f-kube-api-access-wrgfv\") pod \"fc-smoke-test-kserve-7f4687897-6gnmx\" (UID: \"1410313b-9079-4416-babf-eb62a87b7e7f\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-6gnmx"
Sep 08 13:30:25.222588 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:30:25.222518    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-6gnmx"
Sep 08 13:30:25.329073 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:30:25.329036    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:30:25.492376 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:30:25.492351    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-6gnmx"]
Sep 08 13:30:25.494437 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:30:25.494414    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1410313b_9079_4416_babf_eb62a87b7e7f.slice/crio-5e8fb7ac6f699d915060010f928d61082fbf5dbc68abb2f268ae236f037f47d0 WatchSource:0}: Error finding container 5e8fb7ac6f699d915060010f928d61082fbf5dbc68abb2f268ae236f037f47d0: Status 404 returned error can't find the container with id 5e8fb7ac6f699d915060010f928d61082fbf5dbc68abb2f268ae236f037f47d0
Sep 08 13:30:25.496369 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:30:25.496347    2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 08 13:30:26.370645 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:30:26.370607    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-6gnmx" event={"ID":"1410313b-9079-4416-babf-eb62a87b7e7f","Type":"ContainerStarted","Data":"5e8fb7ac6f699d915060010f928d61082fbf5dbc68abb2f268ae236f037f47d0"}
Sep 08 13:30:27.380807 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:30:27.380772    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-6gnmx" event={"ID":"1410313b-9079-4416-babf-eb62a87b7e7f","Type":"ContainerStarted","Data":"8e2cb019be34feaf900e56859ed35b2be677cfb640c3ab52b918a52381d506c9"}
Sep 08 13:30:27.400431 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:30:27.400386    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-6gnmx" podStartSLOduration=1.8519118730000002 podStartE2EDuration="3.400372695s" totalImagesPullingTime="1.548460822s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:30:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:30:27.399386632 +0000 UTC m=+877.670834564" watchObservedRunningTime="2026-09-08 13:30:27.400372695 +0000 UTC m=+877.671820625"
Sep 08 13:30:28.223369 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:30:28.223342    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-6gnmx"
Sep 08 13:30:35.222762 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:30:35.222726    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-6gnmx"
Sep 08 13:30:35.235107 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:30:35.235083    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-6gnmx"
Sep 08 13:30:35.425212 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:30:35.425185    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-6gnmx"
Sep 08 13:30:45.480310 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:30:45.480219    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:30:50.166951 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:30:50.166922    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8h8cn_b936238e-bdf1-4dac-8bf8-6417c313597c/ovn-acl-logging/0.log"
Sep 08 13:30:50.167403 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:30:50.167110    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8h8cn_b936238e-bdf1-4dac-8bf8-6417c313597c/ovn-acl-logging/0.log"
Sep 08 13:30:53.808888 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:30:53.808854    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:30:53.872013 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:30:53.871984    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:31:01.426249 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:31:01.426212    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:31:01.486554 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:31:01.486523    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:31:04.794048 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:31:04.794016    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:31:09.643548 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:31:09.643495    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:31:09.651411 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:31:09.651385    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-6gnmx"]
Sep 08 13:31:09.659967 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:31:09.659923    2573 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 08 13:31:09.659967 ip-10-0-132-254 kubenswrapper[2573]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 08 13:31:09.659967 ip-10-0-132-254 kubenswrapper[2573]:         , stdout: , stderr: , exit code -1
Sep 08 13:31:09.659967 ip-10-0-132-254 kubenswrapper[2573]:  > containerID="8e2cb019be34feaf900e56859ed35b2be677cfb640c3ab52b918a52381d506c9" cmd=["/bin/sleep","15"]
Sep 08 13:31:09.660206 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:31:09.659988    2573 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-6gnmx" message="" err=<
Sep 08 13:31:09.660206 ip-10-0-132-254 kubenswrapper[2573]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 08 13:31:09.660206 ip-10-0-132-254 kubenswrapper[2573]:         , stdout: , stderr: , exit code -1
Sep 08 13:31:09.660206 ip-10-0-132-254 kubenswrapper[2573]:  >
Sep 08 13:31:09.660206 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:31:09.660005    2573 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 08 13:31:09.660206 ip-10-0-132-254 kubenswrapper[2573]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 08 13:31:09.660206 ip-10-0-132-254 kubenswrapper[2573]:         , stdout: , stderr: , exit code -1
Sep 08 13:31:09.660206 ip-10-0-132-254 kubenswrapper[2573]:  > pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-6gnmx" podUID="1410313b-9079-4416-babf-eb62a87b7e7f" containerName="main" containerID="cri-o://8e2cb019be34feaf900e56859ed35b2be677cfb640c3ab52b918a52381d506c9"
Sep 08 13:31:09.660206 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:31:09.660050    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-6gnmx" podUID="1410313b-9079-4416-babf-eb62a87b7e7f" containerName="main" containerID="cri-o://8e2cb019be34feaf900e56859ed35b2be677cfb640c3ab52b918a52381d506c9" gracePeriod=60
Sep 08 13:31:10.004001 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:31:10.003978    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-6gnmx"
Sep 08 13:31:10.133543 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:31:10.133517    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1410313b-9079-4416-babf-eb62a87b7e7f-tmp-dir\") pod \"1410313b-9079-4416-babf-eb62a87b7e7f\" (UID: \"1410313b-9079-4416-babf-eb62a87b7e7f\") "
Sep 08 13:31:10.133686 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:31:10.133557    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wrgfv\" (UniqueName: \"kubernetes.io/projected/1410313b-9079-4416-babf-eb62a87b7e7f-kube-api-access-wrgfv\") pod \"1410313b-9079-4416-babf-eb62a87b7e7f\" (UID: \"1410313b-9079-4416-babf-eb62a87b7e7f\") "
Sep 08 13:31:10.133686 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:31:10.133580    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1410313b-9079-4416-babf-eb62a87b7e7f-dshm\") pod \"1410313b-9079-4416-babf-eb62a87b7e7f\" (UID: \"1410313b-9079-4416-babf-eb62a87b7e7f\") "
Sep 08 13:31:10.133686 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:31:10.133607    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1410313b-9079-4416-babf-eb62a87b7e7f-home\") pod \"1410313b-9079-4416-babf-eb62a87b7e7f\" (UID: \"1410313b-9079-4416-babf-eb62a87b7e7f\") "
Sep 08 13:31:10.133686 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:31:10.133630    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1410313b-9079-4416-babf-eb62a87b7e7f-model-cache\") pod \"1410313b-9079-4416-babf-eb62a87b7e7f\" (UID: \"1410313b-9079-4416-babf-eb62a87b7e7f\") "
Sep 08 13:31:10.133686 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:31:10.133654    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1410313b-9079-4416-babf-eb62a87b7e7f-tls-certs\") pod \"1410313b-9079-4416-babf-eb62a87b7e7f\" (UID: \"1410313b-9079-4416-babf-eb62a87b7e7f\") "
Sep 08 13:31:10.133943 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:31:10.133761    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1410313b-9079-4416-babf-eb62a87b7e7f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1410313b-9079-4416-babf-eb62a87b7e7f" (UID: "1410313b-9079-4416-babf-eb62a87b7e7f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:31:10.133943 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:31:10.133887    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1410313b-9079-4416-babf-eb62a87b7e7f-model-cache" (OuterVolumeSpecName: "model-cache") pod "1410313b-9079-4416-babf-eb62a87b7e7f" (UID: "1410313b-9079-4416-babf-eb62a87b7e7f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:31:10.133943 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:31:10.133899    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1410313b-9079-4416-babf-eb62a87b7e7f-home" (OuterVolumeSpecName: "home") pod "1410313b-9079-4416-babf-eb62a87b7e7f" (UID: "1410313b-9079-4416-babf-eb62a87b7e7f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:31:10.135809 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:31:10.135785    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1410313b-9079-4416-babf-eb62a87b7e7f-dshm" (OuterVolumeSpecName: "dshm") pod "1410313b-9079-4416-babf-eb62a87b7e7f" (UID: "1410313b-9079-4416-babf-eb62a87b7e7f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:31:10.135894 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:31:10.135864    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1410313b-9079-4416-babf-eb62a87b7e7f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1410313b-9079-4416-babf-eb62a87b7e7f" (UID: "1410313b-9079-4416-babf-eb62a87b7e7f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:31:10.135959 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:31:10.135924    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1410313b-9079-4416-babf-eb62a87b7e7f-kube-api-access-wrgfv" (OuterVolumeSpecName: "kube-api-access-wrgfv") pod "1410313b-9079-4416-babf-eb62a87b7e7f" (UID: "1410313b-9079-4416-babf-eb62a87b7e7f"). InnerVolumeSpecName "kube-api-access-wrgfv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:31:10.234464 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:31:10.234438    2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1410313b-9079-4416-babf-eb62a87b7e7f-home\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 13:31:10.234464 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:31:10.234459    2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1410313b-9079-4416-babf-eb62a87b7e7f-model-cache\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 13:31:10.234626 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:31:10.234485    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1410313b-9079-4416-babf-eb62a87b7e7f-tls-certs\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 13:31:10.234626 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:31:10.234493    2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1410313b-9079-4416-babf-eb62a87b7e7f-tmp-dir\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 13:31:10.234626 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:31:10.234519    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrgfv\" (UniqueName: \"kubernetes.io/projected/1410313b-9079-4416-babf-eb62a87b7e7f-kube-api-access-wrgfv\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 13:31:10.234626 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:31:10.234528    2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1410313b-9079-4416-babf-eb62a87b7e7f-dshm\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 13:31:10.548260 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:31:10.548183    2573 generic.go:356] "Generic (PLEG): container finished" podID="1410313b-9079-4416-babf-eb62a87b7e7f" containerID="8e2cb019be34feaf900e56859ed35b2be677cfb640c3ab52b918a52381d506c9" exitCode=0
Sep 08 13:31:10.548260 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:31:10.548217    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-6gnmx" event={"ID":"1410313b-9079-4416-babf-eb62a87b7e7f","Type":"ContainerDied","Data":"8e2cb019be34feaf900e56859ed35b2be677cfb640c3ab52b918a52381d506c9"}
Sep 08 13:31:10.548260 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:31:10.548239    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-6gnmx"
Sep 08 13:31:10.548477 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:31:10.548280    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-6gnmx" event={"ID":"1410313b-9079-4416-babf-eb62a87b7e7f","Type":"ContainerDied","Data":"5e8fb7ac6f699d915060010f928d61082fbf5dbc68abb2f268ae236f037f47d0"}
Sep 08 13:31:10.548477 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:31:10.548302    2573 scope.go:117] "RemoveContainer" containerID="8e2cb019be34feaf900e56859ed35b2be677cfb640c3ab52b918a52381d506c9"
Sep 08 13:31:10.556309 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:31:10.556294    2573 scope.go:117] "RemoveContainer" containerID="8e2cb019be34feaf900e56859ed35b2be677cfb640c3ab52b918a52381d506c9"
Sep 08 13:31:10.556561 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:31:10.556540    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e2cb019be34feaf900e56859ed35b2be677cfb640c3ab52b918a52381d506c9\": container with ID starting with 8e2cb019be34feaf900e56859ed35b2be677cfb640c3ab52b918a52381d506c9 not found: ID does not exist" containerID="8e2cb019be34feaf900e56859ed35b2be677cfb640c3ab52b918a52381d506c9"
Sep 08 13:31:10.556622 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:31:10.556568    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e2cb019be34feaf900e56859ed35b2be677cfb640c3ab52b918a52381d506c9"} err="failed to get container status \"8e2cb019be34feaf900e56859ed35b2be677cfb640c3ab52b918a52381d506c9\": rpc error: code = NotFound desc = could not find container \"8e2cb019be34feaf900e56859ed35b2be677cfb640c3ab52b918a52381d506c9\": container with ID starting with 8e2cb019be34feaf900e56859ed35b2be677cfb640c3ab52b918a52381d506c9 not found: ID does not exist"
Sep 08 13:31:10.604999 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:31:10.604970    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-6gnmx"]
Sep 08 13:31:10.616982 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:31:10.616960    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-6gnmx"]
Sep 08 13:31:12.174919 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:31:12.174887    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1410313b-9079-4416-babf-eb62a87b7e7f" path="/var/lib/kubelet/pods/1410313b-9079-4416-babf-eb62a87b7e7f/volumes"
Sep 08 13:31:12.423123 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:31:12.423093    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:31:51.125559 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:31:51.125524    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:31:59.761336 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:31:59.761301    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:31:59.849940 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:31:59.849905    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:32:06.168669 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:32:06.168631    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:32:11.785791 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:32:11.785753    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:32:11.852735 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:32:11.852700    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:32:20.307826 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:32:20.307739    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-ll589"]
Sep 08 13:32:20.308286 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:32:20.308074    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1410313b-9079-4416-babf-eb62a87b7e7f" containerName="main"
Sep 08 13:32:20.308286 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:32:20.308087    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="1410313b-9079-4416-babf-eb62a87b7e7f" containerName="main"
Sep 08 13:32:20.308286 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:32:20.308147    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="1410313b-9079-4416-babf-eb62a87b7e7f" containerName="main"
Sep 08 13:32:20.310879 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:32:20.310860    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-ll589"
Sep 08 13:32:20.313931 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:32:20.313911    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"openshift-service-ca.crt\""
Sep 08 13:32:20.313931 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:32:20.313923    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"kube-root-ca.crt\""
Sep 08 13:32:20.314127 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:32:20.313995    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"fc-concurrency-test-kserve-self-signed-certs\""
Sep 08 13:32:20.314127 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:32:20.314081    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"default-dockercfg-dst2n\""
Sep 08 13:32:20.321009 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:32:20.320990    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-ll589"]
Sep 08 13:32:20.394464 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:32:20.394433    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:32:20.468224 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:32:20.468189    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dmtj\" (UniqueName: \"kubernetes.io/projected/7b0fd42f-9d91-41d7-9123-e79b341609ce-kube-api-access-9dmtj\") pod \"fc-concurrency-test-kserve-b5d44467c-ll589\" (UID: \"7b0fd42f-9d91-41d7-9123-e79b341609ce\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-ll589"
Sep 08 13:32:20.468364 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:32:20.468236    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b0fd42f-9d91-41d7-9123-e79b341609ce-tls-certs\") pod \"fc-concurrency-test-kserve-b5d44467c-ll589\" (UID: \"7b0fd42f-9d91-41d7-9123-e79b341609ce\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-ll589"
Sep 08 13:32:20.468364 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:32:20.468258    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b0fd42f-9d91-41d7-9123-e79b341609ce-tmp-dir\") pod \"fc-concurrency-test-kserve-b5d44467c-ll589\" (UID: \"7b0fd42f-9d91-41d7-9123-e79b341609ce\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-ll589"
Sep 08 13:32:20.468364 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:32:20.468333    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b0fd42f-9d91-41d7-9123-e79b341609ce-model-cache\") pod \"fc-concurrency-test-kserve-b5d44467c-ll589\" (UID: \"7b0fd42f-9d91-41d7-9123-e79b341609ce\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-ll589"
Sep 08 13:32:20.468468 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:32:20.468366    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b0fd42f-9d91-41d7-9123-e79b341609ce-home\") pod \"fc-concurrency-test-kserve-b5d44467c-ll589\" (UID: \"7b0fd42f-9d91-41d7-9123-e79b341609ce\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-ll589"
Sep 08 13:32:20.468468 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:32:20.468402    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b0fd42f-9d91-41d7-9123-e79b341609ce-dshm\") pod \"fc-concurrency-test-kserve-b5d44467c-ll589\" (UID: \"7b0fd42f-9d91-41d7-9123-e79b341609ce\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-ll589"
Sep 08 13:32:20.569746 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:32:20.569672    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b0fd42f-9d91-41d7-9123-e79b341609ce-dshm\") pod \"fc-concurrency-test-kserve-b5d44467c-ll589\" (UID: \"7b0fd42f-9d91-41d7-9123-e79b341609ce\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-ll589"
Sep 08 13:32:20.569746 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:32:20.569730    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dmtj\" (UniqueName: \"kubernetes.io/projected/7b0fd42f-9d91-41d7-9123-e79b341609ce-kube-api-access-9dmtj\") pod \"fc-concurrency-test-kserve-b5d44467c-ll589\" (UID: \"7b0fd42f-9d91-41d7-9123-e79b341609ce\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-ll589"
Sep 08 13:32:20.569932 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:32:20.569779    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b0fd42f-9d91-41d7-9123-e79b341609ce-tls-certs\") pod \"fc-concurrency-test-kserve-b5d44467c-ll589\" (UID: \"7b0fd42f-9d91-41d7-9123-e79b341609ce\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-ll589"
Sep 08 13:32:20.569932 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:32:20.569812    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b0fd42f-9d91-41d7-9123-e79b341609ce-tmp-dir\") pod \"fc-concurrency-test-kserve-b5d44467c-ll589\" (UID: \"7b0fd42f-9d91-41d7-9123-e79b341609ce\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-ll589"
Sep 08 13:32:20.569932 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:32:20.569868    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b0fd42f-9d91-41d7-9123-e79b341609ce-model-cache\") pod \"fc-concurrency-test-kserve-b5d44467c-ll589\" (UID: \"7b0fd42f-9d91-41d7-9123-e79b341609ce\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-ll589"
Sep 08 13:32:20.569932 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:32:20.569908    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b0fd42f-9d91-41d7-9123-e79b341609ce-home\") pod \"fc-concurrency-test-kserve-b5d44467c-ll589\" (UID: \"7b0fd42f-9d91-41d7-9123-e79b341609ce\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-ll589"
Sep 08 13:32:20.570378 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:32:20.570350    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b0fd42f-9d91-41d7-9123-e79b341609ce-model-cache\") pod \"fc-concurrency-test-kserve-b5d44467c-ll589\" (UID: \"7b0fd42f-9d91-41d7-9123-e79b341609ce\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-ll589"
Sep 08 13:32:20.570571 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:32:20.570549    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b0fd42f-9d91-41d7-9123-e79b341609ce-tmp-dir\") pod \"fc-concurrency-test-kserve-b5d44467c-ll589\" (UID: \"7b0fd42f-9d91-41d7-9123-e79b341609ce\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-ll589"
Sep 08 13:32:20.570696 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:32:20.570606    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b0fd42f-9d91-41d7-9123-e79b341609ce-home\") pod \"fc-concurrency-test-kserve-b5d44467c-ll589\" (UID: \"7b0fd42f-9d91-41d7-9123-e79b341609ce\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-ll589"
Sep 08 13:32:20.572437 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:32:20.572415    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b0fd42f-9d91-41d7-9123-e79b341609ce-dshm\") pod \"fc-concurrency-test-kserve-b5d44467c-ll589\" (UID: \"7b0fd42f-9d91-41d7-9123-e79b341609ce\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-ll589"
Sep 08 13:32:20.572786 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:32:20.572769    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b0fd42f-9d91-41d7-9123-e79b341609ce-tls-certs\") pod \"fc-concurrency-test-kserve-b5d44467c-ll589\" (UID: \"7b0fd42f-9d91-41d7-9123-e79b341609ce\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-ll589"
Sep 08 13:32:20.578965 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:32:20.578940    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dmtj\" (UniqueName: \"kubernetes.io/projected/7b0fd42f-9d91-41d7-9123-e79b341609ce-kube-api-access-9dmtj\") pod \"fc-concurrency-test-kserve-b5d44467c-ll589\" (UID: \"7b0fd42f-9d91-41d7-9123-e79b341609ce\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-ll589"
Sep 08 13:32:20.615789 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:32:20.615760    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:32:20.623790 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:32:20.623755    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-ll589"
Sep 08 13:32:20.906739 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:32:20.906715    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-ll589"]
Sep 08 13:32:20.909257 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:32:20.909234    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b0fd42f_9d91_41d7_9123_e79b341609ce.slice/crio-5853866d2c30ca9c1b8a558cc6420ea714e097b8ccd927d7d6a73afdf1807176 WatchSource:0}: Error finding container 5853866d2c30ca9c1b8a558cc6420ea714e097b8ccd927d7d6a73afdf1807176: Status 404 returned error can't find the container with id 5853866d2c30ca9c1b8a558cc6420ea714e097b8ccd927d7d6a73afdf1807176
Sep 08 13:32:21.817064 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:32:21.817029    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-ll589" event={"ID":"7b0fd42f-9d91-41d7-9123-e79b341609ce","Type":"ContainerStarted","Data":"121c414f6733fd90c07fef254d50dc5e513c01963efd0182b498705993bb4b69"}
Sep 08 13:32:21.817489 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:32:21.817084    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-ll589" event={"ID":"7b0fd42f-9d91-41d7-9123-e79b341609ce","Type":"ContainerStarted","Data":"5853866d2c30ca9c1b8a558cc6420ea714e097b8ccd927d7d6a73afdf1807176"}
Sep 08 13:32:21.850012 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:32:21.849963    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-ll589" podStartSLOduration=1.849947292 podStartE2EDuration="1.849947292s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:32:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:32:21.848452602 +0000 UTC m=+992.119900553" watchObservedRunningTime="2026-09-08 13:32:21.849947292 +0000 UTC m=+992.121395223"
Sep 08 13:32:22.624624 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:32:22.624582    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-ll589"
Sep 08 13:32:30.624202 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:32:30.624168    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-ll589"
Sep 08 13:32:30.636702 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:32:30.636680    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-ll589"
Sep 08 13:32:30.861188 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:32:30.861115    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-ll589"
Sep 08 13:32:32.634601 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:32:32.634570    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:32:52.474854 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:32:52.474821    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:33:00.042853 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:33:00.042821    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-ll589"]
Sep 08 13:33:00.043448 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:33:00.042918    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:33:00.051781 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:33:00.051738    2573 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 08 13:33:00.051781 ip-10-0-132-254 kubenswrapper[2573]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 08 13:33:00.051781 ip-10-0-132-254 kubenswrapper[2573]:         , stdout: , stderr: , exit code -1
Sep 08 13:33:00.051781 ip-10-0-132-254 kubenswrapper[2573]:  > containerID="121c414f6733fd90c07fef254d50dc5e513c01963efd0182b498705993bb4b69" cmd=["/bin/sleep","15"]
Sep 08 13:33:00.052084 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:33:00.051795    2573 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-ll589" message="" err=<
Sep 08 13:33:00.052084 ip-10-0-132-254 kubenswrapper[2573]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 08 13:33:00.052084 ip-10-0-132-254 kubenswrapper[2573]:         , stdout: , stderr: , exit code -1
Sep 08 13:33:00.052084 ip-10-0-132-254 kubenswrapper[2573]:  >
Sep 08 13:33:00.052084 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:33:00.051807    2573 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 08 13:33:00.052084 ip-10-0-132-254 kubenswrapper[2573]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 08 13:33:00.052084 ip-10-0-132-254 kubenswrapper[2573]:         , stdout: , stderr: , exit code -1
Sep 08 13:33:00.052084 ip-10-0-132-254 kubenswrapper[2573]:  > pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-ll589" podUID="7b0fd42f-9d91-41d7-9123-e79b341609ce" containerName="main" containerID="cri-o://121c414f6733fd90c07fef254d50dc5e513c01963efd0182b498705993bb4b69"
Sep 08 13:33:00.052084 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:33:00.051852    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-ll589" podUID="7b0fd42f-9d91-41d7-9123-e79b341609ce" containerName="main" containerID="cri-o://121c414f6733fd90c07fef254d50dc5e513c01963efd0182b498705993bb4b69" gracePeriod=60
Sep 08 13:33:00.137462 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:33:00.137432    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:33:00.397293 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:33:00.397269    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-ll589"
Sep 08 13:33:00.476590 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:33:00.476561    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9dmtj\" (UniqueName: \"kubernetes.io/projected/7b0fd42f-9d91-41d7-9123-e79b341609ce-kube-api-access-9dmtj\") pod \"7b0fd42f-9d91-41d7-9123-e79b341609ce\" (UID: \"7b0fd42f-9d91-41d7-9123-e79b341609ce\") "
Sep 08 13:33:00.476590 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:33:00.476595    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b0fd42f-9d91-41d7-9123-e79b341609ce-tls-certs\") pod \"7b0fd42f-9d91-41d7-9123-e79b341609ce\" (UID: \"7b0fd42f-9d91-41d7-9123-e79b341609ce\") "
Sep 08 13:33:00.476768 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:33:00.476658    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b0fd42f-9d91-41d7-9123-e79b341609ce-tmp-dir\") pod \"7b0fd42f-9d91-41d7-9123-e79b341609ce\" (UID: \"7b0fd42f-9d91-41d7-9123-e79b341609ce\") "
Sep 08 13:33:00.476768 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:33:00.476673    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b0fd42f-9d91-41d7-9123-e79b341609ce-model-cache\") pod \"7b0fd42f-9d91-41d7-9123-e79b341609ce\" (UID: \"7b0fd42f-9d91-41d7-9123-e79b341609ce\") "
Sep 08 13:33:00.476768 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:33:00.476695    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b0fd42f-9d91-41d7-9123-e79b341609ce-dshm\") pod \"7b0fd42f-9d91-41d7-9123-e79b341609ce\" (UID: \"7b0fd42f-9d91-41d7-9123-e79b341609ce\") "
Sep 08 13:33:00.476768 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:33:00.476731    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b0fd42f-9d91-41d7-9123-e79b341609ce-home\") pod \"7b0fd42f-9d91-41d7-9123-e79b341609ce\" (UID: \"7b0fd42f-9d91-41d7-9123-e79b341609ce\") "
Sep 08 13:33:00.476980 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:33:00.476950    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b0fd42f-9d91-41d7-9123-e79b341609ce-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7b0fd42f-9d91-41d7-9123-e79b341609ce" (UID: "7b0fd42f-9d91-41d7-9123-e79b341609ce"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:33:00.477083 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:33:00.476965    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b0fd42f-9d91-41d7-9123-e79b341609ce-model-cache" (OuterVolumeSpecName: "model-cache") pod "7b0fd42f-9d91-41d7-9123-e79b341609ce" (UID: "7b0fd42f-9d91-41d7-9123-e79b341609ce"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:33:00.477083 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:33:00.477041    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b0fd42f-9d91-41d7-9123-e79b341609ce-home" (OuterVolumeSpecName: "home") pod "7b0fd42f-9d91-41d7-9123-e79b341609ce" (UID: "7b0fd42f-9d91-41d7-9123-e79b341609ce"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:33:00.478799 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:33:00.478777    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b0fd42f-9d91-41d7-9123-e79b341609ce-dshm" (OuterVolumeSpecName: "dshm") pod "7b0fd42f-9d91-41d7-9123-e79b341609ce" (UID: "7b0fd42f-9d91-41d7-9123-e79b341609ce"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:33:00.478883 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:33:00.478824    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b0fd42f-9d91-41d7-9123-e79b341609ce-kube-api-access-9dmtj" (OuterVolumeSpecName: "kube-api-access-9dmtj") pod "7b0fd42f-9d91-41d7-9123-e79b341609ce" (UID: "7b0fd42f-9d91-41d7-9123-e79b341609ce"). InnerVolumeSpecName "kube-api-access-9dmtj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:33:00.478883 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:33:00.478833    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b0fd42f-9d91-41d7-9123-e79b341609ce-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7b0fd42f-9d91-41d7-9123-e79b341609ce" (UID: "7b0fd42f-9d91-41d7-9123-e79b341609ce"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:33:00.578319 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:33:00.578239    2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b0fd42f-9d91-41d7-9123-e79b341609ce-tmp-dir\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 13:33:00.578319 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:33:00.578268    2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b0fd42f-9d91-41d7-9123-e79b341609ce-model-cache\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 13:33:00.578319 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:33:00.578277    2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b0fd42f-9d91-41d7-9123-e79b341609ce-dshm\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 13:33:00.578319 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:33:00.578284    2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b0fd42f-9d91-41d7-9123-e79b341609ce-home\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 13:33:00.578319 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:33:00.578292    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9dmtj\" (UniqueName: \"kubernetes.io/projected/7b0fd42f-9d91-41d7-9123-e79b341609ce-kube-api-access-9dmtj\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 13:33:00.578319 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:33:00.578302    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b0fd42f-9d91-41d7-9123-e79b341609ce-tls-certs\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 13:33:00.975241 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:33:00.975210    2573 generic.go:356] "Generic (PLEG): container finished" podID="7b0fd42f-9d91-41d7-9123-e79b341609ce" containerID="121c414f6733fd90c07fef254d50dc5e513c01963efd0182b498705993bb4b69" exitCode=0
Sep 08 13:33:00.975398 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:33:00.975253    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-ll589"
Sep 08 13:33:00.975398 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:33:00.975298    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-ll589" event={"ID":"7b0fd42f-9d91-41d7-9123-e79b341609ce","Type":"ContainerDied","Data":"121c414f6733fd90c07fef254d50dc5e513c01963efd0182b498705993bb4b69"}
Sep 08 13:33:00.975398 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:33:00.975346    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-ll589" event={"ID":"7b0fd42f-9d91-41d7-9123-e79b341609ce","Type":"ContainerDied","Data":"5853866d2c30ca9c1b8a558cc6420ea714e097b8ccd927d7d6a73afdf1807176"}
Sep 08 13:33:00.975398 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:33:00.975372    2573 scope.go:117] "RemoveContainer" containerID="121c414f6733fd90c07fef254d50dc5e513c01963efd0182b498705993bb4b69"
Sep 08 13:33:00.983714 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:33:00.983689    2573 scope.go:117] "RemoveContainer" containerID="121c414f6733fd90c07fef254d50dc5e513c01963efd0182b498705993bb4b69"
Sep 08 13:33:00.984013 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:33:00.983991    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"121c414f6733fd90c07fef254d50dc5e513c01963efd0182b498705993bb4b69\": container with ID starting with 121c414f6733fd90c07fef254d50dc5e513c01963efd0182b498705993bb4b69 not found: ID does not exist" containerID="121c414f6733fd90c07fef254d50dc5e513c01963efd0182b498705993bb4b69"
Sep 08 13:33:00.984103 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:33:00.984017    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"121c414f6733fd90c07fef254d50dc5e513c01963efd0182b498705993bb4b69"} err="failed to get container status \"121c414f6733fd90c07fef254d50dc5e513c01963efd0182b498705993bb4b69\": rpc error: code = NotFound desc = could not find container \"121c414f6733fd90c07fef254d50dc5e513c01963efd0182b498705993bb4b69\": container with ID starting with 121c414f6733fd90c07fef254d50dc5e513c01963efd0182b498705993bb4b69 not found: ID does not exist"
Sep 08 13:33:00.998169 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:33:00.998128    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-ll589"]
Sep 08 13:33:01.004082 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:33:01.004065    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-ll589"]
Sep 08 13:33:02.174241 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:33:02.174200    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b0fd42f-9d91-41d7-9123-e79b341609ce" path="/var/lib/kubelet/pods/7b0fd42f-9d91-41d7-9123-e79b341609ce/volumes"
Sep 08 13:33:12.540042 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:33:12.540008    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:33:15.936651 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:33:15.936616    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:33:18.577676 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:33:18.577647    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:33:24.405757 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:33:24.405725    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:33:24.480145 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:33:24.480114    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:33:34.990267 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:33:34.990227    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 08 13:33:34.990645 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:33:34.990555    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b0fd42f-9d91-41d7-9123-e79b341609ce" containerName="main"
Sep 08 13:33:34.990645 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:33:34.990566    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b0fd42f-9d91-41d7-9123-e79b341609ce" containerName="main"
Sep 08 13:33:34.990645 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:33:34.990616    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b0fd42f-9d91-41d7-9123-e79b341609ce" containerName="main"
Sep 08 13:33:34.993757 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:33:34.993737    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 08 13:33:34.996716 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:33:34.996694    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"seaweedfs-s3-creds\""
Sep 08 13:33:34.996964 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:33:34.996947    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs\""
Sep 08 13:33:34.997455 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:33:34.997439    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"openshift-service-ca.crt\""
Sep 08 13:33:34.997570 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:33:34.997454    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"odh-kserve-custom-ca-bundle\""
Sep 08 13:33:34.997570 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:33:34.997489    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"kube-root-ca.crt\""
Sep 08 13:33:34.997570 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:33:34.997441    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisvc-model-fb-opt-125m-route-dc21cb-57cf7736-dockercfg-xwjms\""
Sep 08 13:33:34.997890 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:33:34.997871    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"default-dockercfg-bxwrt\""
Sep 08 13:33:35.014707 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:33:35.014681    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 08 13:33:35.148408 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:33:35.148380    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2eaeffd2-8f57-4a16-98a5-8f1765e368d3-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"2eaeffd2-8f57-4a16-98a5-8f1765e368d3\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 08 13:33:35.148553 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:33:35.148410    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2eaeffd2-8f57-4a16-98a5-8f1765e368d3-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"2eaeffd2-8f57-4a16-98a5-8f1765e368d3\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 08 13:33:35.148553 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:33:35.148437    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2eaeffd2-8f57-4a16-98a5-8f1765e368d3-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"2eaeffd2-8f57-4a16-98a5-8f1765e368d3\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 08 13:33:35.148553 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:33:35.148466    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2eaeffd2-8f57-4a16-98a5-8f1765e368d3-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"2eaeffd2-8f57-4a16-98a5-8f1765e368d3\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 08 13:33:35.148553 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:33:35.148488    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2eaeffd2-8f57-4a16-98a5-8f1765e368d3-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"2eaeffd2-8f57-4a16-98a5-8f1765e368d3\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 08 13:33:35.148553 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:33:35.148523    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cb7tf\" (UniqueName: \"kubernetes.io/projected/2eaeffd2-8f57-4a16-98a5-8f1765e368d3-kube-api-access-cb7tf\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"2eaeffd2-8f57-4a16-98a5-8f1765e368d3\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 08 13:33:35.148796 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:33:35.148619    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2eaeffd2-8f57-4a16-98a5-8f1765e368d3-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"2eaeffd2-8f57-4a16-98a5-8f1765e368d3\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 08 13:33:35.148796 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:33:35.148681    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2eaeffd2-8f57-4a16-98a5-8f1765e368d3-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"2eaeffd2-8f57-4a16-98a5-8f1765e368d3\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 08 13:33:35.249340 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:33:35.249283    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2eaeffd2-8f57-4a16-98a5-8f1765e368d3-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"2eaeffd2-8f57-4a16-98a5-8f1765e368d3\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 08 13:33:35.249340 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:33:35.249322    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2eaeffd2-8f57-4a16-98a5-8f1765e368d3-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"2eaeffd2-8f57-4a16-98a5-8f1765e368d3\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 08 13:33:35.249493 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:33:35.249343    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2eaeffd2-8f57-4a16-98a5-8f1765e368d3-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"2eaeffd2-8f57-4a16-98a5-8f1765e368d3\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 08 13:33:35.249493 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:33:35.249361    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2eaeffd2-8f57-4a16-98a5-8f1765e368d3-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"2eaeffd2-8f57-4a16-98a5-8f1765e368d3\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 08 13:33:35.249493 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:33:35.249393    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2eaeffd2-8f57-4a16-98a5-8f1765e368d3-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"2eaeffd2-8f57-4a16-98a5-8f1765e368d3\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 08 13:33:35.249493 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:33:35.249425    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2eaeffd2-8f57-4a16-98a5-8f1765e368d3-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"2eaeffd2-8f57-4a16-98a5-8f1765e368d3\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 08 13:33:35.249493 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:33:35.249459    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2eaeffd2-8f57-4a16-98a5-8f1765e368d3-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"2eaeffd2-8f57-4a16-98a5-8f1765e368d3\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 08 13:33:35.249493 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:33:35.249485    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cb7tf\" (UniqueName: \"kubernetes.io/projected/2eaeffd2-8f57-4a16-98a5-8f1765e368d3-kube-api-access-cb7tf\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"2eaeffd2-8f57-4a16-98a5-8f1765e368d3\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 08 13:33:35.249805 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:33:35.249656    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2eaeffd2-8f57-4a16-98a5-8f1765e368d3-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"2eaeffd2-8f57-4a16-98a5-8f1765e368d3\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 08 13:33:35.249805 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:33:35.249713    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2eaeffd2-8f57-4a16-98a5-8f1765e368d3-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"2eaeffd2-8f57-4a16-98a5-8f1765e368d3\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 08 13:33:35.249915 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:33:35.249826    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2eaeffd2-8f57-4a16-98a5-8f1765e368d3-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"2eaeffd2-8f57-4a16-98a5-8f1765e368d3\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 08 13:33:35.250021 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:33:35.249943    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2eaeffd2-8f57-4a16-98a5-8f1765e368d3-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"2eaeffd2-8f57-4a16-98a5-8f1765e368d3\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 08 13:33:35.250082 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:33:35.250059    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2eaeffd2-8f57-4a16-98a5-8f1765e368d3-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"2eaeffd2-8f57-4a16-98a5-8f1765e368d3\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 08 13:33:35.251571 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:33:35.251551    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2eaeffd2-8f57-4a16-98a5-8f1765e368d3-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"2eaeffd2-8f57-4a16-98a5-8f1765e368d3\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 08 13:33:35.251758 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:33:35.251741    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2eaeffd2-8f57-4a16-98a5-8f1765e368d3-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"2eaeffd2-8f57-4a16-98a5-8f1765e368d3\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 08 13:33:35.257481 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:33:35.257461    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cb7tf\" (UniqueName: \"kubernetes.io/projected/2eaeffd2-8f57-4a16-98a5-8f1765e368d3-kube-api-access-cb7tf\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"2eaeffd2-8f57-4a16-98a5-8f1765e368d3\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 08 13:33:35.304083 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:33:35.304060    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 08 13:33:35.560783 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:33:35.560701    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:33:35.583642 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:33:35.583621    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 08 13:33:35.586293 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:33:35.586270    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2eaeffd2_8f57_4a16_98a5_8f1765e368d3.slice/crio-bda771aca6ac2356393cf88232f69d3bdacd87382041a8e5ed54bfb17c745722 WatchSource:0}: Error finding container bda771aca6ac2356393cf88232f69d3bdacd87382041a8e5ed54bfb17c745722: Status 404 returned error can't find the container with id bda771aca6ac2356393cf88232f69d3bdacd87382041a8e5ed54bfb17c745722
Sep 08 13:33:35.660703 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:33:35.660679    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:33:36.115843 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:33:36.115806    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"2eaeffd2-8f57-4a16-98a5-8f1765e368d3","Type":"ContainerStarted","Data":"bda771aca6ac2356393cf88232f69d3bdacd87382041a8e5ed54bfb17c745722"}
Sep 08 13:33:39.132605 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:33:39.132574    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"2eaeffd2-8f57-4a16-98a5-8f1765e368d3","Type":"ContainerStarted","Data":"7fe055aa919f222295a7815fa2a59f387965042fe92dccd636890bda7ab3f6e2"}
Sep 08 13:33:41.142782 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:33:41.142751    2573 generic.go:356] "Generic (PLEG): container finished" podID="2eaeffd2-8f57-4a16-98a5-8f1765e368d3" containerID="7fe055aa919f222295a7815fa2a59f387965042fe92dccd636890bda7ab3f6e2" exitCode=0
Sep 08 13:33:41.143067 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:33:41.142820    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"2eaeffd2-8f57-4a16-98a5-8f1765e368d3","Type":"ContainerDied","Data":"7fe055aa919f222295a7815fa2a59f387965042fe92dccd636890bda7ab3f6e2"}
Sep 08 13:33:50.090821 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:33:50.090735    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:33:50.192998 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:33:50.192962    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:33:50.300908 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:33:50.300870    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:33:50.398808 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:33:50.398725    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:33:50.469554 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:33:50.469438    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:33:53.724442 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:33:53.724345    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:33:54.925639 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:33:54.925598    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:33:56.975122 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:33:56.975089    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:34:02.826208 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:34:02.826162    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:34:02.890284 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:34:02.890245    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:34:10.298587 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:34:10.298551    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"2eaeffd2-8f57-4a16-98a5-8f1765e368d3","Type":"ContainerStarted","Data":"963210dcd6e4c54b7c64ea80424767a4f84a7a311b678078932156673a821bfa"}
Sep 08 13:34:10.320772 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:34:10.320709    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" podStartSLOduration=2.845338478 podStartE2EDuration="36.320689103s" totalImagesPullingTime="31.44224698s" totalInitContainerRuntime="2.033103645s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:33:34 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:34:10.319008673 +0000 UTC m=+1100.590456604" watchObservedRunningTime="2026-09-08 13:34:10.320689103 +0000 UTC m=+1100.592137057"
Sep 08 13:34:13.145637 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:34:13.145604    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:34:18.181656 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:34:18.181627    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:34:20.752581 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:34:20.752542    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:34:23.749198 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:34:23.749163    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:34:27.597547 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:34:27.597515    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:34:30.583885 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:34:30.583839    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:34:30.689813 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:34:30.689778    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:34:34.081483 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:34:34.081450    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:34:35.411288 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:34:35.411258    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:34:35.483313 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:34:35.483282    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:34:39.348340 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:34:39.348308    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:34:42.151380 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:34:42.151340    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:34:42.251245 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:34:42.251215    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:34:43.541980 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:34:43.541941    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:34:47.236621 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:34:47.236581    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:34:52.840159 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:34:52.840122    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:34:53.942276 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:34:53.942239    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:34:57.803371 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:34:57.803333    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-cfc4f4c99-89f9s"]
Sep 08 13:34:57.806920 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:34:57.806899    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-cfc4f4c99-89f9s"
Sep 08 13:34:57.809303 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:34:57.809278    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"default-dockercfg-8vx65\""
Sep 08 13:34:57.809536 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:34:57.809289    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"openshift-service-ca.crt\""
Sep 08 13:34:57.809631 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:34:57.809334    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"odh-kserve-custom-ca-bundle\""
Sep 08 13:34:57.809689 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:34:57.809357    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"kube-root-ca.crt\""
Sep 08 13:34:57.809743 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:34:57.809365    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"auth-enabled-test-kserve-self-signed-certs\""
Sep 08 13:34:57.809797 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:34:57.809426    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"seaweedfs-s3-creds\""
Sep 08 13:34:57.818975 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:34:57.818954    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-cfc4f4c99-89f9s"]
Sep 08 13:34:57.829416 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:34:57.829383    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:34:57.863514 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:34:57.863466    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e52dc73e-8573-4cdc-a3f1-6be624510265-kserve-provision-location\") pod \"auth-enabled-test-kserve-cfc4f4c99-89f9s\" (UID: \"e52dc73e-8573-4cdc-a3f1-6be624510265\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-cfc4f4c99-89f9s"
Sep 08 13:34:57.863514 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:34:57.863494    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e52dc73e-8573-4cdc-a3f1-6be624510265-cabundle-cert\") pod \"auth-enabled-test-kserve-cfc4f4c99-89f9s\" (UID: \"e52dc73e-8573-4cdc-a3f1-6be624510265\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-cfc4f4c99-89f9s"
Sep 08 13:34:57.863697 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:34:57.863532    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e52dc73e-8573-4cdc-a3f1-6be624510265-tls-certs\") pod \"auth-enabled-test-kserve-cfc4f4c99-89f9s\" (UID: \"e52dc73e-8573-4cdc-a3f1-6be624510265\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-cfc4f4c99-89f9s"
Sep 08 13:34:57.863697 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:34:57.863616    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e52dc73e-8573-4cdc-a3f1-6be624510265-tmp-dir\") pod \"auth-enabled-test-kserve-cfc4f4c99-89f9s\" (UID: \"e52dc73e-8573-4cdc-a3f1-6be624510265\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-cfc4f4c99-89f9s"
Sep 08 13:34:57.863697 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:34:57.863655    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e52dc73e-8573-4cdc-a3f1-6be624510265-home\") pod \"auth-enabled-test-kserve-cfc4f4c99-89f9s\" (UID: \"e52dc73e-8573-4cdc-a3f1-6be624510265\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-cfc4f4c99-89f9s"
Sep 08 13:34:57.863697 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:34:57.863674    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e52dc73e-8573-4cdc-a3f1-6be624510265-model-cache\") pod \"auth-enabled-test-kserve-cfc4f4c99-89f9s\" (UID: \"e52dc73e-8573-4cdc-a3f1-6be624510265\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-cfc4f4c99-89f9s"
Sep 08 13:34:57.863930 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:34:57.863717    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fh4j7\" (UniqueName: \"kubernetes.io/projected/e52dc73e-8573-4cdc-a3f1-6be624510265-kube-api-access-fh4j7\") pod \"auth-enabled-test-kserve-cfc4f4c99-89f9s\" (UID: \"e52dc73e-8573-4cdc-a3f1-6be624510265\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-cfc4f4c99-89f9s"
Sep 08 13:34:57.863930 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:34:57.863737    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e52dc73e-8573-4cdc-a3f1-6be624510265-dshm\") pod \"auth-enabled-test-kserve-cfc4f4c99-89f9s\" (UID: \"e52dc73e-8573-4cdc-a3f1-6be624510265\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-cfc4f4c99-89f9s"
Sep 08 13:34:57.965135 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:34:57.965104    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e52dc73e-8573-4cdc-a3f1-6be624510265-home\") pod \"auth-enabled-test-kserve-cfc4f4c99-89f9s\" (UID: \"e52dc73e-8573-4cdc-a3f1-6be624510265\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-cfc4f4c99-89f9s"
Sep 08 13:34:57.965135 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:34:57.965135    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e52dc73e-8573-4cdc-a3f1-6be624510265-model-cache\") pod \"auth-enabled-test-kserve-cfc4f4c99-89f9s\" (UID: \"e52dc73e-8573-4cdc-a3f1-6be624510265\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-cfc4f4c99-89f9s"
Sep 08 13:34:57.965355 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:34:57.965157    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fh4j7\" (UniqueName: \"kubernetes.io/projected/e52dc73e-8573-4cdc-a3f1-6be624510265-kube-api-access-fh4j7\") pod \"auth-enabled-test-kserve-cfc4f4c99-89f9s\" (UID: \"e52dc73e-8573-4cdc-a3f1-6be624510265\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-cfc4f4c99-89f9s"
Sep 08 13:34:57.965355 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:34:57.965177    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e52dc73e-8573-4cdc-a3f1-6be624510265-dshm\") pod \"auth-enabled-test-kserve-cfc4f4c99-89f9s\" (UID: \"e52dc73e-8573-4cdc-a3f1-6be624510265\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-cfc4f4c99-89f9s"
Sep 08 13:34:57.965355 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:34:57.965212    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e52dc73e-8573-4cdc-a3f1-6be624510265-kserve-provision-location\") pod \"auth-enabled-test-kserve-cfc4f4c99-89f9s\" (UID: \"e52dc73e-8573-4cdc-a3f1-6be624510265\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-cfc4f4c99-89f9s"
Sep 08 13:34:57.965355 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:34:57.965229    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e52dc73e-8573-4cdc-a3f1-6be624510265-cabundle-cert\") pod \"auth-enabled-test-kserve-cfc4f4c99-89f9s\" (UID: \"e52dc73e-8573-4cdc-a3f1-6be624510265\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-cfc4f4c99-89f9s"
Sep 08 13:34:57.965355 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:34:57.965257    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e52dc73e-8573-4cdc-a3f1-6be624510265-tls-certs\") pod \"auth-enabled-test-kserve-cfc4f4c99-89f9s\" (UID: \"e52dc73e-8573-4cdc-a3f1-6be624510265\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-cfc4f4c99-89f9s"
Sep 08 13:34:57.965355 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:34:57.965315    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e52dc73e-8573-4cdc-a3f1-6be624510265-tmp-dir\") pod \"auth-enabled-test-kserve-cfc4f4c99-89f9s\" (UID: \"e52dc73e-8573-4cdc-a3f1-6be624510265\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-cfc4f4c99-89f9s"
Sep 08 13:34:57.965718 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:34:57.965550    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e52dc73e-8573-4cdc-a3f1-6be624510265-home\") pod \"auth-enabled-test-kserve-cfc4f4c99-89f9s\" (UID: \"e52dc73e-8573-4cdc-a3f1-6be624510265\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-cfc4f4c99-89f9s"
Sep 08 13:34:57.965718 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:34:57.965582    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e52dc73e-8573-4cdc-a3f1-6be624510265-model-cache\") pod \"auth-enabled-test-kserve-cfc4f4c99-89f9s\" (UID: \"e52dc73e-8573-4cdc-a3f1-6be624510265\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-cfc4f4c99-89f9s"
Sep 08 13:34:57.966075 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:34:57.966046    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e52dc73e-8573-4cdc-a3f1-6be624510265-cabundle-cert\") pod \"auth-enabled-test-kserve-cfc4f4c99-89f9s\" (UID: \"e52dc73e-8573-4cdc-a3f1-6be624510265\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-cfc4f4c99-89f9s"
Sep 08 13:34:57.966184 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:34:57.966130    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e52dc73e-8573-4cdc-a3f1-6be624510265-tmp-dir\") pod \"auth-enabled-test-kserve-cfc4f4c99-89f9s\" (UID: \"e52dc73e-8573-4cdc-a3f1-6be624510265\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-cfc4f4c99-89f9s"
Sep 08 13:34:57.968339 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:34:57.966168    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e52dc73e-8573-4cdc-a3f1-6be624510265-kserve-provision-location\") pod \"auth-enabled-test-kserve-cfc4f4c99-89f9s\" (UID: \"e52dc73e-8573-4cdc-a3f1-6be624510265\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-cfc4f4c99-89f9s"
Sep 08 13:34:57.968339 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:34:57.968247    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e52dc73e-8573-4cdc-a3f1-6be624510265-tls-certs\") pod \"auth-enabled-test-kserve-cfc4f4c99-89f9s\" (UID: \"e52dc73e-8573-4cdc-a3f1-6be624510265\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-cfc4f4c99-89f9s"
Sep 08 13:34:57.975299 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:34:57.968767    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e52dc73e-8573-4cdc-a3f1-6be624510265-dshm\") pod \"auth-enabled-test-kserve-cfc4f4c99-89f9s\" (UID: \"e52dc73e-8573-4cdc-a3f1-6be624510265\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-cfc4f4c99-89f9s"
Sep 08 13:34:57.975299 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:34:57.973240    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fh4j7\" (UniqueName: \"kubernetes.io/projected/e52dc73e-8573-4cdc-a3f1-6be624510265-kube-api-access-fh4j7\") pod \"auth-enabled-test-kserve-cfc4f4c99-89f9s\" (UID: \"e52dc73e-8573-4cdc-a3f1-6be624510265\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-cfc4f4c99-89f9s"
Sep 08 13:34:58.089314 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:34:58.089232    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:34:58.117541 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:34:58.117517    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-cfc4f4c99-89f9s"
Sep 08 13:34:58.396094 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:34:58.396067    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-cfc4f4c99-89f9s"]
Sep 08 13:34:58.398635 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:34:58.398606    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode52dc73e_8573_4cdc_a3f1_6be624510265.slice/crio-abeb44b45b3790a0b1756f75808197f35cf914b83ac0ba1d961e72d5db6366af WatchSource:0}: Error finding container abeb44b45b3790a0b1756f75808197f35cf914b83ac0ba1d961e72d5db6366af: Status 404 returned error can't find the container with id abeb44b45b3790a0b1756f75808197f35cf914b83ac0ba1d961e72d5db6366af
Sep 08 13:34:58.507557 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:34:58.507520    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-cfc4f4c99-89f9s" event={"ID":"e52dc73e-8573-4cdc-a3f1-6be624510265","Type":"ContainerStarted","Data":"c67723530f399d821c9423cb5fc9ba979e4701837b5440caafbc2c4e17e4ad61"}
Sep 08 13:34:58.507727 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:34:58.507581    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-cfc4f4c99-89f9s" event={"ID":"e52dc73e-8573-4cdc-a3f1-6be624510265","Type":"ContainerStarted","Data":"abeb44b45b3790a0b1756f75808197f35cf914b83ac0ba1d961e72d5db6366af"}
Sep 08 13:35:01.524369 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:35:01.523337    2573 generic.go:356] "Generic (PLEG): container finished" podID="e52dc73e-8573-4cdc-a3f1-6be624510265" containerID="c67723530f399d821c9423cb5fc9ba979e4701837b5440caafbc2c4e17e4ad61" exitCode=0
Sep 08 13:35:01.524369 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:35:01.523397    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-cfc4f4c99-89f9s" event={"ID":"e52dc73e-8573-4cdc-a3f1-6be624510265","Type":"ContainerDied","Data":"c67723530f399d821c9423cb5fc9ba979e4701837b5440caafbc2c4e17e4ad61"}
Sep 08 13:35:02.530932 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:35:02.530897    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-cfc4f4c99-89f9s" event={"ID":"e52dc73e-8573-4cdc-a3f1-6be624510265","Type":"ContainerStarted","Data":"eced02329eeb8e329e5ce9466a8d8e18d41eb1a89e378e96e33b368b342b9762"}
Sep 08 13:35:02.554864 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:35:02.554813    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-cfc4f4c99-89f9s" podStartSLOduration=3.258041656 podStartE2EDuration="5.554795761s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.296754105s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:34:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:35:02.55234895 +0000 UTC m=+1152.823796882" watchObservedRunningTime="2026-09-08 13:35:02.554795761 +0000 UTC m=+1152.826243693"
Sep 08 13:35:03.117843 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:35:03.117799    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-cfc4f4c99-89f9s"
Sep 08 13:35:04.730512 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:35:04.730465    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:35:08.117836 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:35:08.117807    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-cfc4f4c99-89f9s"
Sep 08 13:35:08.119489 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:35:08.119456    2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-cfc4f4c99-89f9s" podUID="e52dc73e-8573-4cdc-a3f1-6be624510265" containerName="main" probeResult="failure" output="Get \"https://10.133.0.23:8000/health\": dial tcp 10.133.0.23:8000: connect: connection refused"
Sep 08 13:35:10.380624 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:35:10.380571    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:35:15.147358 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:35:15.147248    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:35:15.211401 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:35:15.211369    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:35:18.117966 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:35:18.117922    2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-cfc4f4c99-89f9s" podUID="e52dc73e-8573-4cdc-a3f1-6be624510265" containerName="main" probeResult="failure" output="Get \"https://10.133.0.23:8000/health\": dial tcp 10.133.0.23:8000: connect: connection refused"
Sep 08 13:35:18.345463 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:35:18.345424    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:35:18.404411 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:35:18.404378    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:35:21.023250 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:35:21.023217    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:35:21.123843 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:35:21.123806    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:35:25.170369 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:35:25.170322    2573 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" secret="" err="secret \"llmisvc-model-fb-opt-125m-route-dc21cb-57cf7736-dockercfg-xwjms\" not found"
Sep 08 13:35:25.313948 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:35:25.313902    2573 secret.go:189] Couldn't get secret e2e-test-llm-inference-service-63a03ef4/llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs: secret "llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs" not found
Sep 08 13:35:25.314150 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:35:25.313983    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2eaeffd2-8f57-4a16-98a5-8f1765e368d3-tls-certs podName:2eaeffd2-8f57-4a16-98a5-8f1765e368d3 nodeName:}" failed. No retries permitted until 2026-09-08 13:35:25.813966485 +0000 UTC m=+1176.085414394 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/2eaeffd2-8f57-4a16-98a5-8f1765e368d3-tls-certs") pod "llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" (UID: "2eaeffd2-8f57-4a16-98a5-8f1765e368d3") : secret "llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs" not found
Sep 08 13:35:25.818407 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:35:25.818361    2573 secret.go:189] Couldn't get secret e2e-test-llm-inference-service-63a03ef4/llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs: secret "llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs" not found
Sep 08 13:35:25.818727 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:35:25.818478    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2eaeffd2-8f57-4a16-98a5-8f1765e368d3-tls-certs podName:2eaeffd2-8f57-4a16-98a5-8f1765e368d3 nodeName:}" failed. No retries permitted until 2026-09-08 13:35:26.818455667 +0000 UTC m=+1177.089903594 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/2eaeffd2-8f57-4a16-98a5-8f1765e368d3-tls-certs") pod "llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" (UID: "2eaeffd2-8f57-4a16-98a5-8f1765e368d3") : secret "llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs" not found
Sep 08 13:35:26.827802 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:35:26.827762    2573 secret.go:189] Couldn't get secret e2e-test-llm-inference-service-63a03ef4/llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs: secret "llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs" not found
Sep 08 13:35:26.828298 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:35:26.827833    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2eaeffd2-8f57-4a16-98a5-8f1765e368d3-tls-certs podName:2eaeffd2-8f57-4a16-98a5-8f1765e368d3 nodeName:}" failed. No retries permitted until 2026-09-08 13:35:28.827815964 +0000 UTC m=+1179.099263874 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/2eaeffd2-8f57-4a16-98a5-8f1765e368d3-tls-certs") pod "llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" (UID: "2eaeffd2-8f57-4a16-98a5-8f1765e368d3") : secret "llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs" not found
Sep 08 13:35:28.118822 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:35:28.118771    2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-cfc4f4c99-89f9s" podUID="e52dc73e-8573-4cdc-a3f1-6be624510265" containerName="main" probeResult="failure" output="Get \"https://10.133.0.23:8000/health\": dial tcp 10.133.0.23:8000: connect: connection refused"
Sep 08 13:35:28.846250 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:35:28.846211    2573 secret.go:189] Couldn't get secret e2e-test-llm-inference-service-63a03ef4/llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs: secret "llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs" not found
Sep 08 13:35:28.846464 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:35:28.846277    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2eaeffd2-8f57-4a16-98a5-8f1765e368d3-tls-certs podName:2eaeffd2-8f57-4a16-98a5-8f1765e368d3 nodeName:}" failed. No retries permitted until 2026-09-08 13:35:32.846263373 +0000 UTC m=+1183.117711283 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/2eaeffd2-8f57-4a16-98a5-8f1765e368d3-tls-certs") pod "llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" (UID: "2eaeffd2-8f57-4a16-98a5-8f1765e368d3") : secret "llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs" not found
Sep 08 13:35:32.884151 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:35:32.884099    2573 secret.go:189] Couldn't get secret e2e-test-llm-inference-service-63a03ef4/llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs: secret "llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs" not found
Sep 08 13:35:32.884655 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:35:32.884214    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2eaeffd2-8f57-4a16-98a5-8f1765e368d3-tls-certs podName:2eaeffd2-8f57-4a16-98a5-8f1765e368d3 nodeName:}" failed. No retries permitted until 2026-09-08 13:35:40.884189722 +0000 UTC m=+1191.155637644 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/2eaeffd2-8f57-4a16-98a5-8f1765e368d3-tls-certs") pod "llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" (UID: "2eaeffd2-8f57-4a16-98a5-8f1765e368d3") : secret "llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs" not found
Sep 08 13:35:37.592383 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:35:37.592339    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 08 13:35:38.118494 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:35:38.118444    2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-cfc4f4c99-89f9s" podUID="e52dc73e-8573-4cdc-a3f1-6be624510265" containerName="main" probeResult="failure" output="Get \"https://10.133.0.23:8000/health\": dial tcp 10.133.0.23:8000: connect: connection refused"
Sep 08 13:35:40.959347 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:35:40.959308    2573 secret.go:189] Couldn't get secret e2e-test-llm-inference-service-63a03ef4/llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs: secret "llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs" not found
Sep 08 13:35:40.959930 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:35:40.959377    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2eaeffd2-8f57-4a16-98a5-8f1765e368d3-tls-certs podName:2eaeffd2-8f57-4a16-98a5-8f1765e368d3 nodeName:}" failed. No retries permitted until 2026-09-08 13:35:56.959362873 +0000 UTC m=+1207.230810782 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/2eaeffd2-8f57-4a16-98a5-8f1765e368d3-tls-certs") pod "llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" (UID: "2eaeffd2-8f57-4a16-98a5-8f1765e368d3") : secret "llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs" not found
Sep 08 13:35:48.118437 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:35:48.118393    2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-cfc4f4c99-89f9s" podUID="e52dc73e-8573-4cdc-a3f1-6be624510265" containerName="main" probeResult="failure" output="Get \"https://10.133.0.23:8000/health\": dial tcp 10.133.0.23:8000: connect: connection refused"
Sep 08 13:35:50.198090 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:35:50.198057    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8h8cn_b936238e-bdf1-4dac-8bf8-6417c313597c/ovn-acl-logging/0.log"
Sep 08 13:35:50.200911 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:35:50.200889    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8h8cn_b936238e-bdf1-4dac-8bf8-6417c313597c/ovn-acl-logging/0.log"
Sep 08 13:35:52.615884 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:35:52.615840    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" podUID="2eaeffd2-8f57-4a16-98a5-8f1765e368d3" containerName="main" containerID="cri-o://963210dcd6e4c54b7c64ea80424767a4f84a7a311b678078932156673a821bfa" gracePeriod=45
Sep 08 13:35:53.269137 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:35:53.269111    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 08 13:35:53.373510 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:35:53.373480    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2eaeffd2-8f57-4a16-98a5-8f1765e368d3-kserve-provision-location\") pod \"2eaeffd2-8f57-4a16-98a5-8f1765e368d3\" (UID: \"2eaeffd2-8f57-4a16-98a5-8f1765e368d3\") "
Sep 08 13:35:53.373675 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:35:53.373566    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2eaeffd2-8f57-4a16-98a5-8f1765e368d3-cabundle-cert\") pod \"2eaeffd2-8f57-4a16-98a5-8f1765e368d3\" (UID: \"2eaeffd2-8f57-4a16-98a5-8f1765e368d3\") "
Sep 08 13:35:53.373675 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:35:53.373597    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2eaeffd2-8f57-4a16-98a5-8f1765e368d3-model-cache\") pod \"2eaeffd2-8f57-4a16-98a5-8f1765e368d3\" (UID: \"2eaeffd2-8f57-4a16-98a5-8f1765e368d3\") "
Sep 08 13:35:53.373675 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:35:53.373619    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2eaeffd2-8f57-4a16-98a5-8f1765e368d3-home\") pod \"2eaeffd2-8f57-4a16-98a5-8f1765e368d3\" (UID: \"2eaeffd2-8f57-4a16-98a5-8f1765e368d3\") "
Sep 08 13:35:53.373884 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:35:53.373834    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cb7tf\" (UniqueName: \"kubernetes.io/projected/2eaeffd2-8f57-4a16-98a5-8f1765e368d3-kube-api-access-cb7tf\") pod \"2eaeffd2-8f57-4a16-98a5-8f1765e368d3\" (UID: \"2eaeffd2-8f57-4a16-98a5-8f1765e368d3\") "
Sep 08 13:35:53.373884 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:35:53.373862    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2eaeffd2-8f57-4a16-98a5-8f1765e368d3-model-cache" (OuterVolumeSpecName: "model-cache") pod "2eaeffd2-8f57-4a16-98a5-8f1765e368d3" (UID: "2eaeffd2-8f57-4a16-98a5-8f1765e368d3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:35:53.374023 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:35:53.373884    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2eaeffd2-8f57-4a16-98a5-8f1765e368d3-tmp-dir\") pod \"2eaeffd2-8f57-4a16-98a5-8f1765e368d3\" (UID: \"2eaeffd2-8f57-4a16-98a5-8f1765e368d3\") "
Sep 08 13:35:53.374023 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:35:53.373920    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2eaeffd2-8f57-4a16-98a5-8f1765e368d3-dshm\") pod \"2eaeffd2-8f57-4a16-98a5-8f1765e368d3\" (UID: \"2eaeffd2-8f57-4a16-98a5-8f1765e368d3\") "
Sep 08 13:35:53.374023 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:35:53.373954    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2eaeffd2-8f57-4a16-98a5-8f1765e368d3-tls-certs\") pod \"2eaeffd2-8f57-4a16-98a5-8f1765e368d3\" (UID: \"2eaeffd2-8f57-4a16-98a5-8f1765e368d3\") "
Sep 08 13:35:53.374164 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:35:53.374016    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2eaeffd2-8f57-4a16-98a5-8f1765e368d3-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2eaeffd2-8f57-4a16-98a5-8f1765e368d3" (UID: "2eaeffd2-8f57-4a16-98a5-8f1765e368d3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:35:53.374263 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:35:53.374246    2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2eaeffd2-8f57-4a16-98a5-8f1765e368d3-cabundle-cert\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 13:35:53.374315 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:35:53.374269    2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2eaeffd2-8f57-4a16-98a5-8f1765e368d3-model-cache\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 13:35:53.374315 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:35:53.374300    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2eaeffd2-8f57-4a16-98a5-8f1765e368d3-home" (OuterVolumeSpecName: "home") pod "2eaeffd2-8f57-4a16-98a5-8f1765e368d3" (UID: "2eaeffd2-8f57-4a16-98a5-8f1765e368d3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:35:53.376294 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:35:53.376253    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2eaeffd2-8f57-4a16-98a5-8f1765e368d3-kube-api-access-cb7tf" (OuterVolumeSpecName: "kube-api-access-cb7tf") pod "2eaeffd2-8f57-4a16-98a5-8f1765e368d3" (UID: "2eaeffd2-8f57-4a16-98a5-8f1765e368d3"). InnerVolumeSpecName "kube-api-access-cb7tf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:35:53.376409 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:35:53.376335    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2eaeffd2-8f57-4a16-98a5-8f1765e368d3-dshm" (OuterVolumeSpecName: "dshm") pod "2eaeffd2-8f57-4a16-98a5-8f1765e368d3" (UID: "2eaeffd2-8f57-4a16-98a5-8f1765e368d3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:35:53.377012 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:35:53.376992    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2eaeffd2-8f57-4a16-98a5-8f1765e368d3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2eaeffd2-8f57-4a16-98a5-8f1765e368d3" (UID: "2eaeffd2-8f57-4a16-98a5-8f1765e368d3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:35:53.386037 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:35:53.386007    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2eaeffd2-8f57-4a16-98a5-8f1765e368d3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2eaeffd2-8f57-4a16-98a5-8f1765e368d3" (UID: "2eaeffd2-8f57-4a16-98a5-8f1765e368d3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:35:53.404907 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:35:53.404869    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2eaeffd2-8f57-4a16-98a5-8f1765e368d3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2eaeffd2-8f57-4a16-98a5-8f1765e368d3" (UID: "2eaeffd2-8f57-4a16-98a5-8f1765e368d3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:35:53.475215 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:35:53.475179    2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2eaeffd2-8f57-4a16-98a5-8f1765e368d3-home\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 13:35:53.475215 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:35:53.475210    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cb7tf\" (UniqueName: \"kubernetes.io/projected/2eaeffd2-8f57-4a16-98a5-8f1765e368d3-kube-api-access-cb7tf\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 13:35:53.475215 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:35:53.475221    2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2eaeffd2-8f57-4a16-98a5-8f1765e368d3-tmp-dir\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 13:35:53.475458 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:35:53.475229    2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2eaeffd2-8f57-4a16-98a5-8f1765e368d3-dshm\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 13:35:53.475458 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:35:53.475238    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2eaeffd2-8f57-4a16-98a5-8f1765e368d3-tls-certs\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 13:35:53.475458 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:35:53.475247    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2eaeffd2-8f57-4a16-98a5-8f1765e368d3-kserve-provision-location\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 13:35:53.792240 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:35:53.792148    2573 generic.go:356] "Generic (PLEG): container finished" podID="2eaeffd2-8f57-4a16-98a5-8f1765e368d3" containerID="963210dcd6e4c54b7c64ea80424767a4f84a7a311b678078932156673a821bfa" exitCode=0
Sep 08 13:35:53.792240 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:35:53.792206    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 08 13:35:53.792240 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:35:53.792229    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"2eaeffd2-8f57-4a16-98a5-8f1765e368d3","Type":"ContainerDied","Data":"963210dcd6e4c54b7c64ea80424767a4f84a7a311b678078932156673a821bfa"}
Sep 08 13:35:53.792834 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:35:53.792283    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"2eaeffd2-8f57-4a16-98a5-8f1765e368d3","Type":"ContainerDied","Data":"bda771aca6ac2356393cf88232f69d3bdacd87382041a8e5ed54bfb17c745722"}
Sep 08 13:35:53.792834 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:35:53.792304    2573 scope.go:117] "RemoveContainer" containerID="963210dcd6e4c54b7c64ea80424767a4f84a7a311b678078932156673a821bfa"
Sep 08 13:35:53.807689 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:35:53.807665    2573 scope.go:117] "RemoveContainer" containerID="7fe055aa919f222295a7815fa2a59f387965042fe92dccd636890bda7ab3f6e2"
Sep 08 13:35:53.815645 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:35:53.815622    2573 scope.go:117] "RemoveContainer" containerID="963210dcd6e4c54b7c64ea80424767a4f84a7a311b678078932156673a821bfa"
Sep 08 13:35:53.816075 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:35:53.816049    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"963210dcd6e4c54b7c64ea80424767a4f84a7a311b678078932156673a821bfa\": container with ID starting with 963210dcd6e4c54b7c64ea80424767a4f84a7a311b678078932156673a821bfa not found: ID does not exist" containerID="963210dcd6e4c54b7c64ea80424767a4f84a7a311b678078932156673a821bfa"
Sep 08 13:35:53.816164 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:35:53.816108    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"963210dcd6e4c54b7c64ea80424767a4f84a7a311b678078932156673a821bfa"} err="failed to get container status \"963210dcd6e4c54b7c64ea80424767a4f84a7a311b678078932156673a821bfa\": rpc error: code = NotFound desc = could not find container \"963210dcd6e4c54b7c64ea80424767a4f84a7a311b678078932156673a821bfa\": container with ID starting with 963210dcd6e4c54b7c64ea80424767a4f84a7a311b678078932156673a821bfa not found: ID does not exist"
Sep 08 13:35:53.816164 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:35:53.816150    2573 scope.go:117] "RemoveContainer" containerID="7fe055aa919f222295a7815fa2a59f387965042fe92dccd636890bda7ab3f6e2"
Sep 08 13:35:53.816622 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:35:53.816601    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fe055aa919f222295a7815fa2a59f387965042fe92dccd636890bda7ab3f6e2\": container with ID starting with 7fe055aa919f222295a7815fa2a59f387965042fe92dccd636890bda7ab3f6e2 not found: ID does not exist" containerID="7fe055aa919f222295a7815fa2a59f387965042fe92dccd636890bda7ab3f6e2"
Sep 08 13:35:53.816706 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:35:53.816636    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fe055aa919f222295a7815fa2a59f387965042fe92dccd636890bda7ab3f6e2"} err="failed to get container status \"7fe055aa919f222295a7815fa2a59f387965042fe92dccd636890bda7ab3f6e2\": rpc error: code = NotFound desc = could not find container \"7fe055aa919f222295a7815fa2a59f387965042fe92dccd636890bda7ab3f6e2\": container with ID starting with 7fe055aa919f222295a7815fa2a59f387965042fe92dccd636890bda7ab3f6e2 not found: ID does not exist"
Sep 08 13:35:53.819070 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:35:53.819045    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 08 13:35:53.822363 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:35:53.822341    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 08 13:35:54.177514 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:35:54.177460    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2eaeffd2-8f57-4a16-98a5-8f1765e368d3" path="/var/lib/kubelet/pods/2eaeffd2-8f57-4a16-98a5-8f1765e368d3/volumes"
Sep 08 13:35:58.117958 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:35:58.117909    2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-cfc4f4c99-89f9s" podUID="e52dc73e-8573-4cdc-a3f1-6be624510265" containerName="main" probeResult="failure" output="Get \"https://10.133.0.23:8000/health\": dial tcp 10.133.0.23:8000: connect: connection refused"
Sep 08 13:36:08.118676 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:36:08.118623    2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-cfc4f4c99-89f9s" podUID="e52dc73e-8573-4cdc-a3f1-6be624510265" containerName="main" probeResult="failure" output="Get \"https://10.133.0.23:8000/health\": dial tcp 10.133.0.23:8000: connect: connection refused"
Sep 08 13:36:11.043912 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:36:11.043865    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:36:11.139418 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:36:11.139361    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:36:15.376217 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:36:15.376176    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:36:15.664204 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:36:15.664068    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:36:18.117952 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:36:18.117909    2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-cfc4f4c99-89f9s" podUID="e52dc73e-8573-4cdc-a3f1-6be624510265" containerName="main" probeResult="failure" output="Get \"https://10.133.0.23:8000/health\": dial tcp 10.133.0.23:8000: connect: connection refused"
Sep 08 13:36:19.384176 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:36:19.384139    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:36:24.461730 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:36:24.461673    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:36:26.338864 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:36:26.338827    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:36:26.440835 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:36:26.440798    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:36:28.118798 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:36:28.118758    2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-cfc4f4c99-89f9s" podUID="e52dc73e-8573-4cdc-a3f1-6be624510265" containerName="main" probeResult="failure" output="Get \"https://10.133.0.23:8000/health\": dial tcp 10.133.0.23:8000: connect: connection refused"
Sep 08 13:36:38.118799 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:36:38.118757    2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-cfc4f4c99-89f9s" podUID="e52dc73e-8573-4cdc-a3f1-6be624510265" containerName="main" probeResult="failure" output="Get \"https://10.133.0.23:8000/health\": dial tcp 10.133.0.23:8000: connect: connection refused"
Sep 08 13:36:48.119030 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:36:48.118918    2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-cfc4f4c99-89f9s" podUID="e52dc73e-8573-4cdc-a3f1-6be624510265" containerName="main" probeResult="failure" output="Get \"https://10.133.0.23:8000/health\": dial tcp 10.133.0.23:8000: connect: connection refused"
Sep 08 13:36:50.279870 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:36:50.279831    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:36:50.383115 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:36:50.383089    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:36:58.127635 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:36:58.127603    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-cfc4f4c99-89f9s"
Sep 08 13:36:58.135155 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:36:58.135130    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-cfc4f4c99-89f9s"
Sep 08 13:37:03.263336 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:37:03.263304    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-cfc4f4c99-89f9s"]
Sep 08 13:37:03.596728 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:37:03.596644    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:37:03.696975 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:37:03.696945    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:37:18.288466 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:37:18.288405    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-cfc4f4c99-89f9s" podUID="e52dc73e-8573-4cdc-a3f1-6be624510265" containerName="main" containerID="cri-o://eced02329eeb8e329e5ce9466a8d8e18d41eb1a89e378e96e33b368b342b9762" gracePeriod=45
Sep 08 13:37:19.032343 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:37:19.032318    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-cfc4f4c99-89f9s"
Sep 08 13:37:19.092340 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:37:19.092278    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e52dc73e-8573-4cdc-a3f1-6be624510265-tls-certs\") pod \"e52dc73e-8573-4cdc-a3f1-6be624510265\" (UID: \"e52dc73e-8573-4cdc-a3f1-6be624510265\") "
Sep 08 13:37:19.092340 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:37:19.092313    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fh4j7\" (UniqueName: \"kubernetes.io/projected/e52dc73e-8573-4cdc-a3f1-6be624510265-kube-api-access-fh4j7\") pod \"e52dc73e-8573-4cdc-a3f1-6be624510265\" (UID: \"e52dc73e-8573-4cdc-a3f1-6be624510265\") "
Sep 08 13:37:19.092538 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:37:19.092359    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e52dc73e-8573-4cdc-a3f1-6be624510265-home\") pod \"e52dc73e-8573-4cdc-a3f1-6be624510265\" (UID: \"e52dc73e-8573-4cdc-a3f1-6be624510265\") "
Sep 08 13:37:19.092538 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:37:19.092395    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e52dc73e-8573-4cdc-a3f1-6be624510265-kserve-provision-location\") pod \"e52dc73e-8573-4cdc-a3f1-6be624510265\" (UID: \"e52dc73e-8573-4cdc-a3f1-6be624510265\") "
Sep 08 13:37:19.092538 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:37:19.092426    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e52dc73e-8573-4cdc-a3f1-6be624510265-tmp-dir\") pod \"e52dc73e-8573-4cdc-a3f1-6be624510265\" (UID: \"e52dc73e-8573-4cdc-a3f1-6be624510265\") "
Sep 08 13:37:19.092702 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:37:19.092540    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e52dc73e-8573-4cdc-a3f1-6be624510265-cabundle-cert\") pod \"e52dc73e-8573-4cdc-a3f1-6be624510265\" (UID: \"e52dc73e-8573-4cdc-a3f1-6be624510265\") "
Sep 08 13:37:19.092702 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:37:19.092590    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e52dc73e-8573-4cdc-a3f1-6be624510265-dshm\") pod \"e52dc73e-8573-4cdc-a3f1-6be624510265\" (UID: \"e52dc73e-8573-4cdc-a3f1-6be624510265\") "
Sep 08 13:37:19.092702 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:37:19.092627    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e52dc73e-8573-4cdc-a3f1-6be624510265-model-cache\") pod \"e52dc73e-8573-4cdc-a3f1-6be624510265\" (UID: \"e52dc73e-8573-4cdc-a3f1-6be624510265\") "
Sep 08 13:37:19.093111 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:37:19.093087    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e52dc73e-8573-4cdc-a3f1-6be624510265-model-cache" (OuterVolumeSpecName: "model-cache") pod "e52dc73e-8573-4cdc-a3f1-6be624510265" (UID: "e52dc73e-8573-4cdc-a3f1-6be624510265"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:37:19.093409 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:37:19.093190    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e52dc73e-8573-4cdc-a3f1-6be624510265-home" (OuterVolumeSpecName: "home") pod "e52dc73e-8573-4cdc-a3f1-6be624510265" (UID: "e52dc73e-8573-4cdc-a3f1-6be624510265"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:37:19.093409 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:37:19.093215    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e52dc73e-8573-4cdc-a3f1-6be624510265-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e52dc73e-8573-4cdc-a3f1-6be624510265" (UID: "e52dc73e-8573-4cdc-a3f1-6be624510265"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:37:19.094777 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:37:19.094744    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e52dc73e-8573-4cdc-a3f1-6be624510265-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e52dc73e-8573-4cdc-a3f1-6be624510265" (UID: "e52dc73e-8573-4cdc-a3f1-6be624510265"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:37:19.094874 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:37:19.094795    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e52dc73e-8573-4cdc-a3f1-6be624510265-dshm" (OuterVolumeSpecName: "dshm") pod "e52dc73e-8573-4cdc-a3f1-6be624510265" (UID: "e52dc73e-8573-4cdc-a3f1-6be624510265"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:37:19.094980 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:37:19.094959    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e52dc73e-8573-4cdc-a3f1-6be624510265-kube-api-access-fh4j7" (OuterVolumeSpecName: "kube-api-access-fh4j7") pod "e52dc73e-8573-4cdc-a3f1-6be624510265" (UID: "e52dc73e-8573-4cdc-a3f1-6be624510265"). InnerVolumeSpecName "kube-api-access-fh4j7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:37:19.104981 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:37:19.104954    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e52dc73e-8573-4cdc-a3f1-6be624510265-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e52dc73e-8573-4cdc-a3f1-6be624510265" (UID: "e52dc73e-8573-4cdc-a3f1-6be624510265"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:37:19.112065 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:37:19.112043    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e52dc73e-8573-4cdc-a3f1-6be624510265-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e52dc73e-8573-4cdc-a3f1-6be624510265" (UID: "e52dc73e-8573-4cdc-a3f1-6be624510265"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:37:19.178677 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:37:19.178644    2573 generic.go:356] "Generic (PLEG): container finished" podID="e52dc73e-8573-4cdc-a3f1-6be624510265" containerID="eced02329eeb8e329e5ce9466a8d8e18d41eb1a89e378e96e33b368b342b9762" exitCode=0
Sep 08 13:37:19.178822 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:37:19.178699    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-cfc4f4c99-89f9s"
Sep 08 13:37:19.178822 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:37:19.178724    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-cfc4f4c99-89f9s" event={"ID":"e52dc73e-8573-4cdc-a3f1-6be624510265","Type":"ContainerDied","Data":"eced02329eeb8e329e5ce9466a8d8e18d41eb1a89e378e96e33b368b342b9762"}
Sep 08 13:37:19.178822 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:37:19.178770    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-cfc4f4c99-89f9s" event={"ID":"e52dc73e-8573-4cdc-a3f1-6be624510265","Type":"ContainerDied","Data":"abeb44b45b3790a0b1756f75808197f35cf914b83ac0ba1d961e72d5db6366af"}
Sep 08 13:37:19.178822 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:37:19.178792    2573 scope.go:117] "RemoveContainer" containerID="eced02329eeb8e329e5ce9466a8d8e18d41eb1a89e378e96e33b368b342b9762"
Sep 08 13:37:19.189910 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:37:19.189857    2573 scope.go:117] "RemoveContainer" containerID="c67723530f399d821c9423cb5fc9ba979e4701837b5440caafbc2c4e17e4ad61"
Sep 08 13:37:19.193301 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:37:19.193279    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e52dc73e-8573-4cdc-a3f1-6be624510265-tls-certs\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 13:37:19.193301 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:37:19.193299    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fh4j7\" (UniqueName: \"kubernetes.io/projected/e52dc73e-8573-4cdc-a3f1-6be624510265-kube-api-access-fh4j7\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 13:37:19.193450 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:37:19.193309    2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e52dc73e-8573-4cdc-a3f1-6be624510265-home\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 13:37:19.193450 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:37:19.193320    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e52dc73e-8573-4cdc-a3f1-6be624510265-kserve-provision-location\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 13:37:19.193450 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:37:19.193329    2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e52dc73e-8573-4cdc-a3f1-6be624510265-tmp-dir\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 13:37:19.193450 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:37:19.193337    2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e52dc73e-8573-4cdc-a3f1-6be624510265-cabundle-cert\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 13:37:19.193450 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:37:19.193344    2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e52dc73e-8573-4cdc-a3f1-6be624510265-dshm\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 13:37:19.193450 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:37:19.193351    2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e52dc73e-8573-4cdc-a3f1-6be624510265-model-cache\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 13:37:19.197382 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:37:19.197367    2573 scope.go:117] "RemoveContainer" containerID="eced02329eeb8e329e5ce9466a8d8e18d41eb1a89e378e96e33b368b342b9762"
Sep 08 13:37:19.197721 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:37:19.197699    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eced02329eeb8e329e5ce9466a8d8e18d41eb1a89e378e96e33b368b342b9762\": container with ID starting with eced02329eeb8e329e5ce9466a8d8e18d41eb1a89e378e96e33b368b342b9762 not found: ID does not exist" containerID="eced02329eeb8e329e5ce9466a8d8e18d41eb1a89e378e96e33b368b342b9762"
Sep 08 13:37:19.197774 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:37:19.197730    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eced02329eeb8e329e5ce9466a8d8e18d41eb1a89e378e96e33b368b342b9762"} err="failed to get container status \"eced02329eeb8e329e5ce9466a8d8e18d41eb1a89e378e96e33b368b342b9762\": rpc error: code = NotFound desc = could not find container \"eced02329eeb8e329e5ce9466a8d8e18d41eb1a89e378e96e33b368b342b9762\": container with ID starting with eced02329eeb8e329e5ce9466a8d8e18d41eb1a89e378e96e33b368b342b9762 not found: ID does not exist"
Sep 08 13:37:19.197774 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:37:19.197747    2573 scope.go:117] "RemoveContainer" containerID="c67723530f399d821c9423cb5fc9ba979e4701837b5440caafbc2c4e17e4ad61"
Sep 08 13:37:19.198013 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:37:19.197999    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c67723530f399d821c9423cb5fc9ba979e4701837b5440caafbc2c4e17e4ad61\": container with ID starting with c67723530f399d821c9423cb5fc9ba979e4701837b5440caafbc2c4e17e4ad61 not found: ID does not exist" containerID="c67723530f399d821c9423cb5fc9ba979e4701837b5440caafbc2c4e17e4ad61"
Sep 08 13:37:19.198059 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:37:19.198016    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c67723530f399d821c9423cb5fc9ba979e4701837b5440caafbc2c4e17e4ad61"} err="failed to get container status \"c67723530f399d821c9423cb5fc9ba979e4701837b5440caafbc2c4e17e4ad61\": rpc error: code = NotFound desc = could not find container \"c67723530f399d821c9423cb5fc9ba979e4701837b5440caafbc2c4e17e4ad61\": container with ID starting with c67723530f399d821c9423cb5fc9ba979e4701837b5440caafbc2c4e17e4ad61 not found: ID does not exist"
Sep 08 13:37:19.203423 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:37:19.203402    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-cfc4f4c99-89f9s"]
Sep 08 13:37:19.207928 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:37:19.207890    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-cfc4f4c99-89f9s"]
Sep 08 13:37:20.174873 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:37:20.174841    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e52dc73e-8573-4cdc-a3f1-6be624510265" path="/var/lib/kubelet/pods/e52dc73e-8573-4cdc-a3f1-6be624510265/volumes"
Sep 08 13:37:39.825514 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:37:39.825467    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:37:39.921270 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:37:39.921232    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:37:46.178374 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:37:46.178338    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:37:52.989438 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:37:52.989396    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:37:53.055184 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:37:53.055156    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:37:53.396972 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:37:53.396943    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:37:53.492564 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:37:53.492494    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:37:59.864771 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:37:59.864702    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:38:01.507436 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:38:01.507403    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:38:01.691666 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:38:01.691633    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:38:05.963972 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:38:05.963918    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:38:06.064670 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:38:06.064625    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:38:07.402191 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:38:07.402134    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:38:13.828006 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:38:13.827919    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7fb4b457f8-dmn2d"]
Sep 08 13:38:13.828355 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:38:13.828211    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e52dc73e-8573-4cdc-a3f1-6be624510265" containerName="main"
Sep 08 13:38:13.828355 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:38:13.828222    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="e52dc73e-8573-4cdc-a3f1-6be624510265" containerName="main"
Sep 08 13:38:13.828355 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:38:13.828235    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e52dc73e-8573-4cdc-a3f1-6be624510265" containerName="storage-initializer"
Sep 08 13:38:13.828355 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:38:13.828241    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="e52dc73e-8573-4cdc-a3f1-6be624510265" containerName="storage-initializer"
Sep 08 13:38:13.828355 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:38:13.828250    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2eaeffd2-8f57-4a16-98a5-8f1765e368d3" containerName="main"
Sep 08 13:38:13.828355 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:38:13.828256    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="2eaeffd2-8f57-4a16-98a5-8f1765e368d3" containerName="main"
Sep 08 13:38:13.828355 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:38:13.828264    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2eaeffd2-8f57-4a16-98a5-8f1765e368d3" containerName="storage-initializer"
Sep 08 13:38:13.828355 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:38:13.828268    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="2eaeffd2-8f57-4a16-98a5-8f1765e368d3" containerName="storage-initializer"
Sep 08 13:38:13.828355 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:38:13.828314    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="e52dc73e-8573-4cdc-a3f1-6be624510265" containerName="main"
Sep 08 13:38:13.828355 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:38:13.828323    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="2eaeffd2-8f57-4a16-98a5-8f1765e368d3" containerName="main"
Sep 08 13:38:13.832294 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:38:13.832278    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7fb4b457f8-dmn2d"
Sep 08 13:38:13.834795 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:38:13.834763    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"seaweedfs-s3-creds\""
Sep 08 13:38:13.834966 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:38:13.834941    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"kube-root-ca.crt\""
Sep 08 13:38:13.835088 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:38:13.834904    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"default-dockercfg-hstrf\""
Sep 08 13:38:13.835088 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:38:13.835015    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"auth-invalid-token-test-kserve-self-signed-certs\""
Sep 08 13:38:13.835088 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:38:13.835028    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"openshift-service-ca.crt\""
Sep 08 13:38:13.835088 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:38:13.834897    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"odh-kserve-custom-ca-bundle\""
Sep 08 13:38:13.848207 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:38:13.848185    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7fb4b457f8-dmn2d"]
Sep 08 13:38:13.907967 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:38:13.907934    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/083b3118-64d1-4b54-a675-a664bb7df9dd-dshm\") pod \"auth-invalid-token-test-kserve-7fb4b457f8-dmn2d\" (UID: \"083b3118-64d1-4b54-a675-a664bb7df9dd\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7fb4b457f8-dmn2d"
Sep 08 13:38:13.908129 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:38:13.907973    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/083b3118-64d1-4b54-a675-a664bb7df9dd-tmp-dir\") pod \"auth-invalid-token-test-kserve-7fb4b457f8-dmn2d\" (UID: \"083b3118-64d1-4b54-a675-a664bb7df9dd\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7fb4b457f8-dmn2d"
Sep 08 13:38:13.908129 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:38:13.907994    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjbnk\" (UniqueName: \"kubernetes.io/projected/083b3118-64d1-4b54-a675-a664bb7df9dd-kube-api-access-jjbnk\") pod \"auth-invalid-token-test-kserve-7fb4b457f8-dmn2d\" (UID: \"083b3118-64d1-4b54-a675-a664bb7df9dd\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7fb4b457f8-dmn2d"
Sep 08 13:38:13.908129 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:38:13.908033    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/083b3118-64d1-4b54-a675-a664bb7df9dd-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-7fb4b457f8-dmn2d\" (UID: \"083b3118-64d1-4b54-a675-a664bb7df9dd\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7fb4b457f8-dmn2d"
Sep 08 13:38:13.908129 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:38:13.908076    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/083b3118-64d1-4b54-a675-a664bb7df9dd-model-cache\") pod \"auth-invalid-token-test-kserve-7fb4b457f8-dmn2d\" (UID: \"083b3118-64d1-4b54-a675-a664bb7df9dd\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7fb4b457f8-dmn2d"
Sep 08 13:38:13.908129 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:38:13.908093    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/083b3118-64d1-4b54-a675-a664bb7df9dd-cabundle-cert\") pod \"auth-invalid-token-test-kserve-7fb4b457f8-dmn2d\" (UID: \"083b3118-64d1-4b54-a675-a664bb7df9dd\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7fb4b457f8-dmn2d"
Sep 08 13:38:13.908360 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:38:13.908141    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/083b3118-64d1-4b54-a675-a664bb7df9dd-tls-certs\") pod \"auth-invalid-token-test-kserve-7fb4b457f8-dmn2d\" (UID: \"083b3118-64d1-4b54-a675-a664bb7df9dd\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7fb4b457f8-dmn2d"
Sep 08 13:38:13.908360 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:38:13.908168    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/083b3118-64d1-4b54-a675-a664bb7df9dd-home\") pod \"auth-invalid-token-test-kserve-7fb4b457f8-dmn2d\" (UID: \"083b3118-64d1-4b54-a675-a664bb7df9dd\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7fb4b457f8-dmn2d"
Sep 08 13:38:13.921811 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:38:13.921783    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:38:14.009339 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:38:14.009297    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/083b3118-64d1-4b54-a675-a664bb7df9dd-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-7fb4b457f8-dmn2d\" (UID: \"083b3118-64d1-4b54-a675-a664bb7df9dd\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7fb4b457f8-dmn2d"
Sep 08 13:38:14.009550 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:38:14.009356    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/083b3118-64d1-4b54-a675-a664bb7df9dd-model-cache\") pod \"auth-invalid-token-test-kserve-7fb4b457f8-dmn2d\" (UID: \"083b3118-64d1-4b54-a675-a664bb7df9dd\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7fb4b457f8-dmn2d"
Sep 08 13:38:14.009550 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:38:14.009377    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/083b3118-64d1-4b54-a675-a664bb7df9dd-cabundle-cert\") pod \"auth-invalid-token-test-kserve-7fb4b457f8-dmn2d\" (UID: \"083b3118-64d1-4b54-a675-a664bb7df9dd\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7fb4b457f8-dmn2d"
Sep 08 13:38:14.009550 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:38:14.009397    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/083b3118-64d1-4b54-a675-a664bb7df9dd-tls-certs\") pod \"auth-invalid-token-test-kserve-7fb4b457f8-dmn2d\" (UID: \"083b3118-64d1-4b54-a675-a664bb7df9dd\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7fb4b457f8-dmn2d"
Sep 08 13:38:14.009550 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:38:14.009415    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/083b3118-64d1-4b54-a675-a664bb7df9dd-home\") pod \"auth-invalid-token-test-kserve-7fb4b457f8-dmn2d\" (UID: \"083b3118-64d1-4b54-a675-a664bb7df9dd\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7fb4b457f8-dmn2d"
Sep 08 13:38:14.009785 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:38:14.009585    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/083b3118-64d1-4b54-a675-a664bb7df9dd-dshm\") pod \"auth-invalid-token-test-kserve-7fb4b457f8-dmn2d\" (UID: \"083b3118-64d1-4b54-a675-a664bb7df9dd\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7fb4b457f8-dmn2d"
Sep 08 13:38:14.009785 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:38:14.009633    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/083b3118-64d1-4b54-a675-a664bb7df9dd-tmp-dir\") pod \"auth-invalid-token-test-kserve-7fb4b457f8-dmn2d\" (UID: \"083b3118-64d1-4b54-a675-a664bb7df9dd\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7fb4b457f8-dmn2d"
Sep 08 13:38:14.009785 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:38:14.009675    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjbnk\" (UniqueName: \"kubernetes.io/projected/083b3118-64d1-4b54-a675-a664bb7df9dd-kube-api-access-jjbnk\") pod \"auth-invalid-token-test-kserve-7fb4b457f8-dmn2d\" (UID: \"083b3118-64d1-4b54-a675-a664bb7df9dd\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7fb4b457f8-dmn2d"
Sep 08 13:38:14.009785 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:38:14.009772    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/083b3118-64d1-4b54-a675-a664bb7df9dd-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-7fb4b457f8-dmn2d\" (UID: \"083b3118-64d1-4b54-a675-a664bb7df9dd\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7fb4b457f8-dmn2d"
Sep 08 13:38:14.009973 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:38:14.009848    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/083b3118-64d1-4b54-a675-a664bb7df9dd-home\") pod \"auth-invalid-token-test-kserve-7fb4b457f8-dmn2d\" (UID: \"083b3118-64d1-4b54-a675-a664bb7df9dd\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7fb4b457f8-dmn2d"
Sep 08 13:38:14.009973 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:38:14.009881    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/083b3118-64d1-4b54-a675-a664bb7df9dd-model-cache\") pod \"auth-invalid-token-test-kserve-7fb4b457f8-dmn2d\" (UID: \"083b3118-64d1-4b54-a675-a664bb7df9dd\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7fb4b457f8-dmn2d"
Sep 08 13:38:14.010094 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:38:14.010072    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/083b3118-64d1-4b54-a675-a664bb7df9dd-tmp-dir\") pod \"auth-invalid-token-test-kserve-7fb4b457f8-dmn2d\" (UID: \"083b3118-64d1-4b54-a675-a664bb7df9dd\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7fb4b457f8-dmn2d"
Sep 08 13:38:14.010148 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:38:14.010122    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/083b3118-64d1-4b54-a675-a664bb7df9dd-cabundle-cert\") pod \"auth-invalid-token-test-kserve-7fb4b457f8-dmn2d\" (UID: \"083b3118-64d1-4b54-a675-a664bb7df9dd\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7fb4b457f8-dmn2d"
Sep 08 13:38:14.011831 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:38:14.011808    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/083b3118-64d1-4b54-a675-a664bb7df9dd-dshm\") pod \"auth-invalid-token-test-kserve-7fb4b457f8-dmn2d\" (UID: \"083b3118-64d1-4b54-a675-a664bb7df9dd\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7fb4b457f8-dmn2d"
Sep 08 13:38:14.011974 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:38:14.011957    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/083b3118-64d1-4b54-a675-a664bb7df9dd-tls-certs\") pod \"auth-invalid-token-test-kserve-7fb4b457f8-dmn2d\" (UID: \"083b3118-64d1-4b54-a675-a664bb7df9dd\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7fb4b457f8-dmn2d"
Sep 08 13:38:14.017854 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:38:14.017830    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjbnk\" (UniqueName: \"kubernetes.io/projected/083b3118-64d1-4b54-a675-a664bb7df9dd-kube-api-access-jjbnk\") pod \"auth-invalid-token-test-kserve-7fb4b457f8-dmn2d\" (UID: \"083b3118-64d1-4b54-a675-a664bb7df9dd\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7fb4b457f8-dmn2d"
Sep 08 13:38:14.142545 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:38:14.142494    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7fb4b457f8-dmn2d"
Sep 08 13:38:14.184490 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:38:14.184434    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:38:14.417778 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:38:14.417706    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7fb4b457f8-dmn2d"]
Sep 08 13:38:14.420612 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:38:14.420571    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod083b3118_64d1_4b54_a675_a664bb7df9dd.slice/crio-31366333f2eca6c092d60aa1ebc9514a7ce477c01fd5bfcd74bc7d193fb52249 WatchSource:0}: Error finding container 31366333f2eca6c092d60aa1ebc9514a7ce477c01fd5bfcd74bc7d193fb52249: Status 404 returned error can't find the container with id 31366333f2eca6c092d60aa1ebc9514a7ce477c01fd5bfcd74bc7d193fb52249
Sep 08 13:38:15.398808 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:38:15.398768    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7fb4b457f8-dmn2d" event={"ID":"083b3118-64d1-4b54-a675-a664bb7df9dd","Type":"ContainerStarted","Data":"a9da6f86d2f9cf0f4df5ff4aea65f896b8ed8b0bb6d72a72accb8ce9b4c0bf69"}
Sep 08 13:38:15.399211 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:38:15.398823    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7fb4b457f8-dmn2d" event={"ID":"083b3118-64d1-4b54-a675-a664bb7df9dd","Type":"ContainerStarted","Data":"31366333f2eca6c092d60aa1ebc9514a7ce477c01fd5bfcd74bc7d193fb52249"}
Sep 08 13:38:17.408680 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:38:17.408641    2573 generic.go:356] "Generic (PLEG): container finished" podID="083b3118-64d1-4b54-a675-a664bb7df9dd" containerID="a9da6f86d2f9cf0f4df5ff4aea65f896b8ed8b0bb6d72a72accb8ce9b4c0bf69" exitCode=0
Sep 08 13:38:17.409139 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:38:17.408698    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7fb4b457f8-dmn2d" event={"ID":"083b3118-64d1-4b54-a675-a664bb7df9dd","Type":"ContainerDied","Data":"a9da6f86d2f9cf0f4df5ff4aea65f896b8ed8b0bb6d72a72accb8ce9b4c0bf69"}
Sep 08 13:38:18.414445 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:38:18.414405    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7fb4b457f8-dmn2d" event={"ID":"083b3118-64d1-4b54-a675-a664bb7df9dd","Type":"ContainerStarted","Data":"9d1e5e29be9650288a9abbdb38d6a094bf49424717295e5a66f465d9f43dc493"}
Sep 08 13:38:18.439093 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:38:18.439041    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7fb4b457f8-dmn2d" podStartSLOduration=3.358738457 podStartE2EDuration="5.439025986s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.080287529s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:38:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:38:18.437624662 +0000 UTC m=+1348.709072614" watchObservedRunningTime="2026-09-08 13:38:18.439025986 +0000 UTC m=+1348.710473917"
Sep 08 13:38:19.143422 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:38:19.143382    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7fb4b457f8-dmn2d"
Sep 08 13:38:24.142946 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:38:24.142910    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7fb4b457f8-dmn2d"
Sep 08 13:38:24.144225 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:38:24.144196    2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7fb4b457f8-dmn2d" podUID="083b3118-64d1-4b54-a675-a664bb7df9dd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.24:8000/health\": dial tcp 10.133.0.24:8000: connect: connection refused"
Sep 08 13:38:24.793582 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:38:24.793543    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:38:28.552386 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:38:28.552335    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:38:34.143075 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:38:34.143022    2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7fb4b457f8-dmn2d" podUID="083b3118-64d1-4b54-a675-a664bb7df9dd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.24:8000/health\": dial tcp 10.133.0.24:8000: connect: connection refused"
Sep 08 13:38:35.078957 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:38:35.078917    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:38:35.176135 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:38:35.176104    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:38:44.142902 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:38:44.142861    2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7fb4b457f8-dmn2d" podUID="083b3118-64d1-4b54-a675-a664bb7df9dd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.24:8000/health\": dial tcp 10.133.0.24:8000: connect: connection refused"
Sep 08 13:38:53.012080 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:38:53.012041    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:38:54.143337 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:38:54.143293    2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7fb4b457f8-dmn2d" podUID="083b3118-64d1-4b54-a675-a664bb7df9dd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.24:8000/health\": dial tcp 10.133.0.24:8000: connect: connection refused"
Sep 08 13:39:04.143222 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:39:04.143181    2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7fb4b457f8-dmn2d" podUID="083b3118-64d1-4b54-a675-a664bb7df9dd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.24:8000/health\": dial tcp 10.133.0.24:8000: connect: connection refused"
Sep 08 13:39:14.143274 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:39:14.143233    2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7fb4b457f8-dmn2d" podUID="083b3118-64d1-4b54-a675-a664bb7df9dd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.24:8000/health\": dial tcp 10.133.0.24:8000: connect: connection refused"
Sep 08 13:39:24.143402 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:39:24.143357    2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7fb4b457f8-dmn2d" podUID="083b3118-64d1-4b54-a675-a664bb7df9dd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.24:8000/health\": dial tcp 10.133.0.24:8000: connect: connection refused"
Sep 08 13:39:25.006571 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:39:25.006533    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:39:26.400984 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:39:26.400952    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:39:31.812236 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:39:31.812206    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:39:32.041207 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:39:32.041172    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:39:32.286691 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:39:32.286653    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:39:34.143272 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:39:34.143229    2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7fb4b457f8-dmn2d" podUID="083b3118-64d1-4b54-a675-a664bb7df9dd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.24:8000/health\": dial tcp 10.133.0.24:8000: connect: connection refused"
Sep 08 13:39:38.908720 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:39:38.908659    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:39:39.008823 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:39:39.008796    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:39:44.142966 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:39:44.142878    2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7fb4b457f8-dmn2d" podUID="083b3118-64d1-4b54-a675-a664bb7df9dd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.24:8000/health\": dial tcp 10.133.0.24:8000: connect: connection refused"
Sep 08 13:39:51.264024 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:39:51.263926    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:39:54.143252 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:39:54.143203    2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7fb4b457f8-dmn2d" podUID="083b3118-64d1-4b54-a675-a664bb7df9dd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.24:8000/health\": dial tcp 10.133.0.24:8000: connect: connection refused"
Sep 08 13:40:04.143927 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:40:04.143877    2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7fb4b457f8-dmn2d" podUID="083b3118-64d1-4b54-a675-a664bb7df9dd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.24:8000/health\": dial tcp 10.133.0.24:8000: connect: connection refused"
Sep 08 13:40:05.718285 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:40:05.718251    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:40:05.830932 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:40:05.830898    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:40:14.153484 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:40:14.153455    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7fb4b457f8-dmn2d"
Sep 08 13:40:14.161334 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:40:14.161308    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7fb4b457f8-dmn2d"
Sep 08 13:40:15.899341 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:40:15.899308    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:40:15.906380 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:40:15.906352    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7fb4b457f8-dmn2d"]
Sep 08 13:40:15.993873 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:40:15.993840    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:40:30.929138 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:40:30.929092    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7fb4b457f8-dmn2d" podUID="083b3118-64d1-4b54-a675-a664bb7df9dd" containerName="main" containerID="cri-o://9d1e5e29be9650288a9abbdb38d6a094bf49424717295e5a66f465d9f43dc493" gracePeriod=45
Sep 08 13:40:31.154820 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:40:31.154776    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7fb4b457f8-dmn2d" podUID="083b3118-64d1-4b54-a675-a664bb7df9dd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.24:8000/health\": dial tcp 10.133.0.24:8000: connect: connection refused"
Sep 08 13:40:31.671806 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:40:31.671777    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7fb4b457f8-dmn2d"
Sep 08 13:40:31.866056 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:40:31.865970    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/083b3118-64d1-4b54-a675-a664bb7df9dd-home\") pod \"083b3118-64d1-4b54-a675-a664bb7df9dd\" (UID: \"083b3118-64d1-4b54-a675-a664bb7df9dd\") "
Sep 08 13:40:31.866056 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:40:31.866014    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/083b3118-64d1-4b54-a675-a664bb7df9dd-model-cache\") pod \"083b3118-64d1-4b54-a675-a664bb7df9dd\" (UID: \"083b3118-64d1-4b54-a675-a664bb7df9dd\") "
Sep 08 13:40:31.866056 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:40:31.866031    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/083b3118-64d1-4b54-a675-a664bb7df9dd-dshm\") pod \"083b3118-64d1-4b54-a675-a664bb7df9dd\" (UID: \"083b3118-64d1-4b54-a675-a664bb7df9dd\") "
Sep 08 13:40:31.866056 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:40:31.866060    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/083b3118-64d1-4b54-a675-a664bb7df9dd-tmp-dir\") pod \"083b3118-64d1-4b54-a675-a664bb7df9dd\" (UID: \"083b3118-64d1-4b54-a675-a664bb7df9dd\") "
Sep 08 13:40:31.866390 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:40:31.866075    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/083b3118-64d1-4b54-a675-a664bb7df9dd-cabundle-cert\") pod \"083b3118-64d1-4b54-a675-a664bb7df9dd\" (UID: \"083b3118-64d1-4b54-a675-a664bb7df9dd\") "
Sep 08 13:40:31.866390 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:40:31.866097    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/083b3118-64d1-4b54-a675-a664bb7df9dd-tls-certs\") pod \"083b3118-64d1-4b54-a675-a664bb7df9dd\" (UID: \"083b3118-64d1-4b54-a675-a664bb7df9dd\") "
Sep 08 13:40:31.866390 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:40:31.866147    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jjbnk\" (UniqueName: \"kubernetes.io/projected/083b3118-64d1-4b54-a675-a664bb7df9dd-kube-api-access-jjbnk\") pod \"083b3118-64d1-4b54-a675-a664bb7df9dd\" (UID: \"083b3118-64d1-4b54-a675-a664bb7df9dd\") "
Sep 08 13:40:31.866390 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:40:31.866171    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/083b3118-64d1-4b54-a675-a664bb7df9dd-kserve-provision-location\") pod \"083b3118-64d1-4b54-a675-a664bb7df9dd\" (UID: \"083b3118-64d1-4b54-a675-a664bb7df9dd\") "
Sep 08 13:40:31.866390 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:40:31.866345    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/083b3118-64d1-4b54-a675-a664bb7df9dd-model-cache" (OuterVolumeSpecName: "model-cache") pod "083b3118-64d1-4b54-a675-a664bb7df9dd" (UID: "083b3118-64d1-4b54-a675-a664bb7df9dd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:40:31.866658 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:40:31.866453    2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/083b3118-64d1-4b54-a675-a664bb7df9dd-model-cache\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 13:40:31.866658 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:40:31.866527    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/083b3118-64d1-4b54-a675-a664bb7df9dd-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "083b3118-64d1-4b54-a675-a664bb7df9dd" (UID: "083b3118-64d1-4b54-a675-a664bb7df9dd"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:40:31.866861 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:40:31.866842    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/083b3118-64d1-4b54-a675-a664bb7df9dd-home" (OuterVolumeSpecName: "home") pod "083b3118-64d1-4b54-a675-a664bb7df9dd" (UID: "083b3118-64d1-4b54-a675-a664bb7df9dd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:40:31.868795 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:40:31.868753    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/083b3118-64d1-4b54-a675-a664bb7df9dd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "083b3118-64d1-4b54-a675-a664bb7df9dd" (UID: "083b3118-64d1-4b54-a675-a664bb7df9dd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:40:31.868901 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:40:31.868806    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/083b3118-64d1-4b54-a675-a664bb7df9dd-dshm" (OuterVolumeSpecName: "dshm") pod "083b3118-64d1-4b54-a675-a664bb7df9dd" (UID: "083b3118-64d1-4b54-a675-a664bb7df9dd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:40:31.868901 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:40:31.868862    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/083b3118-64d1-4b54-a675-a664bb7df9dd-kube-api-access-jjbnk" (OuterVolumeSpecName: "kube-api-access-jjbnk") pod "083b3118-64d1-4b54-a675-a664bb7df9dd" (UID: "083b3118-64d1-4b54-a675-a664bb7df9dd"). InnerVolumeSpecName "kube-api-access-jjbnk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:40:31.879090 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:40:31.879062    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/083b3118-64d1-4b54-a675-a664bb7df9dd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "083b3118-64d1-4b54-a675-a664bb7df9dd" (UID: "083b3118-64d1-4b54-a675-a664bb7df9dd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:40:31.895101 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:40:31.895070    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/083b3118-64d1-4b54-a675-a664bb7df9dd-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "083b3118-64d1-4b54-a675-a664bb7df9dd" (UID: "083b3118-64d1-4b54-a675-a664bb7df9dd"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:40:31.967680 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:40:31.967655    2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/083b3118-64d1-4b54-a675-a664bb7df9dd-home\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 13:40:31.967680 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:40:31.967675    2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/083b3118-64d1-4b54-a675-a664bb7df9dd-dshm\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 13:40:31.968118 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:40:31.967687    2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/083b3118-64d1-4b54-a675-a664bb7df9dd-tmp-dir\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 13:40:31.968118 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:40:31.967703    2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/083b3118-64d1-4b54-a675-a664bb7df9dd-cabundle-cert\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 13:40:31.968118 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:40:31.967718    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/083b3118-64d1-4b54-a675-a664bb7df9dd-tls-certs\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 13:40:31.968118 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:40:31.967732    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjbnk\" (UniqueName: \"kubernetes.io/projected/083b3118-64d1-4b54-a675-a664bb7df9dd-kube-api-access-jjbnk\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 13:40:31.968118 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:40:31.967745    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/083b3118-64d1-4b54-a675-a664bb7df9dd-kserve-provision-location\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 13:40:31.968998 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:40:31.968971    2573 generic.go:356] "Generic (PLEG): container finished" podID="083b3118-64d1-4b54-a675-a664bb7df9dd" containerID="9d1e5e29be9650288a9abbdb38d6a094bf49424717295e5a66f465d9f43dc493" exitCode=0
Sep 08 13:40:31.969064 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:40:31.969031    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7fb4b457f8-dmn2d"
Sep 08 13:40:31.969110 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:40:31.969060    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7fb4b457f8-dmn2d" event={"ID":"083b3118-64d1-4b54-a675-a664bb7df9dd","Type":"ContainerDied","Data":"9d1e5e29be9650288a9abbdb38d6a094bf49424717295e5a66f465d9f43dc493"}
Sep 08 13:40:31.969144 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:40:31.969133    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7fb4b457f8-dmn2d" event={"ID":"083b3118-64d1-4b54-a675-a664bb7df9dd","Type":"ContainerDied","Data":"31366333f2eca6c092d60aa1ebc9514a7ce477c01fd5bfcd74bc7d193fb52249"}
Sep 08 13:40:31.969177 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:40:31.969154    2573 scope.go:117] "RemoveContainer" containerID="9d1e5e29be9650288a9abbdb38d6a094bf49424717295e5a66f465d9f43dc493"
Sep 08 13:40:31.977393 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:40:31.977377    2573 scope.go:117] "RemoveContainer" containerID="a9da6f86d2f9cf0f4df5ff4aea65f896b8ed8b0bb6d72a72accb8ce9b4c0bf69"
Sep 08 13:40:31.984690 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:40:31.984675    2573 scope.go:117] "RemoveContainer" containerID="9d1e5e29be9650288a9abbdb38d6a094bf49424717295e5a66f465d9f43dc493"
Sep 08 13:40:31.984947 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:40:31.984928    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d1e5e29be9650288a9abbdb38d6a094bf49424717295e5a66f465d9f43dc493\": container with ID starting with 9d1e5e29be9650288a9abbdb38d6a094bf49424717295e5a66f465d9f43dc493 not found: ID does not exist" containerID="9d1e5e29be9650288a9abbdb38d6a094bf49424717295e5a66f465d9f43dc493"
Sep 08 13:40:31.985015 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:40:31.984958    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d1e5e29be9650288a9abbdb38d6a094bf49424717295e5a66f465d9f43dc493"} err="failed to get container status \"9d1e5e29be9650288a9abbdb38d6a094bf49424717295e5a66f465d9f43dc493\": rpc error: code = NotFound desc = could not find container \"9d1e5e29be9650288a9abbdb38d6a094bf49424717295e5a66f465d9f43dc493\": container with ID starting with 9d1e5e29be9650288a9abbdb38d6a094bf49424717295e5a66f465d9f43dc493 not found: ID does not exist"
Sep 08 13:40:31.985015 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:40:31.984982    2573 scope.go:117] "RemoveContainer" containerID="a9da6f86d2f9cf0f4df5ff4aea65f896b8ed8b0bb6d72a72accb8ce9b4c0bf69"
Sep 08 13:40:31.985225 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:40:31.985210    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9da6f86d2f9cf0f4df5ff4aea65f896b8ed8b0bb6d72a72accb8ce9b4c0bf69\": container with ID starting with a9da6f86d2f9cf0f4df5ff4aea65f896b8ed8b0bb6d72a72accb8ce9b4c0bf69 not found: ID does not exist" containerID="a9da6f86d2f9cf0f4df5ff4aea65f896b8ed8b0bb6d72a72accb8ce9b4c0bf69"
Sep 08 13:40:31.985269 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:40:31.985231    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9da6f86d2f9cf0f4df5ff4aea65f896b8ed8b0bb6d72a72accb8ce9b4c0bf69"} err="failed to get container status \"a9da6f86d2f9cf0f4df5ff4aea65f896b8ed8b0bb6d72a72accb8ce9b4c0bf69\": rpc error: code = NotFound desc = could not find container \"a9da6f86d2f9cf0f4df5ff4aea65f896b8ed8b0bb6d72a72accb8ce9b4c0bf69\": container with ID starting with a9da6f86d2f9cf0f4df5ff4aea65f896b8ed8b0bb6d72a72accb8ce9b4c0bf69 not found: ID does not exist"
Sep 08 13:40:31.994642 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:40:31.994620    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7fb4b457f8-dmn2d"]
Sep 08 13:40:32.001578 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:40:32.001558    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-7fb4b457f8-dmn2d"]
Sep 08 13:40:32.174162 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:40:32.174128    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="083b3118-64d1-4b54-a675-a664bb7df9dd" path="/var/lib/kubelet/pods/083b3118-64d1-4b54-a675-a664bb7df9dd/volumes"
Sep 08 13:40:33.206074 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:40:33.206040    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:40:35.736597 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:40:35.736566    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:40:50.221314 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:40:50.221281    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8h8cn_b936238e-bdf1-4dac-8bf8-6417c313597c/ovn-acl-logging/0.log"
Sep 08 13:40:50.225070 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:40:50.225049    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8h8cn_b936238e-bdf1-4dac-8bf8-6417c313597c/ovn-acl-logging/0.log"
Sep 08 13:40:55.455997 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:40:55.455944    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:40:55.646611 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:40:55.646579    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:41:01.734699 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:41:01.734665    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:41:01.969958 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:41:01.969922    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc56hm77"]
Sep 08 13:41:01.970237 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:41:01.970225    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="083b3118-64d1-4b54-a675-a664bb7df9dd" containerName="storage-initializer"
Sep 08 13:41:01.970282 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:41:01.970239    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="083b3118-64d1-4b54-a675-a664bb7df9dd" containerName="storage-initializer"
Sep 08 13:41:01.970282 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:41:01.970254    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="083b3118-64d1-4b54-a675-a664bb7df9dd" containerName="main"
Sep 08 13:41:01.970282 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:41:01.970259    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="083b3118-64d1-4b54-a675-a664bb7df9dd" containerName="main"
Sep 08 13:41:01.970376 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:41:01.970318    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="083b3118-64d1-4b54-a675-a664bb7df9dd" containerName="main"
Sep 08 13:41:01.974043 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:41:01.974027    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc56hm77"
Sep 08 13:41:01.979050 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:41:01.979021    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-kserve-self-signed-certs\""
Sep 08 13:41:01.979206 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:41:01.979030    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"openshift-service-ca.crt\""
Sep 08 13:41:01.979206 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:41:01.979062    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"kube-root-ca.crt\""
Sep 08 13:41:01.979206 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:41:01.979201    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"default-dockercfg-v7lbz\""
Sep 08 13:41:01.979704 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:41:01.979681    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-epp-sa-dockercfg-z8587\""
Sep 08 13:41:01.991609 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:41:01.991552    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc56hm77"]
Sep 08 13:41:02.011437 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:41:02.011415    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:41:02.069363 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:41:02.069332    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/283e3066-c906-451b-ab9c-a2197f756683-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc56hm77\" (UID: \"283e3066-c906-451b-ab9c-a2197f756683\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc56hm77"
Sep 08 13:41:02.069537 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:41:02.069397    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsm8g\" (UniqueName: \"kubernetes.io/projected/283e3066-c906-451b-ab9c-a2197f756683-kube-api-access-bsm8g\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc56hm77\" (UID: \"283e3066-c906-451b-ab9c-a2197f756683\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc56hm77"
Sep 08 13:41:02.170295 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:41:02.170260    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsm8g\" (UniqueName: \"kubernetes.io/projected/283e3066-c906-451b-ab9c-a2197f756683-kube-api-access-bsm8g\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc56hm77\" (UID: \"283e3066-c906-451b-ab9c-a2197f756683\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc56hm77"
Sep 08 13:41:02.170474 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:41:02.170323    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/283e3066-c906-451b-ab9c-a2197f756683-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc56hm77\" (UID: \"283e3066-c906-451b-ab9c-a2197f756683\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc56hm77"
Sep 08 13:41:02.173005 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:41:02.172976    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/283e3066-c906-451b-ab9c-a2197f756683-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc56hm77\" (UID: \"283e3066-c906-451b-ab9c-a2197f756683\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc56hm77"
Sep 08 13:41:02.181025 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:41:02.181001    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsm8g\" (UniqueName: \"kubernetes.io/projected/283e3066-c906-451b-ab9c-a2197f756683-kube-api-access-bsm8g\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc56hm77\" (UID: \"283e3066-c906-451b-ab9c-a2197f756683\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc56hm77"
Sep 08 13:41:02.284232 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:41:02.284142    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc56hm77"
Sep 08 13:41:02.581634 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:41:02.581555    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc56hm77"]
Sep 08 13:41:02.587459 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:41:02.587430    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod283e3066_c906_451b_ab9c_a2197f756683.slice/crio-875d4df4662b19afa92475215f6305ad9cbfd573d1a2b118967e2f1132c0b90e WatchSource:0}: Error finding container 875d4df4662b19afa92475215f6305ad9cbfd573d1a2b118967e2f1132c0b90e: Status 404 returned error can't find the container with id 875d4df4662b19afa92475215f6305ad9cbfd573d1a2b118967e2f1132c0b90e
Sep 08 13:41:02.589369 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:41:02.589354    2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 08 13:41:03.091671 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:41:03.091641    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc56hm77" event={"ID":"283e3066-c906-451b-ab9c-a2197f756683","Type":"ContainerStarted","Data":"875d4df4662b19afa92475215f6305ad9cbfd573d1a2b118967e2f1132c0b90e"}
Sep 08 13:41:04.402518 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:41:04.402467    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:41:06.105258 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:41:06.105216    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc56hm77" event={"ID":"283e3066-c906-451b-ab9c-a2197f756683","Type":"ContainerStarted","Data":"d3c4b05499b583ad053636124c9517c65725cea5db68d5f0fe0e12b312130f0c"}
Sep 08 13:41:06.105786 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:41:06.105351    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc56hm77"
Sep 08 13:41:06.128902 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:41:06.128855    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc56hm77" podStartSLOduration=2.6273214019999998 podStartE2EDuration="5.128840537s" totalImagesPullingTime="2.501519135s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:41:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:41:06.127236405 +0000 UTC m=+1516.398684354" watchObservedRunningTime="2026-09-08 13:41:06.128840537 +0000 UTC m=+1516.400288468"
Sep 08 13:41:06.359457 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:41:06.359377    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:41:06.465511 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:41:06.465463    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:41:09.445116 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:41:09.445082    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:41:10.074249 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:41:10.074221    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:41:10.135914 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:41:10.135879    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:41:12.195199 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:41:12.195164    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:41:17.865388 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:41:17.865306    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:41:19.083877 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:41:19.083843    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:41:19.180688 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:41:19.180655    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:41:24.538494 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:41:24.538458    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7749f6cdf-5vphz"]
Sep 08 13:41:24.545281 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:41:24.545260    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7749f6cdf-5vphz"
Sep 08 13:41:24.548398 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:41:24.548377    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"kube-root-ca.crt\""
Sep 08 13:41:24.548723 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:41:24.548705    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"default-dockercfg-tdbxb\""
Sep 08 13:41:24.548828 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:41:24.548743    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"auth-disabled-test-kserve-self-signed-certs\""
Sep 08 13:41:24.549021 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:41:24.549004    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"odh-kserve-custom-ca-bundle\""
Sep 08 13:41:24.549216 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:41:24.549201    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"openshift-service-ca.crt\""
Sep 08 13:41:24.549359 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:41:24.549342    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"seaweedfs-s3-creds\""
Sep 08 13:41:24.566633 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:41:24.566608    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7749f6cdf-5vphz"]
Sep 08 13:41:24.593160 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:41:24.593125    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:41:24.649858 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:41:24.649832    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6e58049-e11c-4156-b6bf-56624ca5c124-model-cache\") pod \"auth-disabled-test-kserve-7749f6cdf-5vphz\" (UID: \"f6e58049-e11c-4156-b6bf-56624ca5c124\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7749f6cdf-5vphz"
Sep 08 13:41:24.649979 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:41:24.649865    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9dll\" (UniqueName: \"kubernetes.io/projected/f6e58049-e11c-4156-b6bf-56624ca5c124-kube-api-access-d9dll\") pod \"auth-disabled-test-kserve-7749f6cdf-5vphz\" (UID: \"f6e58049-e11c-4156-b6bf-56624ca5c124\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7749f6cdf-5vphz"
Sep 08 13:41:24.649979 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:41:24.649896    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6e58049-e11c-4156-b6bf-56624ca5c124-home\") pod \"auth-disabled-test-kserve-7749f6cdf-5vphz\" (UID: \"f6e58049-e11c-4156-b6bf-56624ca5c124\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7749f6cdf-5vphz"
Sep 08 13:41:24.650050 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:41:24.649980    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f6e58049-e11c-4156-b6bf-56624ca5c124-cabundle-cert\") pod \"auth-disabled-test-kserve-7749f6cdf-5vphz\" (UID: \"f6e58049-e11c-4156-b6bf-56624ca5c124\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7749f6cdf-5vphz"
Sep 08 13:41:24.650050 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:41:24.650020    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6e58049-e11c-4156-b6bf-56624ca5c124-tmp-dir\") pod \"auth-disabled-test-kserve-7749f6cdf-5vphz\" (UID: \"f6e58049-e11c-4156-b6bf-56624ca5c124\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7749f6cdf-5vphz"
Sep 08 13:41:24.650050 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:41:24.650046    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6e58049-e11c-4156-b6bf-56624ca5c124-tls-certs\") pod \"auth-disabled-test-kserve-7749f6cdf-5vphz\" (UID: \"f6e58049-e11c-4156-b6bf-56624ca5c124\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7749f6cdf-5vphz"
Sep 08 13:41:24.650147 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:41:24.650066    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6e58049-e11c-4156-b6bf-56624ca5c124-dshm\") pod \"auth-disabled-test-kserve-7749f6cdf-5vphz\" (UID: \"f6e58049-e11c-4156-b6bf-56624ca5c124\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7749f6cdf-5vphz"
Sep 08 13:41:24.650147 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:41:24.650093    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f6e58049-e11c-4156-b6bf-56624ca5c124-kserve-provision-location\") pod \"auth-disabled-test-kserve-7749f6cdf-5vphz\" (UID: \"f6e58049-e11c-4156-b6bf-56624ca5c124\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7749f6cdf-5vphz"
Sep 08 13:41:24.750942 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:41:24.750910    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6e58049-e11c-4156-b6bf-56624ca5c124-tmp-dir\") pod \"auth-disabled-test-kserve-7749f6cdf-5vphz\" (UID: \"f6e58049-e11c-4156-b6bf-56624ca5c124\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7749f6cdf-5vphz"
Sep 08 13:41:24.750942 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:41:24.750946    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6e58049-e11c-4156-b6bf-56624ca5c124-tls-certs\") pod \"auth-disabled-test-kserve-7749f6cdf-5vphz\" (UID: \"f6e58049-e11c-4156-b6bf-56624ca5c124\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7749f6cdf-5vphz"
Sep 08 13:41:24.751172 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:41:24.750968    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6e58049-e11c-4156-b6bf-56624ca5c124-dshm\") pod \"auth-disabled-test-kserve-7749f6cdf-5vphz\" (UID: \"f6e58049-e11c-4156-b6bf-56624ca5c124\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7749f6cdf-5vphz"
Sep 08 13:41:24.751172 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:41:24.750998    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f6e58049-e11c-4156-b6bf-56624ca5c124-kserve-provision-location\") pod \"auth-disabled-test-kserve-7749f6cdf-5vphz\" (UID: \"f6e58049-e11c-4156-b6bf-56624ca5c124\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7749f6cdf-5vphz"
Sep 08 13:41:24.751172 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:41:24.751042    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6e58049-e11c-4156-b6bf-56624ca5c124-model-cache\") pod \"auth-disabled-test-kserve-7749f6cdf-5vphz\" (UID: \"f6e58049-e11c-4156-b6bf-56624ca5c124\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7749f6cdf-5vphz"
Sep 08 13:41:24.751172 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:41:24.751074    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9dll\" (UniqueName: \"kubernetes.io/projected/f6e58049-e11c-4156-b6bf-56624ca5c124-kube-api-access-d9dll\") pod \"auth-disabled-test-kserve-7749f6cdf-5vphz\" (UID: \"f6e58049-e11c-4156-b6bf-56624ca5c124\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7749f6cdf-5vphz"
Sep 08 13:41:24.751172 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:41:24.751103    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6e58049-e11c-4156-b6bf-56624ca5c124-home\") pod \"auth-disabled-test-kserve-7749f6cdf-5vphz\" (UID: \"f6e58049-e11c-4156-b6bf-56624ca5c124\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7749f6cdf-5vphz"
Sep 08 13:41:24.751172 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:41:24.751133    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f6e58049-e11c-4156-b6bf-56624ca5c124-cabundle-cert\") pod \"auth-disabled-test-kserve-7749f6cdf-5vphz\" (UID: \"f6e58049-e11c-4156-b6bf-56624ca5c124\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7749f6cdf-5vphz"
Sep 08 13:41:24.751471 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:41:24.751393    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6e58049-e11c-4156-b6bf-56624ca5c124-tmp-dir\") pod \"auth-disabled-test-kserve-7749f6cdf-5vphz\" (UID: \"f6e58049-e11c-4156-b6bf-56624ca5c124\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7749f6cdf-5vphz"
Sep 08 13:41:24.751569 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:41:24.751551    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6e58049-e11c-4156-b6bf-56624ca5c124-model-cache\") pod \"auth-disabled-test-kserve-7749f6cdf-5vphz\" (UID: \"f6e58049-e11c-4156-b6bf-56624ca5c124\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7749f6cdf-5vphz"
Sep 08 13:41:24.751616 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:41:24.751593    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6e58049-e11c-4156-b6bf-56624ca5c124-home\") pod \"auth-disabled-test-kserve-7749f6cdf-5vphz\" (UID: \"f6e58049-e11c-4156-b6bf-56624ca5c124\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7749f6cdf-5vphz"
Sep 08 13:41:24.751715 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:41:24.751698    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f6e58049-e11c-4156-b6bf-56624ca5c124-kserve-provision-location\") pod \"auth-disabled-test-kserve-7749f6cdf-5vphz\" (UID: \"f6e58049-e11c-4156-b6bf-56624ca5c124\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7749f6cdf-5vphz"
Sep 08 13:41:24.751873 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:41:24.751853    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f6e58049-e11c-4156-b6bf-56624ca5c124-cabundle-cert\") pod \"auth-disabled-test-kserve-7749f6cdf-5vphz\" (UID: \"f6e58049-e11c-4156-b6bf-56624ca5c124\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7749f6cdf-5vphz"
Sep 08 13:41:24.753382 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:41:24.753361    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6e58049-e11c-4156-b6bf-56624ca5c124-dshm\") pod \"auth-disabled-test-kserve-7749f6cdf-5vphz\" (UID: \"f6e58049-e11c-4156-b6bf-56624ca5c124\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7749f6cdf-5vphz"
Sep 08 13:41:24.753591 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:41:24.753575    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6e58049-e11c-4156-b6bf-56624ca5c124-tls-certs\") pod \"auth-disabled-test-kserve-7749f6cdf-5vphz\" (UID: \"f6e58049-e11c-4156-b6bf-56624ca5c124\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7749f6cdf-5vphz"
Sep 08 13:41:24.780948 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:41:24.780904    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9dll\" (UniqueName: \"kubernetes.io/projected/f6e58049-e11c-4156-b6bf-56624ca5c124-kube-api-access-d9dll\") pod \"auth-disabled-test-kserve-7749f6cdf-5vphz\" (UID: \"f6e58049-e11c-4156-b6bf-56624ca5c124\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7749f6cdf-5vphz"
Sep 08 13:41:24.855680 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:41:24.855613    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7749f6cdf-5vphz"
Sep 08 13:41:24.994753 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:41:24.994717    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:41:25.137323 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:41:25.137293    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7749f6cdf-5vphz"]
Sep 08 13:41:25.140368 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:41:25.140343    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6e58049_e11c_4156_b6bf_56624ca5c124.slice/crio-00e2837007d5dc64678c890869d537662ab6b8c006bbb1175847d399bf1d3aaf WatchSource:0}: Error finding container 00e2837007d5dc64678c890869d537662ab6b8c006bbb1175847d399bf1d3aaf: Status 404 returned error can't find the container with id 00e2837007d5dc64678c890869d537662ab6b8c006bbb1175847d399bf1d3aaf
Sep 08 13:41:25.182586 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:41:25.182563    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7749f6cdf-5vphz" event={"ID":"f6e58049-e11c-4156-b6bf-56624ca5c124","Type":"ContainerStarted","Data":"00e2837007d5dc64678c890869d537662ab6b8c006bbb1175847d399bf1d3aaf"}
Sep 08 13:41:26.187432 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:41:26.187399    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7749f6cdf-5vphz" event={"ID":"f6e58049-e11c-4156-b6bf-56624ca5c124","Type":"ContainerStarted","Data":"180b363da9c6b86a341df8c3b4eb2b037fc038133e58e2bf5c43b24ffa028a33"}
Sep 08 13:41:28.196968 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:41:28.196935    2573 generic.go:356] "Generic (PLEG): container finished" podID="f6e58049-e11c-4156-b6bf-56624ca5c124" containerID="180b363da9c6b86a341df8c3b4eb2b037fc038133e58e2bf5c43b24ffa028a33" exitCode=0
Sep 08 13:41:28.197366 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:41:28.197007    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7749f6cdf-5vphz" event={"ID":"f6e58049-e11c-4156-b6bf-56624ca5c124","Type":"ContainerDied","Data":"180b363da9c6b86a341df8c3b4eb2b037fc038133e58e2bf5c43b24ffa028a33"}
Sep 08 13:41:29.203470 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:41:29.203437    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7749f6cdf-5vphz" event={"ID":"f6e58049-e11c-4156-b6bf-56624ca5c124","Type":"ContainerStarted","Data":"af704b62b6f84c8141e7d93164a116609f3f3e1bf5cd350cddb11151c1dec870"}
Sep 08 13:41:29.279923 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:41:29.279852    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7749f6cdf-5vphz" podStartSLOduration=3.20763448 podStartE2EDuration="5.279817564s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.072183084s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:41:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:41:29.278190931 +0000 UTC m=+1539.549638875" watchObservedRunningTime="2026-09-08 13:41:29.279817564 +0000 UTC m=+1539.551265494"
Sep 08 13:41:29.856559 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:41:29.856521    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7749f6cdf-5vphz"
Sep 08 13:41:33.868935 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:41:33.868900    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:41:34.855919 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:41:34.855883    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7749f6cdf-5vphz"
Sep 08 13:41:34.857496 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:41:34.857463    2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7749f6cdf-5vphz" podUID="f6e58049-e11c-4156-b6bf-56624ca5c124" containerName="main" probeResult="failure" output="Get \"https://10.133.0.26:8000/health\": dial tcp 10.133.0.26:8000: connect: connection refused"
Sep 08 13:41:37.110831 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:41:37.110792    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc56hm77" podUID="283e3066-c906-451b-ab9c-a2197f756683" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 08 13:41:38.355403 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:41:38.355368    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:41:38.377415 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:41:38.377382    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc56hm77"]
Sep 08 13:41:38.379224 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:41:38.379180    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc56hm77" podUID="283e3066-c906-451b-ab9c-a2197f756683" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 08 13:41:38.431081 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:41:38.431037    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:41:44.856687 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:41:44.856635    2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7749f6cdf-5vphz" podUID="f6e58049-e11c-4156-b6bf-56624ca5c124" containerName="main" probeResult="failure" output="Get \"https://10.133.0.26:8000/health\": dial tcp 10.133.0.26:8000: connect: connection refused"
Sep 08 13:41:48.379273 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:41:48.379230    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc56hm77" podUID="283e3066-c906-451b-ab9c-a2197f756683" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 08 13:41:53.400693 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:41:53.400642    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc56hm77" podUID="283e3066-c906-451b-ab9c-a2197f756683" containerName="main" containerID="cri-o://d3c4b05499b583ad053636124c9517c65725cea5db68d5f0fe0e12b312130f0c" gracePeriod=45
Sep 08 13:41:54.856561 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:41:54.856523    2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7749f6cdf-5vphz" podUID="f6e58049-e11c-4156-b6bf-56624ca5c124" containerName="main" probeResult="failure" output="Get \"https://10.133.0.26:8000/health\": dial tcp 10.133.0.26:8000: connect: connection refused"
Sep 08 13:41:58.379531 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:41:58.379474    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc56hm77" podUID="283e3066-c906-451b-ab9c-a2197f756683" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 08 13:42:04.856256 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:42:04.856212    2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7749f6cdf-5vphz" podUID="f6e58049-e11c-4156-b6bf-56624ca5c124" containerName="main" probeResult="failure" output="Get \"https://10.133.0.26:8000/health\": dial tcp 10.133.0.26:8000: connect: connection refused"
Sep 08 13:42:08.379487 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:42:08.379451    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc56hm77" podUID="283e3066-c906-451b-ab9c-a2197f756683" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 08 13:42:14.855991 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:42:14.855943    2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7749f6cdf-5vphz" podUID="f6e58049-e11c-4156-b6bf-56624ca5c124" containerName="main" probeResult="failure" output="Get \"https://10.133.0.26:8000/health\": dial tcp 10.133.0.26:8000: connect: connection refused"
Sep 08 13:42:18.379220 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:42:18.379180    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc56hm77" podUID="283e3066-c906-451b-ab9c-a2197f756683" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 08 13:42:23.646579 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:42:23.646550    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc56hm77"
Sep 08 13:42:23.775131 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:42:23.775049    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/283e3066-c906-451b-ab9c-a2197f756683-tls-certs\") pod \"283e3066-c906-451b-ab9c-a2197f756683\" (UID: \"283e3066-c906-451b-ab9c-a2197f756683\") "
Sep 08 13:42:23.775131 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:42:23.775125    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bsm8g\" (UniqueName: \"kubernetes.io/projected/283e3066-c906-451b-ab9c-a2197f756683-kube-api-access-bsm8g\") pod \"283e3066-c906-451b-ab9c-a2197f756683\" (UID: \"283e3066-c906-451b-ab9c-a2197f756683\") "
Sep 08 13:42:23.777360 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:42:23.777327    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/283e3066-c906-451b-ab9c-a2197f756683-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "283e3066-c906-451b-ab9c-a2197f756683" (UID: "283e3066-c906-451b-ab9c-a2197f756683"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:42:23.777478 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:42:23.777389    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/283e3066-c906-451b-ab9c-a2197f756683-kube-api-access-bsm8g" (OuterVolumeSpecName: "kube-api-access-bsm8g") pod "283e3066-c906-451b-ab9c-a2197f756683" (UID: "283e3066-c906-451b-ab9c-a2197f756683"). InnerVolumeSpecName "kube-api-access-bsm8g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:42:23.876177 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:42:23.876139    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/283e3066-c906-451b-ab9c-a2197f756683-tls-certs\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 13:42:23.876177 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:42:23.876172    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bsm8g\" (UniqueName: \"kubernetes.io/projected/283e3066-c906-451b-ab9c-a2197f756683-kube-api-access-bsm8g\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 13:42:24.460831 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:42:24.460740    2573 generic.go:356] "Generic (PLEG): container finished" podID="283e3066-c906-451b-ab9c-a2197f756683" containerID="d3c4b05499b583ad053636124c9517c65725cea5db68d5f0fe0e12b312130f0c" exitCode=0
Sep 08 13:42:24.460831 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:42:24.460821    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc56hm77"
Sep 08 13:42:24.461028 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:42:24.460823    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc56hm77" event={"ID":"283e3066-c906-451b-ab9c-a2197f756683","Type":"ContainerDied","Data":"d3c4b05499b583ad053636124c9517c65725cea5db68d5f0fe0e12b312130f0c"}
Sep 08 13:42:24.461028 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:42:24.460875    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc56hm77" event={"ID":"283e3066-c906-451b-ab9c-a2197f756683","Type":"ContainerDied","Data":"875d4df4662b19afa92475215f6305ad9cbfd573d1a2b118967e2f1132c0b90e"}
Sep 08 13:42:24.461028 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:42:24.460897    2573 scope.go:117] "RemoveContainer" containerID="d3c4b05499b583ad053636124c9517c65725cea5db68d5f0fe0e12b312130f0c"
Sep 08 13:42:24.470594 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:42:24.470576    2573 scope.go:117] "RemoveContainer" containerID="d3c4b05499b583ad053636124c9517c65725cea5db68d5f0fe0e12b312130f0c"
Sep 08 13:42:24.470907 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:42:24.470881    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3c4b05499b583ad053636124c9517c65725cea5db68d5f0fe0e12b312130f0c\": container with ID starting with d3c4b05499b583ad053636124c9517c65725cea5db68d5f0fe0e12b312130f0c not found: ID does not exist" containerID="d3c4b05499b583ad053636124c9517c65725cea5db68d5f0fe0e12b312130f0c"
Sep 08 13:42:24.470957 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:42:24.470908    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3c4b05499b583ad053636124c9517c65725cea5db68d5f0fe0e12b312130f0c"} err="failed to get container status \"d3c4b05499b583ad053636124c9517c65725cea5db68d5f0fe0e12b312130f0c\": rpc error: code = NotFound desc = could not find container \"d3c4b05499b583ad053636124c9517c65725cea5db68d5f0fe0e12b312130f0c\": container with ID starting with d3c4b05499b583ad053636124c9517c65725cea5db68d5f0fe0e12b312130f0c not found: ID does not exist"
Sep 08 13:42:24.479918 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:42:24.479896    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc56hm77"]
Sep 08 13:42:24.493658 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:42:24.493632    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc56hm77"]
Sep 08 13:42:24.856954 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:42:24.856859    2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7749f6cdf-5vphz" podUID="f6e58049-e11c-4156-b6bf-56624ca5c124" containerName="main" probeResult="failure" output="Get \"https://10.133.0.26:8000/health\": dial tcp 10.133.0.26:8000: connect: connection refused"
Sep 08 13:42:26.176041 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:42:26.176005    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="283e3066-c906-451b-ab9c-a2197f756683" path="/var/lib/kubelet/pods/283e3066-c906-451b-ab9c-a2197f756683/volumes"
Sep 08 13:42:27.140513 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:42:27.140461    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:42:27.240136 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:42:27.240102    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:42:30.134809 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:42:30.134775    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:42:30.350958 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:42:30.350922    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcfx7h6"]
Sep 08 13:42:30.351232 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:42:30.351218    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="283e3066-c906-451b-ab9c-a2197f756683" containerName="main"
Sep 08 13:42:30.351282 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:42:30.351233    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="283e3066-c906-451b-ab9c-a2197f756683" containerName="main"
Sep 08 13:42:30.351322 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:42:30.351291    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="283e3066-c906-451b-ab9c-a2197f756683" containerName="main"
Sep 08 13:42:30.355663 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:42:30.355640    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcfx7h6"
Sep 08 13:42:30.359648 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:42:30.359623    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"default-dockercfg-qb4bh\""
Sep 08 13:42:30.360661 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:42:30.360639    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"scheduler-custom-template-test-epp-sa-dockercfg-6b6b8\""
Sep 08 13:42:30.360879 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:42:30.360708    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"openshift-service-ca.crt\""
Sep 08 13:42:30.360996 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:42:30.360739    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"kube-root-ca.crt\""
Sep 08 13:42:30.361070 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:42:30.360769    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"scheduler-custom-template-test-kserve-self-signed-certs\""
Sep 08 13:42:30.370367 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:42:30.370342    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcfx7h6"]
Sep 08 13:42:30.535807 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:42:30.535775    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac6521e7-6202-42ef-b7f2-46129a24f836-cachi2-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcfx7h6\" (UID: \"ac6521e7-6202-42ef-b7f2-46129a24f836\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcfx7h6"
Sep 08 13:42:30.535943 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:42:30.535828    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac6521e7-6202-42ef-b7f2-46129a24f836-tokenizer-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcfx7h6\" (UID: \"ac6521e7-6202-42ef-b7f2-46129a24f836\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcfx7h6"
Sep 08 13:42:30.535943 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:42:30.535908    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac6521e7-6202-42ef-b7f2-46129a24f836-tls-certs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcfx7h6\" (UID: \"ac6521e7-6202-42ef-b7f2-46129a24f836\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcfx7h6"
Sep 08 13:42:30.536075 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:42:30.535957    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgpf8\" (UniqueName: \"kubernetes.io/projected/ac6521e7-6202-42ef-b7f2-46129a24f836-kube-api-access-hgpf8\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcfx7h6\" (UID: \"ac6521e7-6202-42ef-b7f2-46129a24f836\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcfx7h6"
Sep 08 13:42:30.636457 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:42:30.636422    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac6521e7-6202-42ef-b7f2-46129a24f836-cachi2-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcfx7h6\" (UID: \"ac6521e7-6202-42ef-b7f2-46129a24f836\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcfx7h6"
Sep 08 13:42:30.636676 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:42:30.636468    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac6521e7-6202-42ef-b7f2-46129a24f836-tokenizer-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcfx7h6\" (UID: \"ac6521e7-6202-42ef-b7f2-46129a24f836\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcfx7h6"
Sep 08 13:42:30.636676 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:42:30.636543    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac6521e7-6202-42ef-b7f2-46129a24f836-tls-certs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcfx7h6\" (UID: \"ac6521e7-6202-42ef-b7f2-46129a24f836\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcfx7h6"
Sep 08 13:42:30.636676 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:42:30.636570    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgpf8\" (UniqueName: \"kubernetes.io/projected/ac6521e7-6202-42ef-b7f2-46129a24f836-kube-api-access-hgpf8\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcfx7h6\" (UID: \"ac6521e7-6202-42ef-b7f2-46129a24f836\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcfx7h6"
Sep 08 13:42:30.637022 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:42:30.636999    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac6521e7-6202-42ef-b7f2-46129a24f836-cachi2-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcfx7h6\" (UID: \"ac6521e7-6202-42ef-b7f2-46129a24f836\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcfx7h6"
Sep 08 13:42:30.637105 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:42:30.637044    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac6521e7-6202-42ef-b7f2-46129a24f836-tokenizer-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcfx7h6\" (UID: \"ac6521e7-6202-42ef-b7f2-46129a24f836\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcfx7h6"
Sep 08 13:42:30.639109 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:42:30.639077    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac6521e7-6202-42ef-b7f2-46129a24f836-tls-certs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcfx7h6\" (UID: \"ac6521e7-6202-42ef-b7f2-46129a24f836\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcfx7h6"
Sep 08 13:42:30.649774 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:42:30.649744    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgpf8\" (UniqueName: \"kubernetes.io/projected/ac6521e7-6202-42ef-b7f2-46129a24f836-kube-api-access-hgpf8\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcfx7h6\" (UID: \"ac6521e7-6202-42ef-b7f2-46129a24f836\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcfx7h6"
Sep 08 13:42:30.666954 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:42:30.666931    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcfx7h6"
Sep 08 13:42:30.958492 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:42:30.958457    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcfx7h6"]
Sep 08 13:42:30.962319 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:42:30.962287    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac6521e7_6202_42ef_b7f2_46129a24f836.slice/crio-b3d68c37ac3ba2057da473a6941acdc0b12765f01e78679786ea5831a125b24c WatchSource:0}: Error finding container b3d68c37ac3ba2057da473a6941acdc0b12765f01e78679786ea5831a125b24c: Status 404 returned error can't find the container with id b3d68c37ac3ba2057da473a6941acdc0b12765f01e78679786ea5831a125b24c
Sep 08 13:42:31.493585 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:42:31.493548    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcfx7h6" event={"ID":"ac6521e7-6202-42ef-b7f2-46129a24f836","Type":"ContainerStarted","Data":"29b5300fad16920b0fb21506e04669505c7b23fa9748ea8e01ae7996509d9bed"}
Sep 08 13:42:31.493981 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:42:31.493599    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcfx7h6" event={"ID":"ac6521e7-6202-42ef-b7f2-46129a24f836","Type":"ContainerStarted","Data":"b3d68c37ac3ba2057da473a6941acdc0b12765f01e78679786ea5831a125b24c"}
Sep 08 13:42:31.493981 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:42:31.493665    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcfx7h6"
Sep 08 13:42:31.514696 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:42:31.514645    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcfx7h6" podStartSLOduration=1.514630846 podStartE2EDuration="1.514630846s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:42:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:42:31.512620065 +0000 UTC m=+1601.784068008" watchObservedRunningTime="2026-09-08 13:42:31.514630846 +0000 UTC m=+1601.786078777"
Sep 08 13:42:31.594824 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:42:31.594788    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:42:34.357796 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:42:34.357755    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:42:34.856788 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:42:34.856749    2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7749f6cdf-5vphz" podUID="f6e58049-e11c-4156-b6bf-56624ca5c124" containerName="main" probeResult="failure" output="Get \"https://10.133.0.26:8000/health\": dial tcp 10.133.0.26:8000: connect: connection refused"
Sep 08 13:42:40.291837 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:42:40.291801    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:42:40.390360 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:42:40.390325    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:42:40.989369 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:42:40.989336    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:42:44.856848 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:42:44.856747    2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7749f6cdf-5vphz" podUID="f6e58049-e11c-4156-b6bf-56624ca5c124" containerName="main" probeResult="failure" output="Get \"https://10.133.0.26:8000/health\": dial tcp 10.133.0.26:8000: connect: connection refused"
Sep 08 13:42:54.856221 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:42:54.856167    2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7749f6cdf-5vphz" podUID="f6e58049-e11c-4156-b6bf-56624ca5c124" containerName="main" probeResult="failure" output="Get \"https://10.133.0.26:8000/health\": dial tcp 10.133.0.26:8000: connect: connection refused"
Sep 08 13:43:02.502182 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:43:02.502152    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcfx7h6"
Sep 08 13:43:04.320351 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:43:04.320309    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcfx7h6"]
Sep 08 13:43:04.326012 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:43:04.325984    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:43:04.529664 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:43:04.529629    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:43:04.856105 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:43:04.856064    2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7749f6cdf-5vphz" podUID="f6e58049-e11c-4156-b6bf-56624ca5c124" containerName="main" probeResult="failure" output="Get \"https://10.133.0.26:8000/health\": dial tcp 10.133.0.26:8000: connect: connection refused"
Sep 08 13:43:12.502274 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:43:12.502228    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcfx7h6" podUID="ac6521e7-6202-42ef-b7f2-46129a24f836" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 08 13:43:14.856442 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:43:14.856392    2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7749f6cdf-5vphz" podUID="f6e58049-e11c-4156-b6bf-56624ca5c124" containerName="main" probeResult="failure" output="Get \"https://10.133.0.26:8000/health\": dial tcp 10.133.0.26:8000: connect: connection refused"
Sep 08 13:43:19.340235 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:43:19.340166    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcfx7h6" podUID="ac6521e7-6202-42ef-b7f2-46129a24f836" containerName="main" containerID="cri-o://29b5300fad16920b0fb21506e04669505c7b23fa9748ea8e01ae7996509d9bed" gracePeriod=45
Sep 08 13:43:22.501829 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:43:22.501794    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcfx7h6" podUID="ac6521e7-6202-42ef-b7f2-46129a24f836" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 08 13:43:24.866490 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:43:24.866459    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7749f6cdf-5vphz"
Sep 08 13:43:24.874400 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:43:24.874379    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7749f6cdf-5vphz"
Sep 08 13:43:29.939378 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:43:29.939347    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7749f6cdf-5vphz"]
Sep 08 13:43:29.973146 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:43:29.973118    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:43:30.050398 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:43:30.050369    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:43:32.502025 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:43:32.501992    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcfx7h6" podUID="ac6521e7-6202-42ef-b7f2-46129a24f836" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 08 13:43:32.502445 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:43:32.502093    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcfx7h6"
Sep 08 13:43:42.502367 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:43:42.502325    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcfx7h6" podUID="ac6521e7-6202-42ef-b7f2-46129a24f836" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 08 13:43:44.962136 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:43:44.962079    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7749f6cdf-5vphz" podUID="f6e58049-e11c-4156-b6bf-56624ca5c124" containerName="main" containerID="cri-o://af704b62b6f84c8141e7d93164a116609f3f3e1bf5cd350cddb11151c1dec870" gracePeriod=45
Sep 08 13:43:45.607473 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:43:45.607449    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7749f6cdf-5vphz"
Sep 08 13:43:45.645531 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:43:45.645488    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f6e58049-e11c-4156-b6bf-56624ca5c124-kserve-provision-location\") pod \"f6e58049-e11c-4156-b6bf-56624ca5c124\" (UID: \"f6e58049-e11c-4156-b6bf-56624ca5c124\") "
Sep 08 13:43:45.645729 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:43:45.645700    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6e58049-e11c-4156-b6bf-56624ca5c124-tls-certs\") pod \"f6e58049-e11c-4156-b6bf-56624ca5c124\" (UID: \"f6e58049-e11c-4156-b6bf-56624ca5c124\") "
Sep 08 13:43:45.645851 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:43:45.645774    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6e58049-e11c-4156-b6bf-56624ca5c124-home\") pod \"f6e58049-e11c-4156-b6bf-56624ca5c124\" (UID: \"f6e58049-e11c-4156-b6bf-56624ca5c124\") "
Sep 08 13:43:45.645851 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:43:45.645809    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d9dll\" (UniqueName: \"kubernetes.io/projected/f6e58049-e11c-4156-b6bf-56624ca5c124-kube-api-access-d9dll\") pod \"f6e58049-e11c-4156-b6bf-56624ca5c124\" (UID: \"f6e58049-e11c-4156-b6bf-56624ca5c124\") "
Sep 08 13:43:45.645976 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:43:45.645856    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6e58049-e11c-4156-b6bf-56624ca5c124-dshm\") pod \"f6e58049-e11c-4156-b6bf-56624ca5c124\" (UID: \"f6e58049-e11c-4156-b6bf-56624ca5c124\") "
Sep 08 13:43:45.645976 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:43:45.645914    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6e58049-e11c-4156-b6bf-56624ca5c124-model-cache\") pod \"f6e58049-e11c-4156-b6bf-56624ca5c124\" (UID: \"f6e58049-e11c-4156-b6bf-56624ca5c124\") "
Sep 08 13:43:45.645976 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:43:45.645970    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6e58049-e11c-4156-b6bf-56624ca5c124-tmp-dir\") pod \"f6e58049-e11c-4156-b6bf-56624ca5c124\" (UID: \"f6e58049-e11c-4156-b6bf-56624ca5c124\") "
Sep 08 13:43:45.646136 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:43:45.646008    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f6e58049-e11c-4156-b6bf-56624ca5c124-cabundle-cert\") pod \"f6e58049-e11c-4156-b6bf-56624ca5c124\" (UID: \"f6e58049-e11c-4156-b6bf-56624ca5c124\") "
Sep 08 13:43:45.646494 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:43:45.646465    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6e58049-e11c-4156-b6bf-56624ca5c124-model-cache" (OuterVolumeSpecName: "model-cache") pod "f6e58049-e11c-4156-b6bf-56624ca5c124" (UID: "f6e58049-e11c-4156-b6bf-56624ca5c124"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:43:45.646891 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:43:45.646865    2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6e58049-e11c-4156-b6bf-56624ca5c124-model-cache\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 13:43:45.646891 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:43:45.646607    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6e58049-e11c-4156-b6bf-56624ca5c124-home" (OuterVolumeSpecName: "home") pod "f6e58049-e11c-4156-b6bf-56624ca5c124" (UID: "f6e58049-e11c-4156-b6bf-56624ca5c124"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:43:45.647524 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:43:45.647484    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6e58049-e11c-4156-b6bf-56624ca5c124-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f6e58049-e11c-4156-b6bf-56624ca5c124" (UID: "f6e58049-e11c-4156-b6bf-56624ca5c124"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:43:45.649114 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:43:45.649068    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6e58049-e11c-4156-b6bf-56624ca5c124-dshm" (OuterVolumeSpecName: "dshm") pod "f6e58049-e11c-4156-b6bf-56624ca5c124" (UID: "f6e58049-e11c-4156-b6bf-56624ca5c124"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:43:45.649114 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:43:45.649091    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6e58049-e11c-4156-b6bf-56624ca5c124-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f6e58049-e11c-4156-b6bf-56624ca5c124" (UID: "f6e58049-e11c-4156-b6bf-56624ca5c124"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:43:45.649343 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:43:45.649321    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6e58049-e11c-4156-b6bf-56624ca5c124-kube-api-access-d9dll" (OuterVolumeSpecName: "kube-api-access-d9dll") pod "f6e58049-e11c-4156-b6bf-56624ca5c124" (UID: "f6e58049-e11c-4156-b6bf-56624ca5c124"). InnerVolumeSpecName "kube-api-access-d9dll". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:43:45.666056 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:43:45.666019    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6e58049-e11c-4156-b6bf-56624ca5c124-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f6e58049-e11c-4156-b6bf-56624ca5c124" (UID: "f6e58049-e11c-4156-b6bf-56624ca5c124"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:43:45.672383 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:43:45.672359    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6e58049-e11c-4156-b6bf-56624ca5c124-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f6e58049-e11c-4156-b6bf-56624ca5c124" (UID: "f6e58049-e11c-4156-b6bf-56624ca5c124"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:43:45.747428 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:43:45.747381    2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f6e58049-e11c-4156-b6bf-56624ca5c124-cabundle-cert\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 13:43:45.747428 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:43:45.747401    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f6e58049-e11c-4156-b6bf-56624ca5c124-kserve-provision-location\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 13:43:45.747428 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:43:45.747411    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6e58049-e11c-4156-b6bf-56624ca5c124-tls-certs\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 13:43:45.747428 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:43:45.747420    2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6e58049-e11c-4156-b6bf-56624ca5c124-home\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 13:43:45.747428 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:43:45.747428    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9dll\" (UniqueName: \"kubernetes.io/projected/f6e58049-e11c-4156-b6bf-56624ca5c124-kube-api-access-d9dll\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 13:43:45.747657 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:43:45.747435    2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6e58049-e11c-4156-b6bf-56624ca5c124-dshm\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 13:43:45.747657 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:43:45.747443    2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6e58049-e11c-4156-b6bf-56624ca5c124-tmp-dir\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 13:43:45.831696 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:43:45.831671    2573 generic.go:356] "Generic (PLEG): container finished" podID="f6e58049-e11c-4156-b6bf-56624ca5c124" containerID="af704b62b6f84c8141e7d93164a116609f3f3e1bf5cd350cddb11151c1dec870" exitCode=0
Sep 08 13:43:45.831780 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:43:45.831742    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7749f6cdf-5vphz"
Sep 08 13:43:45.831780 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:43:45.831753    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7749f6cdf-5vphz" event={"ID":"f6e58049-e11c-4156-b6bf-56624ca5c124","Type":"ContainerDied","Data":"af704b62b6f84c8141e7d93164a116609f3f3e1bf5cd350cddb11151c1dec870"}
Sep 08 13:43:45.831855 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:43:45.831826    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7749f6cdf-5vphz" event={"ID":"f6e58049-e11c-4156-b6bf-56624ca5c124","Type":"ContainerDied","Data":"00e2837007d5dc64678c890869d537662ab6b8c006bbb1175847d399bf1d3aaf"}
Sep 08 13:43:45.831889 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:43:45.831858    2573 scope.go:117] "RemoveContainer" containerID="af704b62b6f84c8141e7d93164a116609f3f3e1bf5cd350cddb11151c1dec870"
Sep 08 13:43:45.840284 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:43:45.840265    2573 scope.go:117] "RemoveContainer" containerID="180b363da9c6b86a341df8c3b4eb2b037fc038133e58e2bf5c43b24ffa028a33"
Sep 08 13:43:45.847410 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:43:45.847393    2573 scope.go:117] "RemoveContainer" containerID="af704b62b6f84c8141e7d93164a116609f3f3e1bf5cd350cddb11151c1dec870"
Sep 08 13:43:45.847696 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:43:45.847677    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af704b62b6f84c8141e7d93164a116609f3f3e1bf5cd350cddb11151c1dec870\": container with ID starting with af704b62b6f84c8141e7d93164a116609f3f3e1bf5cd350cddb11151c1dec870 not found: ID does not exist" containerID="af704b62b6f84c8141e7d93164a116609f3f3e1bf5cd350cddb11151c1dec870"
Sep 08 13:43:45.847756 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:43:45.847702    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af704b62b6f84c8141e7d93164a116609f3f3e1bf5cd350cddb11151c1dec870"} err="failed to get container status \"af704b62b6f84c8141e7d93164a116609f3f3e1bf5cd350cddb11151c1dec870\": rpc error: code = NotFound desc = could not find container \"af704b62b6f84c8141e7d93164a116609f3f3e1bf5cd350cddb11151c1dec870\": container with ID starting with af704b62b6f84c8141e7d93164a116609f3f3e1bf5cd350cddb11151c1dec870 not found: ID does not exist"
Sep 08 13:43:45.847756 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:43:45.847718    2573 scope.go:117] "RemoveContainer" containerID="180b363da9c6b86a341df8c3b4eb2b037fc038133e58e2bf5c43b24ffa028a33"
Sep 08 13:43:45.847973 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:43:45.847956    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"180b363da9c6b86a341df8c3b4eb2b037fc038133e58e2bf5c43b24ffa028a33\": container with ID starting with 180b363da9c6b86a341df8c3b4eb2b037fc038133e58e2bf5c43b24ffa028a33 not found: ID does not exist" containerID="180b363da9c6b86a341df8c3b4eb2b037fc038133e58e2bf5c43b24ffa028a33"
Sep 08 13:43:45.848015 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:43:45.847979    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"180b363da9c6b86a341df8c3b4eb2b037fc038133e58e2bf5c43b24ffa028a33"} err="failed to get container status \"180b363da9c6b86a341df8c3b4eb2b037fc038133e58e2bf5c43b24ffa028a33\": rpc error: code = NotFound desc = could not find container \"180b363da9c6b86a341df8c3b4eb2b037fc038133e58e2bf5c43b24ffa028a33\": container with ID starting with 180b363da9c6b86a341df8c3b4eb2b037fc038133e58e2bf5c43b24ffa028a33 not found: ID does not exist"
Sep 08 13:43:45.854645 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:43:45.854625    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7749f6cdf-5vphz"]
Sep 08 13:43:45.860124 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:43:45.860100    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7749f6cdf-5vphz"]
Sep 08 13:43:46.174840 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:43:46.174810    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f6e58049-e11c-4156-b6bf-56624ca5c124" path="/var/lib/kubelet/pods/f6e58049-e11c-4156-b6bf-56624ca5c124/volumes"
Sep 08 13:43:49.576888 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:43:49.576861    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcfx7h6"
Sep 08 13:43:49.677582 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:43:49.677555    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac6521e7-6202-42ef-b7f2-46129a24f836-tokenizer-cache\") pod \"ac6521e7-6202-42ef-b7f2-46129a24f836\" (UID: \"ac6521e7-6202-42ef-b7f2-46129a24f836\") "
Sep 08 13:43:49.677717 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:43:49.677613    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hgpf8\" (UniqueName: \"kubernetes.io/projected/ac6521e7-6202-42ef-b7f2-46129a24f836-kube-api-access-hgpf8\") pod \"ac6521e7-6202-42ef-b7f2-46129a24f836\" (UID: \"ac6521e7-6202-42ef-b7f2-46129a24f836\") "
Sep 08 13:43:49.677717 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:43:49.677644    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac6521e7-6202-42ef-b7f2-46129a24f836-tls-certs\") pod \"ac6521e7-6202-42ef-b7f2-46129a24f836\" (UID: \"ac6521e7-6202-42ef-b7f2-46129a24f836\") "
Sep 08 13:43:49.677717 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:43:49.677690    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac6521e7-6202-42ef-b7f2-46129a24f836-cachi2-cache\") pod \"ac6521e7-6202-42ef-b7f2-46129a24f836\" (UID: \"ac6521e7-6202-42ef-b7f2-46129a24f836\") "
Sep 08 13:43:49.677886 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:43:49.677790    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac6521e7-6202-42ef-b7f2-46129a24f836-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "ac6521e7-6202-42ef-b7f2-46129a24f836" (UID: "ac6521e7-6202-42ef-b7f2-46129a24f836"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:43:49.677886 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:43:49.677869    2573 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac6521e7-6202-42ef-b7f2-46129a24f836-tokenizer-cache\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 13:43:49.678185 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:43:49.678165    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac6521e7-6202-42ef-b7f2-46129a24f836-cachi2-cache" (OuterVolumeSpecName: "cachi2-cache") pod "ac6521e7-6202-42ef-b7f2-46129a24f836" (UID: "ac6521e7-6202-42ef-b7f2-46129a24f836"). InnerVolumeSpecName "cachi2-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:43:49.679827 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:43:49.679803    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac6521e7-6202-42ef-b7f2-46129a24f836-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ac6521e7-6202-42ef-b7f2-46129a24f836" (UID: "ac6521e7-6202-42ef-b7f2-46129a24f836"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:43:49.679927 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:43:49.679831    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac6521e7-6202-42ef-b7f2-46129a24f836-kube-api-access-hgpf8" (OuterVolumeSpecName: "kube-api-access-hgpf8") pod "ac6521e7-6202-42ef-b7f2-46129a24f836" (UID: "ac6521e7-6202-42ef-b7f2-46129a24f836"). InnerVolumeSpecName "kube-api-access-hgpf8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:43:49.778340 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:43:49.778306    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hgpf8\" (UniqueName: \"kubernetes.io/projected/ac6521e7-6202-42ef-b7f2-46129a24f836-kube-api-access-hgpf8\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 13:43:49.778340 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:43:49.778338    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac6521e7-6202-42ef-b7f2-46129a24f836-tls-certs\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 13:43:49.778340 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:43:49.778348    2573 reconciler_common.go:299] "Volume detached for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac6521e7-6202-42ef-b7f2-46129a24f836-cachi2-cache\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 13:43:49.851474 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:43:49.851449    2573 generic.go:356] "Generic (PLEG): container finished" podID="ac6521e7-6202-42ef-b7f2-46129a24f836" containerID="29b5300fad16920b0fb21506e04669505c7b23fa9748ea8e01ae7996509d9bed" exitCode=0
Sep 08 13:43:49.851577 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:43:49.851492    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcfx7h6" event={"ID":"ac6521e7-6202-42ef-b7f2-46129a24f836","Type":"ContainerDied","Data":"29b5300fad16920b0fb21506e04669505c7b23fa9748ea8e01ae7996509d9bed"}
Sep 08 13:43:49.851577 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:43:49.851548    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcfx7h6" event={"ID":"ac6521e7-6202-42ef-b7f2-46129a24f836","Type":"ContainerDied","Data":"b3d68c37ac3ba2057da473a6941acdc0b12765f01e78679786ea5831a125b24c"}
Sep 08 13:43:49.851577 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:43:49.851572    2573 scope.go:117] "RemoveContainer" containerID="29b5300fad16920b0fb21506e04669505c7b23fa9748ea8e01ae7996509d9bed"
Sep 08 13:43:49.851737 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:43:49.851513    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcfx7h6"
Sep 08 13:43:49.859380 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:43:49.859365    2573 scope.go:117] "RemoveContainer" containerID="29b5300fad16920b0fb21506e04669505c7b23fa9748ea8e01ae7996509d9bed"
Sep 08 13:43:49.859682 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:43:49.859660    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29b5300fad16920b0fb21506e04669505c7b23fa9748ea8e01ae7996509d9bed\": container with ID starting with 29b5300fad16920b0fb21506e04669505c7b23fa9748ea8e01ae7996509d9bed not found: ID does not exist" containerID="29b5300fad16920b0fb21506e04669505c7b23fa9748ea8e01ae7996509d9bed"
Sep 08 13:43:49.859770 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:43:49.859690    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29b5300fad16920b0fb21506e04669505c7b23fa9748ea8e01ae7996509d9bed"} err="failed to get container status \"29b5300fad16920b0fb21506e04669505c7b23fa9748ea8e01ae7996509d9bed\": rpc error: code = NotFound desc = could not find container \"29b5300fad16920b0fb21506e04669505c7b23fa9748ea8e01ae7996509d9bed\": container with ID starting with 29b5300fad16920b0fb21506e04669505c7b23fa9748ea8e01ae7996509d9bed not found: ID does not exist"
Sep 08 13:43:49.875768 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:43:49.875746    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcfx7h6"]
Sep 08 13:43:49.880692 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:43:49.880671    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcfx7h6"]
Sep 08 13:43:50.179787 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:43:50.179761    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ac6521e7-6202-42ef-b7f2-46129a24f836" path="/var/lib/kubelet/pods/ac6521e7-6202-42ef-b7f2-46129a24f836/volumes"
Sep 08 13:43:53.790108 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:43:53.790079    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:43:53.882083 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:43:53.882048    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:44:01.367521 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:44:01.367476    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:44:02.060156 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:44:02.060122    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-747767b756-qxmnn"]
Sep 08 13:44:02.060406 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:44:02.060394    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6e58049-e11c-4156-b6bf-56624ca5c124" containerName="main"
Sep 08 13:44:02.060459 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:44:02.060407    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6e58049-e11c-4156-b6bf-56624ca5c124" containerName="main"
Sep 08 13:44:02.060459 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:44:02.060421    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6e58049-e11c-4156-b6bf-56624ca5c124" containerName="storage-initializer"
Sep 08 13:44:02.060459 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:44:02.060426    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6e58049-e11c-4156-b6bf-56624ca5c124" containerName="storage-initializer"
Sep 08 13:44:02.060459 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:44:02.060440    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac6521e7-6202-42ef-b7f2-46129a24f836" containerName="main"
Sep 08 13:44:02.060459 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:44:02.060445    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac6521e7-6202-42ef-b7f2-46129a24f836" containerName="main"
Sep 08 13:44:02.060659 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:44:02.060491    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac6521e7-6202-42ef-b7f2-46129a24f836" containerName="main"
Sep 08 13:44:02.060659 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:44:02.060514    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="f6e58049-e11c-4156-b6bf-56624ca5c124" containerName="main"
Sep 08 13:44:02.064604 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:44:02.064587    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-747767b756-qxmnn"
Sep 08 13:44:02.067468 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:44:02.067443    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"default-dockercfg-4sgjm\""
Sep 08 13:44:02.068112 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:44:02.068097    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"openshift-service-ca.crt\""
Sep 08 13:44:02.068177 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:44:02.068162    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-kserve-self-signed-certs\""
Sep 08 13:44:02.068271 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:44:02.068233    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"kube-root-ca.crt\""
Sep 08 13:44:02.078014 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:44:02.077991    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-747767b756-qxmnn"]
Sep 08 13:44:02.168658 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:44:02.168603    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa631cd9-393d-4d28-ab29-9444a277cf8c-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-747767b756-qxmnn\" (UID: \"aa631cd9-393d-4d28-ab29-9444a277cf8c\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-747767b756-qxmnn"
Sep 08 13:44:02.168841 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:44:02.168664    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa631cd9-393d-4d28-ab29-9444a277cf8c-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-747767b756-qxmnn\" (UID: \"aa631cd9-393d-4d28-ab29-9444a277cf8c\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-747767b756-qxmnn"
Sep 08 13:44:02.168841 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:44:02.168701    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa631cd9-393d-4d28-ab29-9444a277cf8c-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-747767b756-qxmnn\" (UID: \"aa631cd9-393d-4d28-ab29-9444a277cf8c\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-747767b756-qxmnn"
Sep 08 13:44:02.168841 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:44:02.168728    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa631cd9-393d-4d28-ab29-9444a277cf8c-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-747767b756-qxmnn\" (UID: \"aa631cd9-393d-4d28-ab29-9444a277cf8c\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-747767b756-qxmnn"
Sep 08 13:44:02.168841 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:44:02.168772    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65g6d\" (UniqueName: \"kubernetes.io/projected/aa631cd9-393d-4d28-ab29-9444a277cf8c-kube-api-access-65g6d\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-747767b756-qxmnn\" (UID: \"aa631cd9-393d-4d28-ab29-9444a277cf8c\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-747767b756-qxmnn"
Sep 08 13:44:02.168841 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:44:02.168799    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa631cd9-393d-4d28-ab29-9444a277cf8c-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-747767b756-qxmnn\" (UID: \"aa631cd9-393d-4d28-ab29-9444a277cf8c\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-747767b756-qxmnn"
Sep 08 13:44:02.190120 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:44:02.190092    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:44:02.269723 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:44:02.269700    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65g6d\" (UniqueName: \"kubernetes.io/projected/aa631cd9-393d-4d28-ab29-9444a277cf8c-kube-api-access-65g6d\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-747767b756-qxmnn\" (UID: \"aa631cd9-393d-4d28-ab29-9444a277cf8c\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-747767b756-qxmnn"
Sep 08 13:44:02.269840 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:44:02.269729    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa631cd9-393d-4d28-ab29-9444a277cf8c-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-747767b756-qxmnn\" (UID: \"aa631cd9-393d-4d28-ab29-9444a277cf8c\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-747767b756-qxmnn"
Sep 08 13:44:02.269840 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:44:02.269768    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa631cd9-393d-4d28-ab29-9444a277cf8c-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-747767b756-qxmnn\" (UID: \"aa631cd9-393d-4d28-ab29-9444a277cf8c\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-747767b756-qxmnn"
Sep 08 13:44:02.269840 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:44:02.269787    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa631cd9-393d-4d28-ab29-9444a277cf8c-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-747767b756-qxmnn\" (UID: \"aa631cd9-393d-4d28-ab29-9444a277cf8c\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-747767b756-qxmnn"
Sep 08 13:44:02.269840 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:44:02.269809    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa631cd9-393d-4d28-ab29-9444a277cf8c-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-747767b756-qxmnn\" (UID: \"aa631cd9-393d-4d28-ab29-9444a277cf8c\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-747767b756-qxmnn"
Sep 08 13:44:02.269840 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:44:02.269827    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa631cd9-393d-4d28-ab29-9444a277cf8c-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-747767b756-qxmnn\" (UID: \"aa631cd9-393d-4d28-ab29-9444a277cf8c\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-747767b756-qxmnn"
Sep 08 13:44:02.270202 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:44:02.270183    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa631cd9-393d-4d28-ab29-9444a277cf8c-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-747767b756-qxmnn\" (UID: \"aa631cd9-393d-4d28-ab29-9444a277cf8c\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-747767b756-qxmnn"
Sep 08 13:44:02.270262 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:44:02.270224    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa631cd9-393d-4d28-ab29-9444a277cf8c-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-747767b756-qxmnn\" (UID: \"aa631cd9-393d-4d28-ab29-9444a277cf8c\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-747767b756-qxmnn"
Sep 08 13:44:02.270304 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:44:02.270257    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa631cd9-393d-4d28-ab29-9444a277cf8c-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-747767b756-qxmnn\" (UID: \"aa631cd9-393d-4d28-ab29-9444a277cf8c\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-747767b756-qxmnn"
Sep 08 13:44:02.272095 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:44:02.272077    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa631cd9-393d-4d28-ab29-9444a277cf8c-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-747767b756-qxmnn\" (UID: \"aa631cd9-393d-4d28-ab29-9444a277cf8c\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-747767b756-qxmnn"
Sep 08 13:44:02.272196 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:44:02.272180    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa631cd9-393d-4d28-ab29-9444a277cf8c-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-747767b756-qxmnn\" (UID: \"aa631cd9-393d-4d28-ab29-9444a277cf8c\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-747767b756-qxmnn"
Sep 08 13:44:02.284058 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:44:02.284035    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-65g6d\" (UniqueName: \"kubernetes.io/projected/aa631cd9-393d-4d28-ab29-9444a277cf8c-kube-api-access-65g6d\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-747767b756-qxmnn\" (UID: \"aa631cd9-393d-4d28-ab29-9444a277cf8c\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-747767b756-qxmnn"
Sep 08 13:44:02.374859 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:44:02.374837    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-747767b756-qxmnn"
Sep 08 13:44:02.388166 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:44:02.388145    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75f829f"]
Sep 08 13:44:02.391301 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:44:02.391284    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75f829f"
Sep 08 13:44:02.397834 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:44:02.397812    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-epp-sa-dockercfg-94j2g\""
Sep 08 13:44:02.406025 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:44:02.406006    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75f829f"]
Sep 08 13:44:02.472166 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:44:02.472134    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/398de0d0-ca25-4d34-a34a-50354414177a-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-75f829f\" (UID: \"398de0d0-ca25-4d34-a34a-50354414177a\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75f829f"
Sep 08 13:44:02.472332 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:44:02.472172    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwksc\" (UniqueName: \"kubernetes.io/projected/398de0d0-ca25-4d34-a34a-50354414177a-kube-api-access-bwksc\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-75f829f\" (UID: \"398de0d0-ca25-4d34-a34a-50354414177a\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75f829f"
Sep 08 13:44:02.572759 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:44:02.572730    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/398de0d0-ca25-4d34-a34a-50354414177a-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-75f829f\" (UID: \"398de0d0-ca25-4d34-a34a-50354414177a\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75f829f"
Sep 08 13:44:02.572900 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:44:02.572772    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwksc\" (UniqueName: \"kubernetes.io/projected/398de0d0-ca25-4d34-a34a-50354414177a-kube-api-access-bwksc\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-75f829f\" (UID: \"398de0d0-ca25-4d34-a34a-50354414177a\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75f829f"
Sep 08 13:44:02.575099 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:44:02.575079    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/398de0d0-ca25-4d34-a34a-50354414177a-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-75f829f\" (UID: \"398de0d0-ca25-4d34-a34a-50354414177a\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75f829f"
Sep 08 13:44:02.582784 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:44:02.582766    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwksc\" (UniqueName: \"kubernetes.io/projected/398de0d0-ca25-4d34-a34a-50354414177a-kube-api-access-bwksc\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-75f829f\" (UID: \"398de0d0-ca25-4d34-a34a-50354414177a\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75f829f"
Sep 08 13:44:02.655969 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:44:02.655944    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-747767b756-qxmnn"]
Sep 08 13:44:02.657986 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:44:02.657963    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa631cd9_393d_4d28_ab29_9444a277cf8c.slice/crio-fac7460df5ba2383eab6fe3e470340a151f4ef1aae679559201423b1c407d2ea WatchSource:0}: Error finding container fac7460df5ba2383eab6fe3e470340a151f4ef1aae679559201423b1c407d2ea: Status 404 returned error can't find the container with id fac7460df5ba2383eab6fe3e470340a151f4ef1aae679559201423b1c407d2ea
Sep 08 13:44:02.705416 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:44:02.705390    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75f829f"
Sep 08 13:44:02.903109 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:44:02.903078    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-747767b756-qxmnn" event={"ID":"aa631cd9-393d-4d28-ab29-9444a277cf8c","Type":"ContainerStarted","Data":"4809feeaf35edada7a8615be88a31fc01bb39d42a70288baad866f4738f50060"}
Sep 08 13:44:02.903258 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:44:02.903132    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-747767b756-qxmnn" event={"ID":"aa631cd9-393d-4d28-ab29-9444a277cf8c","Type":"ContainerStarted","Data":"fac7460df5ba2383eab6fe3e470340a151f4ef1aae679559201423b1c407d2ea"}
Sep 08 13:44:02.922439 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:44:02.922362    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-747767b756-qxmnn" podStartSLOduration=0.922346875 podStartE2EDuration="922.346875ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:44:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:44:02.92055957 +0000 UTC m=+1693.192007515" watchObservedRunningTime="2026-09-08 13:44:02.922346875 +0000 UTC m=+1693.193794806"
Sep 08 13:44:02.977973 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:44:02.977946    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75f829f"]
Sep 08 13:44:02.981217 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:44:02.981183    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod398de0d0_ca25_4d34_a34a_50354414177a.slice/crio-efaa38cf7d5c21d73dde09175e9defd87008f3bb3731e428dd326b5886af29ea WatchSource:0}: Error finding container efaa38cf7d5c21d73dde09175e9defd87008f3bb3731e428dd326b5886af29ea: Status 404 returned error can't find the container with id efaa38cf7d5c21d73dde09175e9defd87008f3bb3731e428dd326b5886af29ea
Sep 08 13:44:03.304801 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:44:03.304687    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:44:03.375975 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:44:03.375948    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-747767b756-qxmnn"
Sep 08 13:44:03.908332 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:44:03.908297    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75f829f" event={"ID":"398de0d0-ca25-4d34-a34a-50354414177a","Type":"ContainerStarted","Data":"8d0ea7b5994167459dad0dd2f26519746e4eb041efac05bada0a068072e780ab"}
Sep 08 13:44:03.908517 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:44:03.908346    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75f829f" event={"ID":"398de0d0-ca25-4d34-a34a-50354414177a","Type":"ContainerStarted","Data":"efaa38cf7d5c21d73dde09175e9defd87008f3bb3731e428dd326b5886af29ea"}
Sep 08 13:44:03.908578 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:44:03.908553    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75f829f"
Sep 08 13:44:03.933610 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:44:03.933556    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75f829f" podStartSLOduration=1.933541739 podStartE2EDuration="1.933541739s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:44:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:44:03.932414175 +0000 UTC m=+1694.203862131" watchObservedRunningTime="2026-09-08 13:44:03.933541739 +0000 UTC m=+1694.204989670"
Sep 08 13:44:07.558611 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:44:07.558583    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:44:08.558363 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:44:08.558328    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:44:08.691200 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:44:08.691158    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:44:12.375550 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:44:12.375517    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-747767b756-qxmnn"
Sep 08 13:44:12.388098 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:44:12.388074    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-747767b756-qxmnn"
Sep 08 13:44:12.958014 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:44:12.957983    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-747767b756-qxmnn"
Sep 08 13:44:20.072045 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:44:20.072011    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:44:20.172715 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:44:20.172681    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:44:20.280066 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:44:20.280037    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:44:25.957052 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:44:25.957017    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:44:28.411684 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:44:28.411651    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:44:33.976643 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:44:33.976607    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:44:34.342877 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:44:34.342771    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:44:34.469458 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:44:34.469418    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:44:34.915318 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:44:34.915288    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75f829f"
Sep 08 13:45:00.307039 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:45:00.306998    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75f829f"]
Sep 08 13:45:00.322596 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:45:00.322563    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-747767b756-qxmnn"]
Sep 08 13:45:00.329368 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:45:00.329333    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:45:00.330450 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:45:00.330418    2573 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 08 13:45:00.330450 ip-10-0-132-254 kubenswrapper[2573]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 08 13:45:00.330450 ip-10-0-132-254 kubenswrapper[2573]:         , stdout: , stderr: , exit code -1
Sep 08 13:45:00.330450 ip-10-0-132-254 kubenswrapper[2573]:  > containerID="4809feeaf35edada7a8615be88a31fc01bb39d42a70288baad866f4738f50060" cmd=["/bin/sleep","15"]
Sep 08 13:45:00.330653 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:45:00.330460    2573 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-747767b756-qxmnn" message="" err=<
Sep 08 13:45:00.330653 ip-10-0-132-254 kubenswrapper[2573]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 08 13:45:00.330653 ip-10-0-132-254 kubenswrapper[2573]:         , stdout: , stderr: , exit code -1
Sep 08 13:45:00.330653 ip-10-0-132-254 kubenswrapper[2573]:  >
Sep 08 13:45:00.330653 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:45:00.330472    2573 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 08 13:45:00.330653 ip-10-0-132-254 kubenswrapper[2573]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 08 13:45:00.330653 ip-10-0-132-254 kubenswrapper[2573]:         , stdout: , stderr: , exit code -1
Sep 08 13:45:00.330653 ip-10-0-132-254 kubenswrapper[2573]:  > pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-747767b756-qxmnn" podUID="aa631cd9-393d-4d28-ab29-9444a277cf8c" containerName="main" containerID="cri-o://4809feeaf35edada7a8615be88a31fc01bb39d42a70288baad866f4738f50060"
Sep 08 13:45:00.330653 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:45:00.330533    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-747767b756-qxmnn" podUID="aa631cd9-393d-4d28-ab29-9444a277cf8c" containerName="main" containerID="cri-o://4809feeaf35edada7a8615be88a31fc01bb39d42a70288baad866f4738f50060" gracePeriod=60
Sep 08 13:45:00.441153 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:45:00.441121    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:45:00.673959 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:45:00.673936    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-747767b756-qxmnn"
Sep 08 13:45:00.757164 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:45:00.757131    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa631cd9-393d-4d28-ab29-9444a277cf8c-dshm\") pod \"aa631cd9-393d-4d28-ab29-9444a277cf8c\" (UID: \"aa631cd9-393d-4d28-ab29-9444a277cf8c\") "
Sep 08 13:45:00.757337 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:45:00.757191    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa631cd9-393d-4d28-ab29-9444a277cf8c-home\") pod \"aa631cd9-393d-4d28-ab29-9444a277cf8c\" (UID: \"aa631cd9-393d-4d28-ab29-9444a277cf8c\") "
Sep 08 13:45:00.757337 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:45:00.757267    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa631cd9-393d-4d28-ab29-9444a277cf8c-tls-certs\") pod \"aa631cd9-393d-4d28-ab29-9444a277cf8c\" (UID: \"aa631cd9-393d-4d28-ab29-9444a277cf8c\") "
Sep 08 13:45:00.757337 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:45:00.757296    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa631cd9-393d-4d28-ab29-9444a277cf8c-tmp-dir\") pod \"aa631cd9-393d-4d28-ab29-9444a277cf8c\" (UID: \"aa631cd9-393d-4d28-ab29-9444a277cf8c\") "
Sep 08 13:45:00.757337 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:45:00.757326    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa631cd9-393d-4d28-ab29-9444a277cf8c-model-cache\") pod \"aa631cd9-393d-4d28-ab29-9444a277cf8c\" (UID: \"aa631cd9-393d-4d28-ab29-9444a277cf8c\") "
Sep 08 13:45:00.757602 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:45:00.757365    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-65g6d\" (UniqueName: \"kubernetes.io/projected/aa631cd9-393d-4d28-ab29-9444a277cf8c-kube-api-access-65g6d\") pod \"aa631cd9-393d-4d28-ab29-9444a277cf8c\" (UID: \"aa631cd9-393d-4d28-ab29-9444a277cf8c\") "
Sep 08 13:45:00.757602 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:45:00.757534    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa631cd9-393d-4d28-ab29-9444a277cf8c-home" (OuterVolumeSpecName: "home") pod "aa631cd9-393d-4d28-ab29-9444a277cf8c" (UID: "aa631cd9-393d-4d28-ab29-9444a277cf8c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:45:00.757736 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:45:00.757608    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa631cd9-393d-4d28-ab29-9444a277cf8c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "aa631cd9-393d-4d28-ab29-9444a277cf8c" (UID: "aa631cd9-393d-4d28-ab29-9444a277cf8c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:45:00.757736 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:45:00.757625    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa631cd9-393d-4d28-ab29-9444a277cf8c-model-cache" (OuterVolumeSpecName: "model-cache") pod "aa631cd9-393d-4d28-ab29-9444a277cf8c" (UID: "aa631cd9-393d-4d28-ab29-9444a277cf8c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:45:00.759419 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:45:00.759399    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa631cd9-393d-4d28-ab29-9444a277cf8c-dshm" (OuterVolumeSpecName: "dshm") pod "aa631cd9-393d-4d28-ab29-9444a277cf8c" (UID: "aa631cd9-393d-4d28-ab29-9444a277cf8c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:45:00.759516 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:45:00.759475    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa631cd9-393d-4d28-ab29-9444a277cf8c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "aa631cd9-393d-4d28-ab29-9444a277cf8c" (UID: "aa631cd9-393d-4d28-ab29-9444a277cf8c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:45:00.759563 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:45:00.759537    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa631cd9-393d-4d28-ab29-9444a277cf8c-kube-api-access-65g6d" (OuterVolumeSpecName: "kube-api-access-65g6d") pod "aa631cd9-393d-4d28-ab29-9444a277cf8c" (UID: "aa631cd9-393d-4d28-ab29-9444a277cf8c"). InnerVolumeSpecName "kube-api-access-65g6d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:45:00.858875 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:45:00.858810    2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa631cd9-393d-4d28-ab29-9444a277cf8c-dshm\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 13:45:00.858875 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:45:00.858832    2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa631cd9-393d-4d28-ab29-9444a277cf8c-home\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 13:45:00.858875 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:45:00.858840    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa631cd9-393d-4d28-ab29-9444a277cf8c-tls-certs\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 13:45:00.858875 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:45:00.858850    2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa631cd9-393d-4d28-ab29-9444a277cf8c-tmp-dir\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 13:45:00.858875 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:45:00.858860    2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa631cd9-393d-4d28-ab29-9444a277cf8c-model-cache\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 13:45:00.858875 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:45:00.858868    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65g6d\" (UniqueName: \"kubernetes.io/projected/aa631cd9-393d-4d28-ab29-9444a277cf8c-kube-api-access-65g6d\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 13:45:01.143130 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:45:01.143096    2573 generic.go:356] "Generic (PLEG): container finished" podID="aa631cd9-393d-4d28-ab29-9444a277cf8c" containerID="4809feeaf35edada7a8615be88a31fc01bb39d42a70288baad866f4738f50060" exitCode=0
Sep 08 13:45:01.143298 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:45:01.143141    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-747767b756-qxmnn"
Sep 08 13:45:01.143298 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:45:01.143155    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-747767b756-qxmnn" event={"ID":"aa631cd9-393d-4d28-ab29-9444a277cf8c","Type":"ContainerDied","Data":"4809feeaf35edada7a8615be88a31fc01bb39d42a70288baad866f4738f50060"}
Sep 08 13:45:01.143298 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:45:01.143197    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-747767b756-qxmnn" event={"ID":"aa631cd9-393d-4d28-ab29-9444a277cf8c","Type":"ContainerDied","Data":"fac7460df5ba2383eab6fe3e470340a151f4ef1aae679559201423b1c407d2ea"}
Sep 08 13:45:01.143298 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:45:01.143218    2573 scope.go:117] "RemoveContainer" containerID="4809feeaf35edada7a8615be88a31fc01bb39d42a70288baad866f4738f50060"
Sep 08 13:45:01.151777 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:45:01.151758    2573 scope.go:117] "RemoveContainer" containerID="4809feeaf35edada7a8615be88a31fc01bb39d42a70288baad866f4738f50060"
Sep 08 13:45:01.152058 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:45:01.152035    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4809feeaf35edada7a8615be88a31fc01bb39d42a70288baad866f4738f50060\": container with ID starting with 4809feeaf35edada7a8615be88a31fc01bb39d42a70288baad866f4738f50060 not found: ID does not exist" containerID="4809feeaf35edada7a8615be88a31fc01bb39d42a70288baad866f4738f50060"
Sep 08 13:45:01.152122 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:45:01.152072    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4809feeaf35edada7a8615be88a31fc01bb39d42a70288baad866f4738f50060"} err="failed to get container status \"4809feeaf35edada7a8615be88a31fc01bb39d42a70288baad866f4738f50060\": rpc error: code = NotFound desc = could not find container \"4809feeaf35edada7a8615be88a31fc01bb39d42a70288baad866f4738f50060\": container with ID starting with 4809feeaf35edada7a8615be88a31fc01bb39d42a70288baad866f4738f50060 not found: ID does not exist"
Sep 08 13:45:01.166206 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:45:01.166182    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-747767b756-qxmnn"]
Sep 08 13:45:01.171005 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:45:01.170981    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-747767b756-qxmnn"]
Sep 08 13:45:02.174619 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:45:02.174587    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aa631cd9-393d-4d28-ab29-9444a277cf8c" path="/var/lib/kubelet/pods/aa631cd9-393d-4d28-ab29-9444a277cf8c/volumes"
Sep 08 13:45:04.914976 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:45:04.914940    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75f829f" podUID="398de0d0-ca25-4d34-a34a-50354414177a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 08 13:45:14.915311 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:45:14.915269    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75f829f" podUID="398de0d0-ca25-4d34-a34a-50354414177a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 08 13:45:15.326710 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:45:15.326604    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75f829f" podUID="398de0d0-ca25-4d34-a34a-50354414177a" containerName="main" containerID="cri-o://8d0ea7b5994167459dad0dd2f26519746e4eb041efac05bada0a068072e780ab" gracePeriod=45
Sep 08 13:45:24.914747 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:45:24.914705    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75f829f" podUID="398de0d0-ca25-4d34-a34a-50354414177a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 08 13:45:24.915218 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:45:24.914835    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75f829f"
Sep 08 13:45:34.914626 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:45:34.914588    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75f829f" podUID="398de0d0-ca25-4d34-a34a-50354414177a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 08 13:45:44.914699 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:45:44.914609    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75f829f" podUID="398de0d0-ca25-4d34-a34a-50354414177a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 08 13:45:45.567219 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:45:45.567191    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75f829f"
Sep 08 13:45:45.584463 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:45:45.584397    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bwksc\" (UniqueName: \"kubernetes.io/projected/398de0d0-ca25-4d34-a34a-50354414177a-kube-api-access-bwksc\") pod \"398de0d0-ca25-4d34-a34a-50354414177a\" (UID: \"398de0d0-ca25-4d34-a34a-50354414177a\") "
Sep 08 13:45:45.584463 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:45:45.584453    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/398de0d0-ca25-4d34-a34a-50354414177a-tls-certs\") pod \"398de0d0-ca25-4d34-a34a-50354414177a\" (UID: \"398de0d0-ca25-4d34-a34a-50354414177a\") "
Sep 08 13:45:45.586441 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:45:45.586412    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/398de0d0-ca25-4d34-a34a-50354414177a-kube-api-access-bwksc" (OuterVolumeSpecName: "kube-api-access-bwksc") pod "398de0d0-ca25-4d34-a34a-50354414177a" (UID: "398de0d0-ca25-4d34-a34a-50354414177a"). InnerVolumeSpecName "kube-api-access-bwksc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:45:45.586560 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:45:45.586447    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/398de0d0-ca25-4d34-a34a-50354414177a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "398de0d0-ca25-4d34-a34a-50354414177a" (UID: "398de0d0-ca25-4d34-a34a-50354414177a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:45:45.685874 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:45:45.685837    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/398de0d0-ca25-4d34-a34a-50354414177a-tls-certs\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 13:45:45.685874 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:45:45.685867    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwksc\" (UniqueName: \"kubernetes.io/projected/398de0d0-ca25-4d34-a34a-50354414177a-kube-api-access-bwksc\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 13:45:46.333579 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:45:46.333543    2573 generic.go:356] "Generic (PLEG): container finished" podID="398de0d0-ca25-4d34-a34a-50354414177a" containerID="8d0ea7b5994167459dad0dd2f26519746e4eb041efac05bada0a068072e780ab" exitCode=0
Sep 08 13:45:46.333976 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:45:46.333590    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75f829f" event={"ID":"398de0d0-ca25-4d34-a34a-50354414177a","Type":"ContainerDied","Data":"8d0ea7b5994167459dad0dd2f26519746e4eb041efac05bada0a068072e780ab"}
Sep 08 13:45:46.333976 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:45:46.333603    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75f829f"
Sep 08 13:45:46.333976 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:45:46.333671    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75f829f" event={"ID":"398de0d0-ca25-4d34-a34a-50354414177a","Type":"ContainerDied","Data":"efaa38cf7d5c21d73dde09175e9defd87008f3bb3731e428dd326b5886af29ea"}
Sep 08 13:45:46.333976 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:45:46.333705    2573 scope.go:117] "RemoveContainer" containerID="8d0ea7b5994167459dad0dd2f26519746e4eb041efac05bada0a068072e780ab"
Sep 08 13:45:46.341455 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:45:46.341437    2573 scope.go:117] "RemoveContainer" containerID="8d0ea7b5994167459dad0dd2f26519746e4eb041efac05bada0a068072e780ab"
Sep 08 13:45:46.341786 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:45:46.341766    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d0ea7b5994167459dad0dd2f26519746e4eb041efac05bada0a068072e780ab\": container with ID starting with 8d0ea7b5994167459dad0dd2f26519746e4eb041efac05bada0a068072e780ab not found: ID does not exist" containerID="8d0ea7b5994167459dad0dd2f26519746e4eb041efac05bada0a068072e780ab"
Sep 08 13:45:46.341870 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:45:46.341790    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d0ea7b5994167459dad0dd2f26519746e4eb041efac05bada0a068072e780ab"} err="failed to get container status \"8d0ea7b5994167459dad0dd2f26519746e4eb041efac05bada0a068072e780ab\": rpc error: code = NotFound desc = could not find container \"8d0ea7b5994167459dad0dd2f26519746e4eb041efac05bada0a068072e780ab\": container with ID starting with 8d0ea7b5994167459dad0dd2f26519746e4eb041efac05bada0a068072e780ab not found: ID does not exist"
Sep 08 13:45:46.361441 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:45:46.361413    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75f829f"]
Sep 08 13:45:46.367179 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:45:46.367159    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75f829f"]
Sep 08 13:45:48.174391 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:45:48.174358    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="398de0d0-ca25-4d34-a34a-50354414177a" path="/var/lib/kubelet/pods/398de0d0-ca25-4d34-a34a-50354414177a/volumes"
Sep 08 13:45:50.245384 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:45:50.245355    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8h8cn_b936238e-bdf1-4dac-8bf8-6417c313597c/ovn-acl-logging/0.log"
Sep 08 13:45:50.252660 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:45:50.252637    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8h8cn_b936238e-bdf1-4dac-8bf8-6417c313597c/ovn-acl-logging/0.log"
Sep 08 13:45:50.406101 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:45:50.406038    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:45:50.523980 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:45:50.523898    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:45:57.125208 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:45:57.125172    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:46:02.280178 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:46:02.280145    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b48c7dz98"]
Sep 08 13:46:02.280581 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:46:02.280424    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="398de0d0-ca25-4d34-a34a-50354414177a" containerName="main"
Sep 08 13:46:02.280581 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:46:02.280436    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="398de0d0-ca25-4d34-a34a-50354414177a" containerName="main"
Sep 08 13:46:02.280581 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:46:02.280455    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa631cd9-393d-4d28-ab29-9444a277cf8c" containerName="main"
Sep 08 13:46:02.280581 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:46:02.280461    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa631cd9-393d-4d28-ab29-9444a277cf8c" containerName="main"
Sep 08 13:46:02.280581 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:46:02.280537    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa631cd9-393d-4d28-ab29-9444a277cf8c" containerName="main"
Sep 08 13:46:02.280581 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:46:02.280546    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="398de0d0-ca25-4d34-a34a-50354414177a" containerName="main"
Sep 08 13:46:02.283400 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:46:02.283384    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b48c7dz98"
Sep 08 13:46:02.288049 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:46:02.288025    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"kube-root-ca.crt\""
Sep 08 13:46:02.288201 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:46:02.288025    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"default-dockercfg-t8rsf\""
Sep 08 13:46:02.289144 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:46:02.289121    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"openshift-service-ca.crt\""
Sep 08 13:46:02.289144 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:46:02.289141    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-kserve-self-signed-certs\""
Sep 08 13:46:02.299920 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:46:02.299898    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b48c7dz98"]
Sep 08 13:46:02.313602 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:46:02.313573    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7t2r6\" (UniqueName: \"kubernetes.io/projected/80b10e52-ee4a-4f4e-8d86-b3c9f2e6694d-kube-api-access-7t2r6\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b48c7dz98\" (UID: \"80b10e52-ee4a-4f4e-8d86-b3c9f2e6694d\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b48c7dz98"
Sep 08 13:46:02.313690 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:46:02.313610    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80b10e52-ee4a-4f4e-8d86-b3c9f2e6694d-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b48c7dz98\" (UID: \"80b10e52-ee4a-4f4e-8d86-b3c9f2e6694d\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b48c7dz98"
Sep 08 13:46:02.313690 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:46:02.313642    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80b10e52-ee4a-4f4e-8d86-b3c9f2e6694d-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b48c7dz98\" (UID: \"80b10e52-ee4a-4f4e-8d86-b3c9f2e6694d\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b48c7dz98"
Sep 08 13:46:02.313764 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:46:02.313749    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80b10e52-ee4a-4f4e-8d86-b3c9f2e6694d-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b48c7dz98\" (UID: \"80b10e52-ee4a-4f4e-8d86-b3c9f2e6694d\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b48c7dz98"
Sep 08 13:46:02.313800 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:46:02.313779    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80b10e52-ee4a-4f4e-8d86-b3c9f2e6694d-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b48c7dz98\" (UID: \"80b10e52-ee4a-4f4e-8d86-b3c9f2e6694d\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b48c7dz98"
Sep 08 13:46:02.313837 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:46:02.313799    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80b10e52-ee4a-4f4e-8d86-b3c9f2e6694d-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b48c7dz98\" (UID: \"80b10e52-ee4a-4f4e-8d86-b3c9f2e6694d\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b48c7dz98"
Sep 08 13:46:02.335706 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:46:02.335678    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:46:02.414762 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:46:02.414734    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80b10e52-ee4a-4f4e-8d86-b3c9f2e6694d-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b48c7dz98\" (UID: \"80b10e52-ee4a-4f4e-8d86-b3c9f2e6694d\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b48c7dz98"
Sep 08 13:46:02.414916 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:46:02.414768    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80b10e52-ee4a-4f4e-8d86-b3c9f2e6694d-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b48c7dz98\" (UID: \"80b10e52-ee4a-4f4e-8d86-b3c9f2e6694d\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b48c7dz98"
Sep 08 13:46:02.414916 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:46:02.414785    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80b10e52-ee4a-4f4e-8d86-b3c9f2e6694d-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b48c7dz98\" (UID: \"80b10e52-ee4a-4f4e-8d86-b3c9f2e6694d\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b48c7dz98"
Sep 08 13:46:02.414916 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:46:02.414904    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7t2r6\" (UniqueName: \"kubernetes.io/projected/80b10e52-ee4a-4f4e-8d86-b3c9f2e6694d-kube-api-access-7t2r6\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b48c7dz98\" (UID: \"80b10e52-ee4a-4f4e-8d86-b3c9f2e6694d\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b48c7dz98"
Sep 08 13:46:02.415092 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:46:02.414952    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80b10e52-ee4a-4f4e-8d86-b3c9f2e6694d-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b48c7dz98\" (UID: \"80b10e52-ee4a-4f4e-8d86-b3c9f2e6694d\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b48c7dz98"
Sep 08 13:46:02.415174 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:46:02.415141    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80b10e52-ee4a-4f4e-8d86-b3c9f2e6694d-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b48c7dz98\" (UID: \"80b10e52-ee4a-4f4e-8d86-b3c9f2e6694d\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b48c7dz98"
Sep 08 13:46:02.415174 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:46:02.415160    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80b10e52-ee4a-4f4e-8d86-b3c9f2e6694d-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b48c7dz98\" (UID: \"80b10e52-ee4a-4f4e-8d86-b3c9f2e6694d\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b48c7dz98"
Sep 08 13:46:02.415298 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:46:02.415199    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80b10e52-ee4a-4f4e-8d86-b3c9f2e6694d-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b48c7dz98\" (UID: \"80b10e52-ee4a-4f4e-8d86-b3c9f2e6694d\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b48c7dz98"
Sep 08 13:46:02.415409 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:46:02.415387    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80b10e52-ee4a-4f4e-8d86-b3c9f2e6694d-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b48c7dz98\" (UID: \"80b10e52-ee4a-4f4e-8d86-b3c9f2e6694d\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b48c7dz98"
Sep 08 13:46:02.417136 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:46:02.417115    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80b10e52-ee4a-4f4e-8d86-b3c9f2e6694d-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b48c7dz98\" (UID: \"80b10e52-ee4a-4f4e-8d86-b3c9f2e6694d\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b48c7dz98"
Sep 08 13:46:02.417318 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:46:02.417302    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80b10e52-ee4a-4f4e-8d86-b3c9f2e6694d-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b48c7dz98\" (UID: \"80b10e52-ee4a-4f4e-8d86-b3c9f2e6694d\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b48c7dz98"
Sep 08 13:46:02.423801 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:46:02.423777    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7t2r6\" (UniqueName: \"kubernetes.io/projected/80b10e52-ee4a-4f4e-8d86-b3c9f2e6694d-kube-api-access-7t2r6\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b48c7dz98\" (UID: \"80b10e52-ee4a-4f4e-8d86-b3c9f2e6694d\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b48c7dz98"
Sep 08 13:46:02.578220 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:46:02.578130    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:46:02.593181 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:46:02.593152    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b48c7dz98"
Sep 08 13:46:02.866189 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:46:02.866161    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b48c7dz98"]
Sep 08 13:46:02.867362 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:46:02.867331    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80b10e52_ee4a_4f4e_8d86_b3c9f2e6694d.slice/crio-a1b2c0a0b154be859d7464e01a6ab14f92d02beda749e96b8e83477cd8aaed12 WatchSource:0}: Error finding container a1b2c0a0b154be859d7464e01a6ab14f92d02beda749e96b8e83477cd8aaed12: Status 404 returned error can't find the container with id a1b2c0a0b154be859d7464e01a6ab14f92d02beda749e96b8e83477cd8aaed12
Sep 08 13:46:03.401784 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:46:03.401746    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b48c7dz98" event={"ID":"80b10e52-ee4a-4f4e-8d86-b3c9f2e6694d","Type":"ContainerStarted","Data":"5662980b40f33a40dca9d4ed1ecd13bea76052bface5c0fe97678811a3e47a92"}
Sep 08 13:46:03.402187 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:46:03.401799    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b48c7dz98" event={"ID":"80b10e52-ee4a-4f4e-8d86-b3c9f2e6694d","Type":"ContainerStarted","Data":"a1b2c0a0b154be859d7464e01a6ab14f92d02beda749e96b8e83477cd8aaed12"}
Sep 08 13:46:03.424516 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:46:03.424459    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b48c7dz98" podStartSLOduration=1.424444576 podStartE2EDuration="1.424444576s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:46:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:46:03.423635647 +0000 UTC m=+1813.695083617" watchObservedRunningTime="2026-09-08 13:46:03.424444576 +0000 UTC m=+1813.695892544"
Sep 08 13:46:03.593671 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:46:03.593633    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b48c7dz98"
Sep 08 13:46:04.934474 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:46:04.934438    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:46:05.038362 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:46:05.038330    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:46:12.593547 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:46:12.593490    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b48c7dz98"
Sep 08 13:46:12.605823 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:46:12.605803    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b48c7dz98"
Sep 08 13:46:13.449859 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:46:13.449830    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b48c7dz98"
Sep 08 13:46:15.535649 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:46:15.535562    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:46:15.662698 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:46:15.662665    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:46:15.778918 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:46:15.778885    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:46:19.339133 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:46:19.339091    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:46:20.598548 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:46:20.598493    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:46:21.410365 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:46:21.410324    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:46:22.437907 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:46:22.437875    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:46:29.317988 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:46:29.317950    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-ctbzv"]
Sep 08 13:46:29.321261 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:46:29.321240    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-ctbzv"
Sep 08 13:46:29.324261 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:46:29.324235    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-model-name-divergence-75f1ef44\"/\"default-dockercfg-zfjv2\""
Sep 08 13:46:29.324261 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:46:29.324246    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-model-name-divergence-75f1ef44\"/\"test-div-v1-kserve-self-signed-certs\""
Sep 08 13:46:29.324437 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:46:29.324353    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-model-name-divergence-75f1ef44\"/\"openshift-service-ca.crt\""
Sep 08 13:46:29.324437 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:46:29.324406    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-model-name-divergence-75f1ef44\"/\"kube-root-ca.crt\""
Sep 08 13:46:29.333224 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:46:29.333202    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-ctbzv"]
Sep 08 13:46:29.425876 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:46:29.425844    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44e55827-3c21-4fb2-8bd4-76bb31b18e46-home\") pod \"test-div-v1-kserve-fb566d667-ctbzv\" (UID: \"44e55827-3c21-4fb2-8bd4-76bb31b18e46\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-ctbzv"
Sep 08 13:46:29.426022 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:46:29.425890    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44e55827-3c21-4fb2-8bd4-76bb31b18e46-dshm\") pod \"test-div-v1-kserve-fb566d667-ctbzv\" (UID: \"44e55827-3c21-4fb2-8bd4-76bb31b18e46\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-ctbzv"
Sep 08 13:46:29.426022 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:46:29.425916    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44e55827-3c21-4fb2-8bd4-76bb31b18e46-tls-certs\") pod \"test-div-v1-kserve-fb566d667-ctbzv\" (UID: \"44e55827-3c21-4fb2-8bd4-76bb31b18e46\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-ctbzv"
Sep 08 13:46:29.426022 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:46:29.425952    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44e55827-3c21-4fb2-8bd4-76bb31b18e46-model-cache\") pod \"test-div-v1-kserve-fb566d667-ctbzv\" (UID: \"44e55827-3c21-4fb2-8bd4-76bb31b18e46\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-ctbzv"
Sep 08 13:46:29.426022 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:46:29.426004    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44e55827-3c21-4fb2-8bd4-76bb31b18e46-tmp-dir\") pod \"test-div-v1-kserve-fb566d667-ctbzv\" (UID: \"44e55827-3c21-4fb2-8bd4-76bb31b18e46\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-ctbzv"
Sep 08 13:46:29.426156 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:46:29.426029    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdthm\" (UniqueName: \"kubernetes.io/projected/44e55827-3c21-4fb2-8bd4-76bb31b18e46-kube-api-access-kdthm\") pod \"test-div-v1-kserve-fb566d667-ctbzv\" (UID: \"44e55827-3c21-4fb2-8bd4-76bb31b18e46\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-ctbzv"
Sep 08 13:46:29.522431 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:46:29.522396    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:46:29.526566 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:46:29.526545    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44e55827-3c21-4fb2-8bd4-76bb31b18e46-tmp-dir\") pod \"test-div-v1-kserve-fb566d667-ctbzv\" (UID: \"44e55827-3c21-4fb2-8bd4-76bb31b18e46\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-ctbzv"
Sep 08 13:46:29.526684 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:46:29.526578    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdthm\" (UniqueName: \"kubernetes.io/projected/44e55827-3c21-4fb2-8bd4-76bb31b18e46-kube-api-access-kdthm\") pod \"test-div-v1-kserve-fb566d667-ctbzv\" (UID: \"44e55827-3c21-4fb2-8bd4-76bb31b18e46\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-ctbzv"
Sep 08 13:46:29.526684 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:46:29.526621    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44e55827-3c21-4fb2-8bd4-76bb31b18e46-home\") pod \"test-div-v1-kserve-fb566d667-ctbzv\" (UID: \"44e55827-3c21-4fb2-8bd4-76bb31b18e46\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-ctbzv"
Sep 08 13:46:29.526684 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:46:29.526654    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44e55827-3c21-4fb2-8bd4-76bb31b18e46-dshm\") pod \"test-div-v1-kserve-fb566d667-ctbzv\" (UID: \"44e55827-3c21-4fb2-8bd4-76bb31b18e46\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-ctbzv"
Sep 08 13:46:29.526684 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:46:29.526669    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44e55827-3c21-4fb2-8bd4-76bb31b18e46-tls-certs\") pod \"test-div-v1-kserve-fb566d667-ctbzv\" (UID: \"44e55827-3c21-4fb2-8bd4-76bb31b18e46\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-ctbzv"
Sep 08 13:46:29.526904 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:46:29.526694    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44e55827-3c21-4fb2-8bd4-76bb31b18e46-model-cache\") pod \"test-div-v1-kserve-fb566d667-ctbzv\" (UID: \"44e55827-3c21-4fb2-8bd4-76bb31b18e46\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-ctbzv"
Sep 08 13:46:29.527018 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:46:29.526998    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44e55827-3c21-4fb2-8bd4-76bb31b18e46-tmp-dir\") pod \"test-div-v1-kserve-fb566d667-ctbzv\" (UID: \"44e55827-3c21-4fb2-8bd4-76bb31b18e46\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-ctbzv"
Sep 08 13:46:29.527075 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:46:29.527048    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44e55827-3c21-4fb2-8bd4-76bb31b18e46-model-cache\") pod \"test-div-v1-kserve-fb566d667-ctbzv\" (UID: \"44e55827-3c21-4fb2-8bd4-76bb31b18e46\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-ctbzv"
Sep 08 13:46:29.527113 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:46:29.527073    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44e55827-3c21-4fb2-8bd4-76bb31b18e46-home\") pod \"test-div-v1-kserve-fb566d667-ctbzv\" (UID: \"44e55827-3c21-4fb2-8bd4-76bb31b18e46\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-ctbzv"
Sep 08 13:46:29.528905 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:46:29.528879    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44e55827-3c21-4fb2-8bd4-76bb31b18e46-dshm\") pod \"test-div-v1-kserve-fb566d667-ctbzv\" (UID: \"44e55827-3c21-4fb2-8bd4-76bb31b18e46\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-ctbzv"
Sep 08 13:46:29.529295 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:46:29.529272    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44e55827-3c21-4fb2-8bd4-76bb31b18e46-tls-certs\") pod \"test-div-v1-kserve-fb566d667-ctbzv\" (UID: \"44e55827-3c21-4fb2-8bd4-76bb31b18e46\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-ctbzv"
Sep 08 13:46:29.537049 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:46:29.537024    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdthm\" (UniqueName: \"kubernetes.io/projected/44e55827-3c21-4fb2-8bd4-76bb31b18e46-kube-api-access-kdthm\") pod \"test-div-v1-kserve-fb566d667-ctbzv\" (UID: \"44e55827-3c21-4fb2-8bd4-76bb31b18e46\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-ctbzv"
Sep 08 13:46:29.631933 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:46:29.631907    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-ctbzv"
Sep 08 13:46:29.913166 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:46:29.913001    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-ctbzv"]
Sep 08 13:46:29.915791 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:46:29.915763    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44e55827_3c21_4fb2_8bd4_76bb31b18e46.slice/crio-a660dea7aec7d1b9b28e0c42e407afa9ee221ce1aff4343b093863f217c55ef8 WatchSource:0}: Error finding container a660dea7aec7d1b9b28e0c42e407afa9ee221ce1aff4343b093863f217c55ef8: Status 404 returned error can't find the container with id a660dea7aec7d1b9b28e0c42e407afa9ee221ce1aff4343b093863f217c55ef8
Sep 08 13:46:29.917686 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:46:29.917671    2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 08 13:46:30.511092 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:46:30.511058    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-ctbzv" event={"ID":"44e55827-3c21-4fb2-8bd4-76bb31b18e46","Type":"ContainerStarted","Data":"a660dea7aec7d1b9b28e0c42e407afa9ee221ce1aff4343b093863f217c55ef8"}
Sep 08 13:46:31.517282 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:46:31.517244    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-ctbzv" event={"ID":"44e55827-3c21-4fb2-8bd4-76bb31b18e46","Type":"ContainerStarted","Data":"2e52b57b9aa4b311392551a9e6756394c1f5f5f767cc7405cd74e2674632be4f"}
Sep 08 13:46:31.538167 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:46:31.538118    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-ctbzv" podStartSLOduration=1.348940316 podStartE2EDuration="2.538103101s" totalImagesPullingTime="1.189162785s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:46:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:46:31.535889281 +0000 UTC m=+1841.807337211" watchObservedRunningTime="2026-09-08 13:46:31.538103101 +0000 UTC m=+1841.809551031"
Sep 08 13:46:31.632434 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:46:31.632402    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-ctbzv"
Sep 08 13:46:35.521159 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:46:35.521126    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:46:36.471265 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:46:36.471229    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:46:39.632594 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:46:39.632559    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-ctbzv"
Sep 08 13:46:39.644837 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:46:39.644811    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-ctbzv"
Sep 08 13:46:40.568901 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:46:40.568872    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-ctbzv"
Sep 08 13:46:41.361131 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:46:41.361096    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:46:57.979761 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:46:57.979731    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:47:04.097056 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:47:04.097017    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:47:04.194310 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:47:04.194280    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:47:12.259029 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:47:12.258993    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:47:12.272236 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:47:12.272207    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b48c7dz98"]
Sep 08 13:47:12.280853 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:47:12.280809    2573 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 08 13:47:12.280853 ip-10-0-132-254 kubenswrapper[2573]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 08 13:47:12.280853 ip-10-0-132-254 kubenswrapper[2573]:         , stdout: , stderr: , exit code -1
Sep 08 13:47:12.280853 ip-10-0-132-254 kubenswrapper[2573]:  > containerID="5662980b40f33a40dca9d4ed1ecd13bea76052bface5c0fe97678811a3e47a92" cmd=["/bin/sleep","15"]
Sep 08 13:47:12.281110 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:47:12.280874    2573 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b48c7dz98" message="" err=<
Sep 08 13:47:12.281110 ip-10-0-132-254 kubenswrapper[2573]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 08 13:47:12.281110 ip-10-0-132-254 kubenswrapper[2573]:         , stdout: , stderr: , exit code -1
Sep 08 13:47:12.281110 ip-10-0-132-254 kubenswrapper[2573]:  >
Sep 08 13:47:12.281110 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:47:12.280892    2573 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 08 13:47:12.281110 ip-10-0-132-254 kubenswrapper[2573]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 08 13:47:12.281110 ip-10-0-132-254 kubenswrapper[2573]:         , stdout: , stderr: , exit code -1
Sep 08 13:47:12.281110 ip-10-0-132-254 kubenswrapper[2573]:  > pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b48c7dz98" podUID="80b10e52-ee4a-4f4e-8d86-b3c9f2e6694d" containerName="main" containerID="cri-o://5662980b40f33a40dca9d4ed1ecd13bea76052bface5c0fe97678811a3e47a92"
Sep 08 13:47:12.281110 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:47:12.280947    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b48c7dz98" podUID="80b10e52-ee4a-4f4e-8d86-b3c9f2e6694d" containerName="main" containerID="cri-o://5662980b40f33a40dca9d4ed1ecd13bea76052bface5c0fe97678811a3e47a92" gracePeriod=60
Sep 08 13:47:12.343595 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:47:12.343572    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:47:12.439587 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:47:12.439557    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b48c7dz98" podUID="80b10e52-ee4a-4f4e-8d86-b3c9f2e6694d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/health\": dial tcp 10.133.0.30:8000: connect: connection refused"
Sep 08 13:47:12.620979 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:47:12.620955    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b48c7dz98"
Sep 08 13:47:12.650026 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:47:12.650000    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80b10e52-ee4a-4f4e-8d86-b3c9f2e6694d-home\") pod \"80b10e52-ee4a-4f4e-8d86-b3c9f2e6694d\" (UID: \"80b10e52-ee4a-4f4e-8d86-b3c9f2e6694d\") "
Sep 08 13:47:12.650167 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:47:12.650061    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80b10e52-ee4a-4f4e-8d86-b3c9f2e6694d-tmp-dir\") pod \"80b10e52-ee4a-4f4e-8d86-b3c9f2e6694d\" (UID: \"80b10e52-ee4a-4f4e-8d86-b3c9f2e6694d\") "
Sep 08 13:47:12.650167 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:47:12.650120    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7t2r6\" (UniqueName: \"kubernetes.io/projected/80b10e52-ee4a-4f4e-8d86-b3c9f2e6694d-kube-api-access-7t2r6\") pod \"80b10e52-ee4a-4f4e-8d86-b3c9f2e6694d\" (UID: \"80b10e52-ee4a-4f4e-8d86-b3c9f2e6694d\") "
Sep 08 13:47:12.650167 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:47:12.650153    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80b10e52-ee4a-4f4e-8d86-b3c9f2e6694d-dshm\") pod \"80b10e52-ee4a-4f4e-8d86-b3c9f2e6694d\" (UID: \"80b10e52-ee4a-4f4e-8d86-b3c9f2e6694d\") "
Sep 08 13:47:12.650337 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:47:12.650196    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80b10e52-ee4a-4f4e-8d86-b3c9f2e6694d-model-cache\") pod \"80b10e52-ee4a-4f4e-8d86-b3c9f2e6694d\" (UID: \"80b10e52-ee4a-4f4e-8d86-b3c9f2e6694d\") "
Sep 08 13:47:12.650337 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:47:12.650229    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80b10e52-ee4a-4f4e-8d86-b3c9f2e6694d-tls-certs\") pod \"80b10e52-ee4a-4f4e-8d86-b3c9f2e6694d\" (UID: \"80b10e52-ee4a-4f4e-8d86-b3c9f2e6694d\") "
Sep 08 13:47:12.650337 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:47:12.650272    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80b10e52-ee4a-4f4e-8d86-b3c9f2e6694d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "80b10e52-ee4a-4f4e-8d86-b3c9f2e6694d" (UID: "80b10e52-ee4a-4f4e-8d86-b3c9f2e6694d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:47:12.650337 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:47:12.650283    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80b10e52-ee4a-4f4e-8d86-b3c9f2e6694d-home" (OuterVolumeSpecName: "home") pod "80b10e52-ee4a-4f4e-8d86-b3c9f2e6694d" (UID: "80b10e52-ee4a-4f4e-8d86-b3c9f2e6694d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:47:12.650581 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:47:12.650518    2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80b10e52-ee4a-4f4e-8d86-b3c9f2e6694d-home\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 13:47:12.650581 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:47:12.650555    2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80b10e52-ee4a-4f4e-8d86-b3c9f2e6694d-tmp-dir\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 13:47:12.650581 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:47:12.650552    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80b10e52-ee4a-4f4e-8d86-b3c9f2e6694d-model-cache" (OuterVolumeSpecName: "model-cache") pod "80b10e52-ee4a-4f4e-8d86-b3c9f2e6694d" (UID: "80b10e52-ee4a-4f4e-8d86-b3c9f2e6694d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:47:12.652391 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:47:12.652368    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80b10e52-ee4a-4f4e-8d86-b3c9f2e6694d-dshm" (OuterVolumeSpecName: "dshm") pod "80b10e52-ee4a-4f4e-8d86-b3c9f2e6694d" (UID: "80b10e52-ee4a-4f4e-8d86-b3c9f2e6694d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:47:12.652681 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:47:12.652660    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80b10e52-ee4a-4f4e-8d86-b3c9f2e6694d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "80b10e52-ee4a-4f4e-8d86-b3c9f2e6694d" (UID: "80b10e52-ee4a-4f4e-8d86-b3c9f2e6694d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:47:12.652743 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:47:12.652681    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80b10e52-ee4a-4f4e-8d86-b3c9f2e6694d-kube-api-access-7t2r6" (OuterVolumeSpecName: "kube-api-access-7t2r6") pod "80b10e52-ee4a-4f4e-8d86-b3c9f2e6694d" (UID: "80b10e52-ee4a-4f4e-8d86-b3c9f2e6694d"). InnerVolumeSpecName "kube-api-access-7t2r6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:47:12.693948 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:47:12.693925    2573 generic.go:356] "Generic (PLEG): container finished" podID="80b10e52-ee4a-4f4e-8d86-b3c9f2e6694d" containerID="5662980b40f33a40dca9d4ed1ecd13bea76052bface5c0fe97678811a3e47a92" exitCode=0
Sep 08 13:47:12.694043 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:47:12.693968    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b48c7dz98"
Sep 08 13:47:12.694043 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:47:12.694007    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b48c7dz98" event={"ID":"80b10e52-ee4a-4f4e-8d86-b3c9f2e6694d","Type":"ContainerDied","Data":"5662980b40f33a40dca9d4ed1ecd13bea76052bface5c0fe97678811a3e47a92"}
Sep 08 13:47:12.694139 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:47:12.694060    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b48c7dz98" event={"ID":"80b10e52-ee4a-4f4e-8d86-b3c9f2e6694d","Type":"ContainerDied","Data":"a1b2c0a0b154be859d7464e01a6ab14f92d02beda749e96b8e83477cd8aaed12"}
Sep 08 13:47:12.694139 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:47:12.694083    2573 scope.go:117] "RemoveContainer" containerID="5662980b40f33a40dca9d4ed1ecd13bea76052bface5c0fe97678811a3e47a92"
Sep 08 13:47:12.702408 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:47:12.702391    2573 scope.go:117] "RemoveContainer" containerID="5662980b40f33a40dca9d4ed1ecd13bea76052bface5c0fe97678811a3e47a92"
Sep 08 13:47:12.702721 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:47:12.702702    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5662980b40f33a40dca9d4ed1ecd13bea76052bface5c0fe97678811a3e47a92\": container with ID starting with 5662980b40f33a40dca9d4ed1ecd13bea76052bface5c0fe97678811a3e47a92 not found: ID does not exist" containerID="5662980b40f33a40dca9d4ed1ecd13bea76052bface5c0fe97678811a3e47a92"
Sep 08 13:47:12.702787 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:47:12.702729    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5662980b40f33a40dca9d4ed1ecd13bea76052bface5c0fe97678811a3e47a92"} err="failed to get container status \"5662980b40f33a40dca9d4ed1ecd13bea76052bface5c0fe97678811a3e47a92\": rpc error: code = NotFound desc = could not find container \"5662980b40f33a40dca9d4ed1ecd13bea76052bface5c0fe97678811a3e47a92\": container with ID starting with 5662980b40f33a40dca9d4ed1ecd13bea76052bface5c0fe97678811a3e47a92 not found: ID does not exist"
Sep 08 13:47:12.716244 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:47:12.716223    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b48c7dz98"]
Sep 08 13:47:12.721464 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:47:12.721445    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b48c7dz98"]
Sep 08 13:47:12.751534 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:47:12.751515    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7t2r6\" (UniqueName: \"kubernetes.io/projected/80b10e52-ee4a-4f4e-8d86-b3c9f2e6694d-kube-api-access-7t2r6\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 13:47:12.751534 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:47:12.751532    2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80b10e52-ee4a-4f4e-8d86-b3c9f2e6694d-dshm\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 13:47:12.751646 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:47:12.751542    2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80b10e52-ee4a-4f4e-8d86-b3c9f2e6694d-model-cache\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 13:47:12.751646 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:47:12.751550    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80b10e52-ee4a-4f4e-8d86-b3c9f2e6694d-tls-certs\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 13:47:14.175031 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:47:14.175003    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="80b10e52-ee4a-4f4e-8d86-b3c9f2e6694d" path="/var/lib/kubelet/pods/80b10e52-ee4a-4f4e-8d86-b3c9f2e6694d/volumes"
Sep 08 13:47:16.291801 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:47:16.291771    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:47:16.396793 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:47:16.396759    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:47:16.424914 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:47:16.424883    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-ctbzv"]
Sep 08 13:47:16.689565 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:47:16.689527    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:47:16.835452 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:47:16.835410    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:47:21.344216 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:47:21.344183    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:47:28.372356 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:47:28.372330    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:47:31.443999 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:47:31.443928    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-ctbzv" podUID="44e55827-3c21-4fb2-8bd4-76bb31b18e46" containerName="main" containerID="cri-o://2e52b57b9aa4b311392551a9e6756394c1f5f5f767cc7405cd74e2674632be4f" gracePeriod=45
Sep 08 13:47:31.558738 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:47:31.558705    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-ctbzv" podUID="44e55827-3c21-4fb2-8bd4-76bb31b18e46" containerName="main" probeResult="failure" output="Get \"https://10.133.0.31:8000/health\": dial tcp 10.133.0.31:8000: connect: connection refused"
Sep 08 13:47:31.683704 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:47:31.683682    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-ctbzv"
Sep 08 13:47:31.770849 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:47:31.770778    2573 generic.go:356] "Generic (PLEG): container finished" podID="44e55827-3c21-4fb2-8bd4-76bb31b18e46" containerID="2e52b57b9aa4b311392551a9e6756394c1f5f5f767cc7405cd74e2674632be4f" exitCode=0
Sep 08 13:47:31.770849 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:47:31.770822    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-ctbzv"
Sep 08 13:47:31.770849 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:47:31.770838    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-ctbzv" event={"ID":"44e55827-3c21-4fb2-8bd4-76bb31b18e46","Type":"ContainerDied","Data":"2e52b57b9aa4b311392551a9e6756394c1f5f5f767cc7405cd74e2674632be4f"}
Sep 08 13:47:31.771046 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:47:31.770904    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-ctbzv" event={"ID":"44e55827-3c21-4fb2-8bd4-76bb31b18e46","Type":"ContainerDied","Data":"a660dea7aec7d1b9b28e0c42e407afa9ee221ce1aff4343b093863f217c55ef8"}
Sep 08 13:47:31.771046 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:47:31.770931    2573 scope.go:117] "RemoveContainer" containerID="2e52b57b9aa4b311392551a9e6756394c1f5f5f767cc7405cd74e2674632be4f"
Sep 08 13:47:31.778830 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:47:31.778810    2573 scope.go:117] "RemoveContainer" containerID="2e52b57b9aa4b311392551a9e6756394c1f5f5f767cc7405cd74e2674632be4f"
Sep 08 13:47:31.779114 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:47:31.779096    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e52b57b9aa4b311392551a9e6756394c1f5f5f767cc7405cd74e2674632be4f\": container with ID starting with 2e52b57b9aa4b311392551a9e6756394c1f5f5f767cc7405cd74e2674632be4f not found: ID does not exist" containerID="2e52b57b9aa4b311392551a9e6756394c1f5f5f767cc7405cd74e2674632be4f"
Sep 08 13:47:31.779182 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:47:31.779124    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e52b57b9aa4b311392551a9e6756394c1f5f5f767cc7405cd74e2674632be4f"} err="failed to get container status \"2e52b57b9aa4b311392551a9e6756394c1f5f5f767cc7405cd74e2674632be4f\": rpc error: code = NotFound desc = could not find container \"2e52b57b9aa4b311392551a9e6756394c1f5f5f767cc7405cd74e2674632be4f\": container with ID starting with 2e52b57b9aa4b311392551a9e6756394c1f5f5f767cc7405cd74e2674632be4f not found: ID does not exist"
Sep 08 13:47:31.799398 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:47:31.799376    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44e55827-3c21-4fb2-8bd4-76bb31b18e46-model-cache\") pod \"44e55827-3c21-4fb2-8bd4-76bb31b18e46\" (UID: \"44e55827-3c21-4fb2-8bd4-76bb31b18e46\") "
Sep 08 13:47:31.799464 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:47:31.799412    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44e55827-3c21-4fb2-8bd4-76bb31b18e46-home\") pod \"44e55827-3c21-4fb2-8bd4-76bb31b18e46\" (UID: \"44e55827-3c21-4fb2-8bd4-76bb31b18e46\") "
Sep 08 13:47:31.799527 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:47:31.799464    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44e55827-3c21-4fb2-8bd4-76bb31b18e46-dshm\") pod \"44e55827-3c21-4fb2-8bd4-76bb31b18e46\" (UID: \"44e55827-3c21-4fb2-8bd4-76bb31b18e46\") "
Sep 08 13:47:31.799527 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:47:31.799481    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kdthm\" (UniqueName: \"kubernetes.io/projected/44e55827-3c21-4fb2-8bd4-76bb31b18e46-kube-api-access-kdthm\") pod \"44e55827-3c21-4fb2-8bd4-76bb31b18e46\" (UID: \"44e55827-3c21-4fb2-8bd4-76bb31b18e46\") "
Sep 08 13:47:31.799641 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:47:31.799528    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44e55827-3c21-4fb2-8bd4-76bb31b18e46-tmp-dir\") pod \"44e55827-3c21-4fb2-8bd4-76bb31b18e46\" (UID: \"44e55827-3c21-4fb2-8bd4-76bb31b18e46\") "
Sep 08 13:47:31.799641 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:47:31.799565    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44e55827-3c21-4fb2-8bd4-76bb31b18e46-tls-certs\") pod \"44e55827-3c21-4fb2-8bd4-76bb31b18e46\" (UID: \"44e55827-3c21-4fb2-8bd4-76bb31b18e46\") "
Sep 08 13:47:31.799745 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:47:31.799652    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44e55827-3c21-4fb2-8bd4-76bb31b18e46-model-cache" (OuterVolumeSpecName: "model-cache") pod "44e55827-3c21-4fb2-8bd4-76bb31b18e46" (UID: "44e55827-3c21-4fb2-8bd4-76bb31b18e46"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:47:31.799745 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:47:31.799727    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44e55827-3c21-4fb2-8bd4-76bb31b18e46-home" (OuterVolumeSpecName: "home") pod "44e55827-3c21-4fb2-8bd4-76bb31b18e46" (UID: "44e55827-3c21-4fb2-8bd4-76bb31b18e46"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:47:31.799856 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:47:31.799807    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44e55827-3c21-4fb2-8bd4-76bb31b18e46-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "44e55827-3c21-4fb2-8bd4-76bb31b18e46" (UID: "44e55827-3c21-4fb2-8bd4-76bb31b18e46"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:47:31.801557 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:47:31.801530    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44e55827-3c21-4fb2-8bd4-76bb31b18e46-dshm" (OuterVolumeSpecName: "dshm") pod "44e55827-3c21-4fb2-8bd4-76bb31b18e46" (UID: "44e55827-3c21-4fb2-8bd4-76bb31b18e46"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:47:31.801640 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:47:31.801584    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44e55827-3c21-4fb2-8bd4-76bb31b18e46-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "44e55827-3c21-4fb2-8bd4-76bb31b18e46" (UID: "44e55827-3c21-4fb2-8bd4-76bb31b18e46"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:47:31.801734 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:47:31.801715    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44e55827-3c21-4fb2-8bd4-76bb31b18e46-kube-api-access-kdthm" (OuterVolumeSpecName: "kube-api-access-kdthm") pod "44e55827-3c21-4fb2-8bd4-76bb31b18e46" (UID: "44e55827-3c21-4fb2-8bd4-76bb31b18e46"). InnerVolumeSpecName "kube-api-access-kdthm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:47:31.900246 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:47:31.900223    2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44e55827-3c21-4fb2-8bd4-76bb31b18e46-dshm\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 13:47:31.900246 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:47:31.900244    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdthm\" (UniqueName: \"kubernetes.io/projected/44e55827-3c21-4fb2-8bd4-76bb31b18e46-kube-api-access-kdthm\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 13:47:31.900377 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:47:31.900270    2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44e55827-3c21-4fb2-8bd4-76bb31b18e46-tmp-dir\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 13:47:31.900377 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:47:31.900281    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44e55827-3c21-4fb2-8bd4-76bb31b18e46-tls-certs\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 13:47:31.900377 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:47:31.900288    2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44e55827-3c21-4fb2-8bd4-76bb31b18e46-model-cache\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 13:47:31.900377 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:47:31.900296    2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44e55827-3c21-4fb2-8bd4-76bb31b18e46-home\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 13:47:32.094979 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:47:32.094947    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-ctbzv"]
Sep 08 13:47:32.099056 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:47:32.099030    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-ctbzv"]
Sep 08 13:47:32.174547 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:47:32.174519    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="44e55827-3c21-4fb2-8bd4-76bb31b18e46" path="/var/lib/kubelet/pods/44e55827-3c21-4fb2-8bd4-76bb31b18e46/volumes"
Sep 08 13:47:39.112264 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:47:39.112232    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:47:59.017233 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:47:59.017199    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:48:03.328366 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:48:03.328335    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:48:03.428717 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:48:03.428684    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:48:05.288546 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:48:05.288494    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:48:05.388491 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:48:05.388461    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:48:06.087035 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:48:06.087000    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:48:06.233021 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:48:06.232986    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:48:06.330135 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:48:06.330108    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:48:07.165665 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:48:07.165629    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:48:07.275562 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:48:07.275529    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:48:09.692513 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:48:09.692467    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:48:11.140611 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:48:11.140579    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:48:13.717984 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:48:13.717943    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:48:15.599713 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:48:15.599676    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:48:15.671660 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:48:15.671628    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:48:18.453183 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:48:18.453149    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-559469c4b5-s2258"]
Sep 08 13:48:18.453576 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:48:18.453461    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80b10e52-ee4a-4f4e-8d86-b3c9f2e6694d" containerName="main"
Sep 08 13:48:18.453576 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:48:18.453474    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="80b10e52-ee4a-4f4e-8d86-b3c9f2e6694d" containerName="main"
Sep 08 13:48:18.453576 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:48:18.453487    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44e55827-3c21-4fb2-8bd4-76bb31b18e46" containerName="main"
Sep 08 13:48:18.453576 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:48:18.453495    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="44e55827-3c21-4fb2-8bd4-76bb31b18e46" containerName="main"
Sep 08 13:48:18.453576 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:48:18.453575    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="80b10e52-ee4a-4f4e-8d86-b3c9f2e6694d" containerName="main"
Sep 08 13:48:18.453747 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:48:18.453587    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="44e55827-3c21-4fb2-8bd4-76bb31b18e46" containerName="main"
Sep 08 13:48:18.456649 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:48:18.456633    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-559469c4b5-s2258"
Sep 08 13:48:18.459307 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:48:18.459282    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"default-dockercfg-pbbdj\""
Sep 08 13:48:18.460216 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:48:18.460134    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"odh-kserve-custom-ca-bundle\""
Sep 08 13:48:18.460216 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:48:18.460176    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"openshift-service-ca.crt\""
Sep 08 13:48:18.460398 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:48:18.460176    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"seaweedfs-s3-creds\""
Sep 08 13:48:18.460611 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:48:18.460591    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"tokenizer-clean-path-test-kserve-self-signed-certs\""
Sep 08 13:48:18.460723 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:48:18.460619    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"kube-root-ca.crt\""
Sep 08 13:48:18.472307 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:48:18.472281    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-559469c4b5-s2258"]
Sep 08 13:48:18.535692 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:48:18.535661    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:48:18.540103 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:48:18.540072    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c33c72a9-0a05-4de7-b63b-6c7e52e88c89-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-559469c4b5-s2258\" (UID: \"c33c72a9-0a05-4de7-b63b-6c7e52e88c89\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-559469c4b5-s2258"
Sep 08 13:48:18.540244 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:48:18.540125    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c87v7\" (UniqueName: \"kubernetes.io/projected/c33c72a9-0a05-4de7-b63b-6c7e52e88c89-kube-api-access-c87v7\") pod \"tokenizer-clean-path-test-kserve-559469c4b5-s2258\" (UID: \"c33c72a9-0a05-4de7-b63b-6c7e52e88c89\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-559469c4b5-s2258"
Sep 08 13:48:18.540244 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:48:18.540231    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c33c72a9-0a05-4de7-b63b-6c7e52e88c89-tls-certs\") pod \"tokenizer-clean-path-test-kserve-559469c4b5-s2258\" (UID: \"c33c72a9-0a05-4de7-b63b-6c7e52e88c89\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-559469c4b5-s2258"
Sep 08 13:48:18.540363 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:48:18.540352    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c33c72a9-0a05-4de7-b63b-6c7e52e88c89-dshm\") pod \"tokenizer-clean-path-test-kserve-559469c4b5-s2258\" (UID: \"c33c72a9-0a05-4de7-b63b-6c7e52e88c89\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-559469c4b5-s2258"
Sep 08 13:48:18.540437 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:48:18.540421    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c33c72a9-0a05-4de7-b63b-6c7e52e88c89-model-cache\") pod \"tokenizer-clean-path-test-kserve-559469c4b5-s2258\" (UID: \"c33c72a9-0a05-4de7-b63b-6c7e52e88c89\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-559469c4b5-s2258"
Sep 08 13:48:18.540633 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:48:18.540617    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c33c72a9-0a05-4de7-b63b-6c7e52e88c89-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-559469c4b5-s2258\" (UID: \"c33c72a9-0a05-4de7-b63b-6c7e52e88c89\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-559469c4b5-s2258"
Sep 08 13:48:18.540788 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:48:18.540771    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c33c72a9-0a05-4de7-b63b-6c7e52e88c89-home\") pod \"tokenizer-clean-path-test-kserve-559469c4b5-s2258\" (UID: \"c33c72a9-0a05-4de7-b63b-6c7e52e88c89\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-559469c4b5-s2258"
Sep 08 13:48:18.540916 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:48:18.540900    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c33c72a9-0a05-4de7-b63b-6c7e52e88c89-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-559469c4b5-s2258\" (UID: \"c33c72a9-0a05-4de7-b63b-6c7e52e88c89\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-559469c4b5-s2258"
Sep 08 13:48:18.641372 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:48:18.641341    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c33c72a9-0a05-4de7-b63b-6c7e52e88c89-dshm\") pod \"tokenizer-clean-path-test-kserve-559469c4b5-s2258\" (UID: \"c33c72a9-0a05-4de7-b63b-6c7e52e88c89\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-559469c4b5-s2258"
Sep 08 13:48:18.641558 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:48:18.641383    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c33c72a9-0a05-4de7-b63b-6c7e52e88c89-model-cache\") pod \"tokenizer-clean-path-test-kserve-559469c4b5-s2258\" (UID: \"c33c72a9-0a05-4de7-b63b-6c7e52e88c89\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-559469c4b5-s2258"
Sep 08 13:48:18.641558 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:48:18.641399    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c33c72a9-0a05-4de7-b63b-6c7e52e88c89-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-559469c4b5-s2258\" (UID: \"c33c72a9-0a05-4de7-b63b-6c7e52e88c89\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-559469c4b5-s2258"
Sep 08 13:48:18.641558 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:48:18.641547    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c33c72a9-0a05-4de7-b63b-6c7e52e88c89-home\") pod \"tokenizer-clean-path-test-kserve-559469c4b5-s2258\" (UID: \"c33c72a9-0a05-4de7-b63b-6c7e52e88c89\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-559469c4b5-s2258"
Sep 08 13:48:18.641742 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:48:18.641579    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c33c72a9-0a05-4de7-b63b-6c7e52e88c89-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-559469c4b5-s2258\" (UID: \"c33c72a9-0a05-4de7-b63b-6c7e52e88c89\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-559469c4b5-s2258"
Sep 08 13:48:18.641742 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:48:18.641616    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c33c72a9-0a05-4de7-b63b-6c7e52e88c89-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-559469c4b5-s2258\" (UID: \"c33c72a9-0a05-4de7-b63b-6c7e52e88c89\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-559469c4b5-s2258"
Sep 08 13:48:18.641742 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:48:18.641640    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c87v7\" (UniqueName: \"kubernetes.io/projected/c33c72a9-0a05-4de7-b63b-6c7e52e88c89-kube-api-access-c87v7\") pod \"tokenizer-clean-path-test-kserve-559469c4b5-s2258\" (UID: \"c33c72a9-0a05-4de7-b63b-6c7e52e88c89\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-559469c4b5-s2258"
Sep 08 13:48:18.641742 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:48:18.641666    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c33c72a9-0a05-4de7-b63b-6c7e52e88c89-tls-certs\") pod \"tokenizer-clean-path-test-kserve-559469c4b5-s2258\" (UID: \"c33c72a9-0a05-4de7-b63b-6c7e52e88c89\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-559469c4b5-s2258"
Sep 08 13:48:18.641742 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:48:18.641713    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c33c72a9-0a05-4de7-b63b-6c7e52e88c89-model-cache\") pod \"tokenizer-clean-path-test-kserve-559469c4b5-s2258\" (UID: \"c33c72a9-0a05-4de7-b63b-6c7e52e88c89\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-559469c4b5-s2258"
Sep 08 13:48:18.641961 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:48:18.641771    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c33c72a9-0a05-4de7-b63b-6c7e52e88c89-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-559469c4b5-s2258\" (UID: \"c33c72a9-0a05-4de7-b63b-6c7e52e88c89\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-559469c4b5-s2258"
Sep 08 13:48:18.641961 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:48:18.641896    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c33c72a9-0a05-4de7-b63b-6c7e52e88c89-home\") pod \"tokenizer-clean-path-test-kserve-559469c4b5-s2258\" (UID: \"c33c72a9-0a05-4de7-b63b-6c7e52e88c89\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-559469c4b5-s2258"
Sep 08 13:48:18.642047 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:48:18.641986    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c33c72a9-0a05-4de7-b63b-6c7e52e88c89-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-559469c4b5-s2258\" (UID: \"c33c72a9-0a05-4de7-b63b-6c7e52e88c89\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-559469c4b5-s2258"
Sep 08 13:48:18.642365 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:48:18.642342    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c33c72a9-0a05-4de7-b63b-6c7e52e88c89-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-559469c4b5-s2258\" (UID: \"c33c72a9-0a05-4de7-b63b-6c7e52e88c89\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-559469c4b5-s2258"
Sep 08 13:48:18.643795 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:48:18.643772    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c33c72a9-0a05-4de7-b63b-6c7e52e88c89-dshm\") pod \"tokenizer-clean-path-test-kserve-559469c4b5-s2258\" (UID: \"c33c72a9-0a05-4de7-b63b-6c7e52e88c89\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-559469c4b5-s2258"
Sep 08 13:48:18.644097 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:48:18.644079    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c33c72a9-0a05-4de7-b63b-6c7e52e88c89-tls-certs\") pod \"tokenizer-clean-path-test-kserve-559469c4b5-s2258\" (UID: \"c33c72a9-0a05-4de7-b63b-6c7e52e88c89\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-559469c4b5-s2258"
Sep 08 13:48:18.651560 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:48:18.651535    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c87v7\" (UniqueName: \"kubernetes.io/projected/c33c72a9-0a05-4de7-b63b-6c7e52e88c89-kube-api-access-c87v7\") pod \"tokenizer-clean-path-test-kserve-559469c4b5-s2258\" (UID: \"c33c72a9-0a05-4de7-b63b-6c7e52e88c89\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-559469c4b5-s2258"
Sep 08 13:48:18.769299 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:48:18.769239    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-559469c4b5-s2258"
Sep 08 13:48:18.858788 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:48:18.858761    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:48:18.955381 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:48:18.955346    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:48:19.061410 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:48:19.061389    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-559469c4b5-s2258"]
Sep 08 13:48:19.063470 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:48:19.063443    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc33c72a9_0a05_4de7_b63b_6c7e52e88c89.slice/crio-bb7edaa35fd23b16fae6ac46258cc90b1cb0c1f1e72aaec964da8c81860b4e15 WatchSource:0}: Error finding container bb7edaa35fd23b16fae6ac46258cc90b1cb0c1f1e72aaec964da8c81860b4e15: Status 404 returned error can't find the container with id bb7edaa35fd23b16fae6ac46258cc90b1cb0c1f1e72aaec964da8c81860b4e15
Sep 08 13:48:19.965443 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:48:19.965408    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-559469c4b5-s2258" event={"ID":"c33c72a9-0a05-4de7-b63b-6c7e52e88c89","Type":"ContainerStarted","Data":"cb18921733553b8c3795b6f20e453617a880e1c44805cc8c835ac778327ea754"}
Sep 08 13:48:19.965841 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:48:19.965460    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-559469c4b5-s2258" event={"ID":"c33c72a9-0a05-4de7-b63b-6c7e52e88c89","Type":"ContainerStarted","Data":"bb7edaa35fd23b16fae6ac46258cc90b1cb0c1f1e72aaec964da8c81860b4e15"}
Sep 08 13:48:20.602107 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:48:20.602080    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:48:20.702460 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:48:20.702430    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:48:21.443772 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:48:21.443692    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:48:21.974048 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:48:21.974015    2573 generic.go:356] "Generic (PLEG): container finished" podID="c33c72a9-0a05-4de7-b63b-6c7e52e88c89" containerID="cb18921733553b8c3795b6f20e453617a880e1c44805cc8c835ac778327ea754" exitCode=0
Sep 08 13:48:21.974204 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:48:21.974062    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-559469c4b5-s2258" event={"ID":"c33c72a9-0a05-4de7-b63b-6c7e52e88c89","Type":"ContainerDied","Data":"cb18921733553b8c3795b6f20e453617a880e1c44805cc8c835ac778327ea754"}
Sep 08 13:48:22.531318 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:48:22.531283    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:48:22.979027 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:48:22.978993    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-559469c4b5-s2258" event={"ID":"c33c72a9-0a05-4de7-b63b-6c7e52e88c89","Type":"ContainerStarted","Data":"57bd8259052947189b523c6aa2c983fdc61f6db7a93901ca7598ece79e37e9ef"}
Sep 08 13:48:23.005400 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:48:23.005338    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-559469c4b5-s2258" podStartSLOduration=2.9342999020000002 podStartE2EDuration="5.005318058s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.071018156s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:48:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:48:23.002854555 +0000 UTC m=+1953.274302498" watchObservedRunningTime="2026-09-08 13:48:23.005318058 +0000 UTC m=+1953.276765990"
Sep 08 13:48:23.770321 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:48:23.770275    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-559469c4b5-s2258"
Sep 08 13:48:27.043635 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:48:27.043602    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:48:28.207480 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:48:28.207449    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-fvj5v"]
Sep 08 13:48:28.211670 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:48:28.211653    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-fvj5v"
Sep 08 13:48:28.215375 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:48:28.215351    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-force-stop-3e394e04\"/\"default-dockercfg-xhxgn\""
Sep 08 13:48:28.215528 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:48:28.215387    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-force-stop-3e394e04\"/\"stop-v1-kserve-self-signed-certs\""
Sep 08 13:48:28.215528 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:48:28.215417    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-force-stop-3e394e04\"/\"kube-root-ca.crt\""
Sep 08 13:48:28.215528 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:48:28.215414    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-force-stop-3e394e04\"/\"openshift-service-ca.crt\""
Sep 08 13:48:28.227637 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:48:28.227609    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-fvj5v"]
Sep 08 13:48:28.242679 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:48:28.242653    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:48:28.313851 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:48:28.313826    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b1eb5ea-e6ca-4280-a512-155e22159bef-home\") pod \"stop-v1-kserve-5657b88c95-fvj5v\" (UID: \"5b1eb5ea-e6ca-4280-a512-155e22159bef\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-fvj5v"
Sep 08 13:48:28.313951 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:48:28.313853    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b1eb5ea-e6ca-4280-a512-155e22159bef-tmp-dir\") pod \"stop-v1-kserve-5657b88c95-fvj5v\" (UID: \"5b1eb5ea-e6ca-4280-a512-155e22159bef\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-fvj5v"
Sep 08 13:48:28.313951 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:48:28.313869    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b1eb5ea-e6ca-4280-a512-155e22159bef-tls-certs\") pod \"stop-v1-kserve-5657b88c95-fvj5v\" (UID: \"5b1eb5ea-e6ca-4280-a512-155e22159bef\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-fvj5v"
Sep 08 13:48:28.313951 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:48:28.313897    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qfdb\" (UniqueName: \"kubernetes.io/projected/5b1eb5ea-e6ca-4280-a512-155e22159bef-kube-api-access-2qfdb\") pod \"stop-v1-kserve-5657b88c95-fvj5v\" (UID: \"5b1eb5ea-e6ca-4280-a512-155e22159bef\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-fvj5v"
Sep 08 13:48:28.314065 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:48:28.313953    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b1eb5ea-e6ca-4280-a512-155e22159bef-model-cache\") pod \"stop-v1-kserve-5657b88c95-fvj5v\" (UID: \"5b1eb5ea-e6ca-4280-a512-155e22159bef\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-fvj5v"
Sep 08 13:48:28.314065 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:48:28.313984    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b1eb5ea-e6ca-4280-a512-155e22159bef-dshm\") pod \"stop-v1-kserve-5657b88c95-fvj5v\" (UID: \"5b1eb5ea-e6ca-4280-a512-155e22159bef\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-fvj5v"
Sep 08 13:48:28.415085 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:48:28.415060    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b1eb5ea-e6ca-4280-a512-155e22159bef-home\") pod \"stop-v1-kserve-5657b88c95-fvj5v\" (UID: \"5b1eb5ea-e6ca-4280-a512-155e22159bef\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-fvj5v"
Sep 08 13:48:28.415232 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:48:28.415092    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b1eb5ea-e6ca-4280-a512-155e22159bef-tmp-dir\") pod \"stop-v1-kserve-5657b88c95-fvj5v\" (UID: \"5b1eb5ea-e6ca-4280-a512-155e22159bef\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-fvj5v"
Sep 08 13:48:28.415232 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:48:28.415108    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b1eb5ea-e6ca-4280-a512-155e22159bef-tls-certs\") pod \"stop-v1-kserve-5657b88c95-fvj5v\" (UID: \"5b1eb5ea-e6ca-4280-a512-155e22159bef\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-fvj5v"
Sep 08 13:48:28.415232 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:48:28.415134    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qfdb\" (UniqueName: \"kubernetes.io/projected/5b1eb5ea-e6ca-4280-a512-155e22159bef-kube-api-access-2qfdb\") pod \"stop-v1-kserve-5657b88c95-fvj5v\" (UID: \"5b1eb5ea-e6ca-4280-a512-155e22159bef\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-fvj5v"
Sep 08 13:48:28.415232 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:48:28.415211    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b1eb5ea-e6ca-4280-a512-155e22159bef-model-cache\") pod \"stop-v1-kserve-5657b88c95-fvj5v\" (UID: \"5b1eb5ea-e6ca-4280-a512-155e22159bef\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-fvj5v"
Sep 08 13:48:28.415443 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:48:28.415275    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b1eb5ea-e6ca-4280-a512-155e22159bef-dshm\") pod \"stop-v1-kserve-5657b88c95-fvj5v\" (UID: \"5b1eb5ea-e6ca-4280-a512-155e22159bef\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-fvj5v"
Sep 08 13:48:28.415533 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:48:28.415493    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b1eb5ea-e6ca-4280-a512-155e22159bef-tmp-dir\") pod \"stop-v1-kserve-5657b88c95-fvj5v\" (UID: \"5b1eb5ea-e6ca-4280-a512-155e22159bef\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-fvj5v"
Sep 08 13:48:28.415604 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:48:28.415536    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b1eb5ea-e6ca-4280-a512-155e22159bef-home\") pod \"stop-v1-kserve-5657b88c95-fvj5v\" (UID: \"5b1eb5ea-e6ca-4280-a512-155e22159bef\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-fvj5v"
Sep 08 13:48:28.415604 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:48:28.415582    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b1eb5ea-e6ca-4280-a512-155e22159bef-model-cache\") pod \"stop-v1-kserve-5657b88c95-fvj5v\" (UID: \"5b1eb5ea-e6ca-4280-a512-155e22159bef\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-fvj5v"
Sep 08 13:48:28.417314 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:48:28.417298    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b1eb5ea-e6ca-4280-a512-155e22159bef-dshm\") pod \"stop-v1-kserve-5657b88c95-fvj5v\" (UID: \"5b1eb5ea-e6ca-4280-a512-155e22159bef\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-fvj5v"
Sep 08 13:48:28.417543 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:48:28.417522    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b1eb5ea-e6ca-4280-a512-155e22159bef-tls-certs\") pod \"stop-v1-kserve-5657b88c95-fvj5v\" (UID: \"5b1eb5ea-e6ca-4280-a512-155e22159bef\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-fvj5v"
Sep 08 13:48:28.424266 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:48:28.424242    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qfdb\" (UniqueName: \"kubernetes.io/projected/5b1eb5ea-e6ca-4280-a512-155e22159bef-kube-api-access-2qfdb\") pod \"stop-v1-kserve-5657b88c95-fvj5v\" (UID: \"5b1eb5ea-e6ca-4280-a512-155e22159bef\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-fvj5v"
Sep 08 13:48:28.522136 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:48:28.522083    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-fvj5v"
Sep 08 13:48:28.769909 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:48:28.769878    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-559469c4b5-s2258"
Sep 08 13:48:28.782717 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:48:28.782657    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-559469c4b5-s2258"
Sep 08 13:48:28.799191 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:48:28.799170    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-fvj5v"]
Sep 08 13:48:28.801701 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:48:28.801679    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b1eb5ea_e6ca_4280_a512_155e22159bef.slice/crio-aa966a305cb47b559dde2939fc97197b8e25b226f7874f12f3b063637c2dd4ac WatchSource:0}: Error finding container aa966a305cb47b559dde2939fc97197b8e25b226f7874f12f3b063637c2dd4ac: Status 404 returned error can't find the container with id aa966a305cb47b559dde2939fc97197b8e25b226f7874f12f3b063637c2dd4ac
Sep 08 13:48:29.003062 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:48:29.003027    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-fvj5v" event={"ID":"5b1eb5ea-e6ca-4280-a512-155e22159bef","Type":"ContainerStarted","Data":"09c89e886549bbb509a30fcb5426ccfed1a504e0208fcadeef3e08910eb02fb2"}
Sep 08 13:48:29.003225 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:48:29.003092    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-fvj5v" event={"ID":"5b1eb5ea-e6ca-4280-a512-155e22159bef","Type":"ContainerStarted","Data":"aa966a305cb47b559dde2939fc97197b8e25b226f7874f12f3b063637c2dd4ac"}
Sep 08 13:48:29.014182 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:48:29.014160    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-559469c4b5-s2258"
Sep 08 13:48:29.026094 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:48:29.026037    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-fvj5v" podStartSLOduration=1.026020244 podStartE2EDuration="1.026020244s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:48:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:48:29.025824035 +0000 UTC m=+1959.297271993" watchObservedRunningTime="2026-09-08 13:48:29.026020244 +0000 UTC m=+1959.297468177"
Sep 08 13:48:29.523160 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:48:29.523125    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-fvj5v"
Sep 08 13:48:35.478298 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:48:35.478268    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:48:38.522762 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:48:38.522715    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-fvj5v"
Sep 08 13:48:38.535010 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:48:38.534986    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-fvj5v"
Sep 08 13:48:39.057094 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:48:39.057062    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-fvj5v"
Sep 08 13:48:41.596884 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:48:41.596846    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:48:41.681203 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:48:41.681178    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:48:44.944013 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:48:44.943923    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:48:54.253785 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:48:54.253743    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:49:31.140850 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:49:31.140814    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:49:42.638532 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:49:42.638477    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-559469c4b5-s2258"]
Sep 08 13:49:42.645467 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:49:42.645438    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:49:42.647622 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:49:42.647581    2573 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 08 13:49:42.647622 ip-10-0-132-254 kubenswrapper[2573]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 08 13:49:42.647622 ip-10-0-132-254 kubenswrapper[2573]:         , stdout: , stderr: , exit code -1
Sep 08 13:49:42.647622 ip-10-0-132-254 kubenswrapper[2573]:  > containerID="57bd8259052947189b523c6aa2c983fdc61f6db7a93901ca7598ece79e37e9ef" cmd=["/bin/sleep","15"]
Sep 08 13:49:42.647901 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:49:42.647645    2573 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-559469c4b5-s2258" message="" err=<
Sep 08 13:49:42.647901 ip-10-0-132-254 kubenswrapper[2573]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 08 13:49:42.647901 ip-10-0-132-254 kubenswrapper[2573]:         , stdout: , stderr: , exit code -1
Sep 08 13:49:42.647901 ip-10-0-132-254 kubenswrapper[2573]:  >
Sep 08 13:49:42.647901 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:49:42.647662    2573 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 08 13:49:42.647901 ip-10-0-132-254 kubenswrapper[2573]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 08 13:49:42.647901 ip-10-0-132-254 kubenswrapper[2573]:         , stdout: , stderr: , exit code -1
Sep 08 13:49:42.647901 ip-10-0-132-254 kubenswrapper[2573]:  > pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-559469c4b5-s2258" podUID="c33c72a9-0a05-4de7-b63b-6c7e52e88c89" containerName="main" containerID="cri-o://57bd8259052947189b523c6aa2c983fdc61f6db7a93901ca7598ece79e37e9ef"
Sep 08 13:49:42.647901 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:49:42.647718    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-559469c4b5-s2258" podUID="c33c72a9-0a05-4de7-b63b-6c7e52e88c89" containerName="main" containerID="cri-o://57bd8259052947189b523c6aa2c983fdc61f6db7a93901ca7598ece79e37e9ef" gracePeriod=60
Sep 08 13:49:42.801800 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:49:42.801772    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:49:42.990606 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:49:42.990584    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-559469c4b5-s2258"
Sep 08 13:49:43.074672 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:49:43.074649    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c33c72a9-0a05-4de7-b63b-6c7e52e88c89-tls-certs\") pod \"c33c72a9-0a05-4de7-b63b-6c7e52e88c89\" (UID: \"c33c72a9-0a05-4de7-b63b-6c7e52e88c89\") "
Sep 08 13:49:43.074808 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:49:43.074679    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c33c72a9-0a05-4de7-b63b-6c7e52e88c89-home\") pod \"c33c72a9-0a05-4de7-b63b-6c7e52e88c89\" (UID: \"c33c72a9-0a05-4de7-b63b-6c7e52e88c89\") "
Sep 08 13:49:43.074808 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:49:43.074702    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c33c72a9-0a05-4de7-b63b-6c7e52e88c89-cabundle-cert\") pod \"c33c72a9-0a05-4de7-b63b-6c7e52e88c89\" (UID: \"c33c72a9-0a05-4de7-b63b-6c7e52e88c89\") "
Sep 08 13:49:43.074808 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:49:43.074729    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c33c72a9-0a05-4de7-b63b-6c7e52e88c89-dshm\") pod \"c33c72a9-0a05-4de7-b63b-6c7e52e88c89\" (UID: \"c33c72a9-0a05-4de7-b63b-6c7e52e88c89\") "
Sep 08 13:49:43.074808 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:49:43.074749    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c33c72a9-0a05-4de7-b63b-6c7e52e88c89-kserve-provision-location\") pod \"c33c72a9-0a05-4de7-b63b-6c7e52e88c89\" (UID: \"c33c72a9-0a05-4de7-b63b-6c7e52e88c89\") "
Sep 08 13:49:43.074808 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:49:43.074792    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c33c72a9-0a05-4de7-b63b-6c7e52e88c89-model-cache\") pod \"c33c72a9-0a05-4de7-b63b-6c7e52e88c89\" (UID: \"c33c72a9-0a05-4de7-b63b-6c7e52e88c89\") "
Sep 08 13:49:43.075068 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:49:43.074843    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c33c72a9-0a05-4de7-b63b-6c7e52e88c89-tmp-dir\") pod \"c33c72a9-0a05-4de7-b63b-6c7e52e88c89\" (UID: \"c33c72a9-0a05-4de7-b63b-6c7e52e88c89\") "
Sep 08 13:49:43.075068 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:49:43.074884    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c87v7\" (UniqueName: \"kubernetes.io/projected/c33c72a9-0a05-4de7-b63b-6c7e52e88c89-kube-api-access-c87v7\") pod \"c33c72a9-0a05-4de7-b63b-6c7e52e88c89\" (UID: \"c33c72a9-0a05-4de7-b63b-6c7e52e88c89\") "
Sep 08 13:49:43.075068 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:49:43.074941    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c33c72a9-0a05-4de7-b63b-6c7e52e88c89-home" (OuterVolumeSpecName: "home") pod "c33c72a9-0a05-4de7-b63b-6c7e52e88c89" (UID: "c33c72a9-0a05-4de7-b63b-6c7e52e88c89"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:49:43.075222 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:49:43.075118    2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c33c72a9-0a05-4de7-b63b-6c7e52e88c89-home\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 13:49:43.075222 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:49:43.075132    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c33c72a9-0a05-4de7-b63b-6c7e52e88c89-model-cache" (OuterVolumeSpecName: "model-cache") pod "c33c72a9-0a05-4de7-b63b-6c7e52e88c89" (UID: "c33c72a9-0a05-4de7-b63b-6c7e52e88c89"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:49:43.075222 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:49:43.075147    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c33c72a9-0a05-4de7-b63b-6c7e52e88c89-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c33c72a9-0a05-4de7-b63b-6c7e52e88c89" (UID: "c33c72a9-0a05-4de7-b63b-6c7e52e88c89"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:49:43.075222 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:49:43.075154    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c33c72a9-0a05-4de7-b63b-6c7e52e88c89-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c33c72a9-0a05-4de7-b63b-6c7e52e88c89" (UID: "c33c72a9-0a05-4de7-b63b-6c7e52e88c89"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:49:43.076833 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:49:43.076809    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c33c72a9-0a05-4de7-b63b-6c7e52e88c89-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c33c72a9-0a05-4de7-b63b-6c7e52e88c89" (UID: "c33c72a9-0a05-4de7-b63b-6c7e52e88c89"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:49:43.076947 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:49:43.076875    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c33c72a9-0a05-4de7-b63b-6c7e52e88c89-dshm" (OuterVolumeSpecName: "dshm") pod "c33c72a9-0a05-4de7-b63b-6c7e52e88c89" (UID: "c33c72a9-0a05-4de7-b63b-6c7e52e88c89"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:49:43.077012 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:49:43.076976    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c33c72a9-0a05-4de7-b63b-6c7e52e88c89-kube-api-access-c87v7" (OuterVolumeSpecName: "kube-api-access-c87v7") pod "c33c72a9-0a05-4de7-b63b-6c7e52e88c89" (UID: "c33c72a9-0a05-4de7-b63b-6c7e52e88c89"). InnerVolumeSpecName "kube-api-access-c87v7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:49:43.095388 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:49:43.095363    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c33c72a9-0a05-4de7-b63b-6c7e52e88c89-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c33c72a9-0a05-4de7-b63b-6c7e52e88c89" (UID: "c33c72a9-0a05-4de7-b63b-6c7e52e88c89"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:49:43.176139 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:49:43.176088    2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c33c72a9-0a05-4de7-b63b-6c7e52e88c89-model-cache\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 13:49:43.176139 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:49:43.176108    2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c33c72a9-0a05-4de7-b63b-6c7e52e88c89-tmp-dir\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 13:49:43.176139 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:49:43.176118    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c87v7\" (UniqueName: \"kubernetes.io/projected/c33c72a9-0a05-4de7-b63b-6c7e52e88c89-kube-api-access-c87v7\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 13:49:43.176139 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:49:43.176126    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c33c72a9-0a05-4de7-b63b-6c7e52e88c89-tls-certs\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 13:49:43.176139 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:49:43.176135    2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c33c72a9-0a05-4de7-b63b-6c7e52e88c89-cabundle-cert\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 13:49:43.176139 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:49:43.176142    2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c33c72a9-0a05-4de7-b63b-6c7e52e88c89-dshm\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 13:49:43.176391 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:49:43.176150    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c33c72a9-0a05-4de7-b63b-6c7e52e88c89-kserve-provision-location\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 13:49:43.312660 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:49:43.312628    2573 generic.go:356] "Generic (PLEG): container finished" podID="c33c72a9-0a05-4de7-b63b-6c7e52e88c89" containerID="57bd8259052947189b523c6aa2c983fdc61f6db7a93901ca7598ece79e37e9ef" exitCode=0
Sep 08 13:49:43.312792 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:49:43.312668    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-559469c4b5-s2258" event={"ID":"c33c72a9-0a05-4de7-b63b-6c7e52e88c89","Type":"ContainerDied","Data":"57bd8259052947189b523c6aa2c983fdc61f6db7a93901ca7598ece79e37e9ef"}
Sep 08 13:49:43.312792 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:49:43.312684    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-559469c4b5-s2258"
Sep 08 13:49:43.312792 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:49:43.312735    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-559469c4b5-s2258" event={"ID":"c33c72a9-0a05-4de7-b63b-6c7e52e88c89","Type":"ContainerDied","Data":"bb7edaa35fd23b16fae6ac46258cc90b1cb0c1f1e72aaec964da8c81860b4e15"}
Sep 08 13:49:43.312792 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:49:43.312769    2573 scope.go:117] "RemoveContainer" containerID="57bd8259052947189b523c6aa2c983fdc61f6db7a93901ca7598ece79e37e9ef"
Sep 08 13:49:43.325267 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:49:43.325245    2573 scope.go:117] "RemoveContainer" containerID="cb18921733553b8c3795b6f20e453617a880e1c44805cc8c835ac778327ea754"
Sep 08 13:49:43.332406 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:49:43.332391    2573 scope.go:117] "RemoveContainer" containerID="57bd8259052947189b523c6aa2c983fdc61f6db7a93901ca7598ece79e37e9ef"
Sep 08 13:49:43.332761 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:49:43.332742    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57bd8259052947189b523c6aa2c983fdc61f6db7a93901ca7598ece79e37e9ef\": container with ID starting with 57bd8259052947189b523c6aa2c983fdc61f6db7a93901ca7598ece79e37e9ef not found: ID does not exist" containerID="57bd8259052947189b523c6aa2c983fdc61f6db7a93901ca7598ece79e37e9ef"
Sep 08 13:49:43.332835 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:49:43.332768    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57bd8259052947189b523c6aa2c983fdc61f6db7a93901ca7598ece79e37e9ef"} err="failed to get container status \"57bd8259052947189b523c6aa2c983fdc61f6db7a93901ca7598ece79e37e9ef\": rpc error: code = NotFound desc = could not find container \"57bd8259052947189b523c6aa2c983fdc61f6db7a93901ca7598ece79e37e9ef\": container with ID starting with 57bd8259052947189b523c6aa2c983fdc61f6db7a93901ca7598ece79e37e9ef not found: ID does not exist"
Sep 08 13:49:43.332835 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:49:43.332785    2573 scope.go:117] "RemoveContainer" containerID="cb18921733553b8c3795b6f20e453617a880e1c44805cc8c835ac778327ea754"
Sep 08 13:49:43.333032 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:49:43.333017    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb18921733553b8c3795b6f20e453617a880e1c44805cc8c835ac778327ea754\": container with ID starting with cb18921733553b8c3795b6f20e453617a880e1c44805cc8c835ac778327ea754 not found: ID does not exist" containerID="cb18921733553b8c3795b6f20e453617a880e1c44805cc8c835ac778327ea754"
Sep 08 13:49:43.333073 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:49:43.333039    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb18921733553b8c3795b6f20e453617a880e1c44805cc8c835ac778327ea754"} err="failed to get container status \"cb18921733553b8c3795b6f20e453617a880e1c44805cc8c835ac778327ea754\": rpc error: code = NotFound desc = could not find container \"cb18921733553b8c3795b6f20e453617a880e1c44805cc8c835ac778327ea754\": container with ID starting with cb18921733553b8c3795b6f20e453617a880e1c44805cc8c835ac778327ea754 not found: ID does not exist"
Sep 08 13:49:43.350049 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:49:43.350031    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-559469c4b5-s2258"]
Sep 08 13:49:43.351263 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:49:43.351245    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-559469c4b5-s2258"]
Sep 08 13:49:44.174423 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:49:44.174388    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c33c72a9-0a05-4de7-b63b-6c7e52e88c89" path="/var/lib/kubelet/pods/c33c72a9-0a05-4de7-b63b-6c7e52e88c89/volumes"
Sep 08 13:49:45.394710 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:49:45.394673    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:49:51.349578 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:49:51.349537    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:49:51.532495 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:49:51.532461    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:49:51.669391 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:49:51.669360    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:49:55.042881 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:49:55.042846    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:49:55.141452 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:49:55.141421    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:49:56.313909 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:49:56.313878    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-fvj5v"]
Sep 08 13:49:56.408653 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:49:56.408629    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:50:00.954004 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:50:00.953974    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:50:11.333267 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:50:11.333222    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-fvj5v" podUID="5b1eb5ea-e6ca-4280-a512-155e22159bef" containerName="main" containerID="cri-o://09c89e886549bbb509a30fcb5426ccfed1a504e0208fcadeef3e08910eb02fb2" gracePeriod=45
Sep 08 13:50:11.562533 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:50:11.562494    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-fvj5v"
Sep 08 13:50:11.690236 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:50:11.690210    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b1eb5ea-e6ca-4280-a512-155e22159bef-tls-certs\") pod \"5b1eb5ea-e6ca-4280-a512-155e22159bef\" (UID: \"5b1eb5ea-e6ca-4280-a512-155e22159bef\") "
Sep 08 13:50:11.690412 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:50:11.690246    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b1eb5ea-e6ca-4280-a512-155e22159bef-model-cache\") pod \"5b1eb5ea-e6ca-4280-a512-155e22159bef\" (UID: \"5b1eb5ea-e6ca-4280-a512-155e22159bef\") "
Sep 08 13:50:11.690412 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:50:11.690266    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b1eb5ea-e6ca-4280-a512-155e22159bef-dshm\") pod \"5b1eb5ea-e6ca-4280-a512-155e22159bef\" (UID: \"5b1eb5ea-e6ca-4280-a512-155e22159bef\") "
Sep 08 13:50:11.690412 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:50:11.690344    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b1eb5ea-e6ca-4280-a512-155e22159bef-home\") pod \"5b1eb5ea-e6ca-4280-a512-155e22159bef\" (UID: \"5b1eb5ea-e6ca-4280-a512-155e22159bef\") "
Sep 08 13:50:11.690412 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:50:11.690369    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2qfdb\" (UniqueName: \"kubernetes.io/projected/5b1eb5ea-e6ca-4280-a512-155e22159bef-kube-api-access-2qfdb\") pod \"5b1eb5ea-e6ca-4280-a512-155e22159bef\" (UID: \"5b1eb5ea-e6ca-4280-a512-155e22159bef\") "
Sep 08 13:50:11.690412 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:50:11.690413    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b1eb5ea-e6ca-4280-a512-155e22159bef-tmp-dir\") pod \"5b1eb5ea-e6ca-4280-a512-155e22159bef\" (UID: \"5b1eb5ea-e6ca-4280-a512-155e22159bef\") "
Sep 08 13:50:11.690715 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:50:11.690542    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b1eb5ea-e6ca-4280-a512-155e22159bef-model-cache" (OuterVolumeSpecName: "model-cache") pod "5b1eb5ea-e6ca-4280-a512-155e22159bef" (UID: "5b1eb5ea-e6ca-4280-a512-155e22159bef"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:50:11.690715 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:50:11.690629    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b1eb5ea-e6ca-4280-a512-155e22159bef-home" (OuterVolumeSpecName: "home") pod "5b1eb5ea-e6ca-4280-a512-155e22159bef" (UID: "5b1eb5ea-e6ca-4280-a512-155e22159bef"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:50:11.690715 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:50:11.690666    2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b1eb5ea-e6ca-4280-a512-155e22159bef-model-cache\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 13:50:11.690829 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:50:11.690723    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b1eb5ea-e6ca-4280-a512-155e22159bef-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5b1eb5ea-e6ca-4280-a512-155e22159bef" (UID: "5b1eb5ea-e6ca-4280-a512-155e22159bef"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:50:11.692483 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:50:11.692462    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b1eb5ea-e6ca-4280-a512-155e22159bef-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5b1eb5ea-e6ca-4280-a512-155e22159bef" (UID: "5b1eb5ea-e6ca-4280-a512-155e22159bef"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:50:11.692483 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:50:11.692468    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b1eb5ea-e6ca-4280-a512-155e22159bef-dshm" (OuterVolumeSpecName: "dshm") pod "5b1eb5ea-e6ca-4280-a512-155e22159bef" (UID: "5b1eb5ea-e6ca-4280-a512-155e22159bef"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:50:11.692633 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:50:11.692530    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b1eb5ea-e6ca-4280-a512-155e22159bef-kube-api-access-2qfdb" (OuterVolumeSpecName: "kube-api-access-2qfdb") pod "5b1eb5ea-e6ca-4280-a512-155e22159bef" (UID: "5b1eb5ea-e6ca-4280-a512-155e22159bef"). InnerVolumeSpecName "kube-api-access-2qfdb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:50:11.791938 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:50:11.791911    2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b1eb5ea-e6ca-4280-a512-155e22159bef-home\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 13:50:11.792042 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:50:11.791941    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qfdb\" (UniqueName: \"kubernetes.io/projected/5b1eb5ea-e6ca-4280-a512-155e22159bef-kube-api-access-2qfdb\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 13:50:11.792042 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:50:11.791970    2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b1eb5ea-e6ca-4280-a512-155e22159bef-tmp-dir\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 13:50:11.792042 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:50:11.791982    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b1eb5ea-e6ca-4280-a512-155e22159bef-tls-certs\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 13:50:11.792042 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:50:11.791990    2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b1eb5ea-e6ca-4280-a512-155e22159bef-dshm\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 13:50:12.430992 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:50:12.430955    2573 generic.go:356] "Generic (PLEG): container finished" podID="5b1eb5ea-e6ca-4280-a512-155e22159bef" containerID="09c89e886549bbb509a30fcb5426ccfed1a504e0208fcadeef3e08910eb02fb2" exitCode=0
Sep 08 13:50:12.431450 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:50:12.431004    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-fvj5v"
Sep 08 13:50:12.431450 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:50:12.431007    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-fvj5v" event={"ID":"5b1eb5ea-e6ca-4280-a512-155e22159bef","Type":"ContainerDied","Data":"09c89e886549bbb509a30fcb5426ccfed1a504e0208fcadeef3e08910eb02fb2"}
Sep 08 13:50:12.431450 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:50:12.431149    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-fvj5v" event={"ID":"5b1eb5ea-e6ca-4280-a512-155e22159bef","Type":"ContainerDied","Data":"aa966a305cb47b559dde2939fc97197b8e25b226f7874f12f3b063637c2dd4ac"}
Sep 08 13:50:12.431450 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:50:12.431190    2573 scope.go:117] "RemoveContainer" containerID="09c89e886549bbb509a30fcb5426ccfed1a504e0208fcadeef3e08910eb02fb2"
Sep 08 13:50:12.439240 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:50:12.439224    2573 scope.go:117] "RemoveContainer" containerID="09c89e886549bbb509a30fcb5426ccfed1a504e0208fcadeef3e08910eb02fb2"
Sep 08 13:50:12.439564 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:50:12.439539    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09c89e886549bbb509a30fcb5426ccfed1a504e0208fcadeef3e08910eb02fb2\": container with ID starting with 09c89e886549bbb509a30fcb5426ccfed1a504e0208fcadeef3e08910eb02fb2 not found: ID does not exist" containerID="09c89e886549bbb509a30fcb5426ccfed1a504e0208fcadeef3e08910eb02fb2"
Sep 08 13:50:12.439658 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:50:12.439569    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09c89e886549bbb509a30fcb5426ccfed1a504e0208fcadeef3e08910eb02fb2"} err="failed to get container status \"09c89e886549bbb509a30fcb5426ccfed1a504e0208fcadeef3e08910eb02fb2\": rpc error: code = NotFound desc = could not find container \"09c89e886549bbb509a30fcb5426ccfed1a504e0208fcadeef3e08910eb02fb2\": container with ID starting with 09c89e886549bbb509a30fcb5426ccfed1a504e0208fcadeef3e08910eb02fb2 not found: ID does not exist"
Sep 08 13:50:12.450757 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:50:12.450736    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-fvj5v"]
Sep 08 13:50:12.456910 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:50:12.456891    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-fvj5v"]
Sep 08 13:50:14.174077 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:50:14.174041    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b1eb5ea-e6ca-4280-a512-155e22159bef" path="/var/lib/kubelet/pods/5b1eb5ea-e6ca-4280-a512-155e22159bef/volumes"
Sep 08 13:50:15.434808 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:50:15.434774    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:50:21.361140 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:50:21.361104    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:50:32.682051 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:50:32.682014    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:50:32.798295 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:50:32.798267    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:50:34.645846 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:50:34.645813    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:50:39.751821 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:50:39.751783    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:50:40.807759 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:50:40.807721    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:50:40.918237 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:50:40.918207    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:50:45.818238 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:50:45.818200    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:50:45.904331 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:50:45.904299    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:50:50.272123 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:50:50.272093    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8h8cn_b936238e-bdf1-4dac-8bf8-6417c313597c/ovn-acl-logging/0.log"
Sep 08 13:50:50.277013 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:50:50.276991    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8h8cn_b936238e-bdf1-4dac-8bf8-6417c313597c/ovn-acl-logging/0.log"
Sep 08 13:50:50.524187 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:50:50.524104    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5897d7c8b8-v84qr"]
Sep 08 13:50:50.524416 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:50:50.524401    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c33c72a9-0a05-4de7-b63b-6c7e52e88c89" containerName="main"
Sep 08 13:50:50.524463 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:50:50.524418    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="c33c72a9-0a05-4de7-b63b-6c7e52e88c89" containerName="main"
Sep 08 13:50:50.524463 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:50:50.524431    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b1eb5ea-e6ca-4280-a512-155e22159bef" containerName="main"
Sep 08 13:50:50.524463 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:50:50.524437    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b1eb5ea-e6ca-4280-a512-155e22159bef" containerName="main"
Sep 08 13:50:50.524463 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:50:50.524449    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c33c72a9-0a05-4de7-b63b-6c7e52e88c89" containerName="storage-initializer"
Sep 08 13:50:50.524463 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:50:50.524454    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="c33c72a9-0a05-4de7-b63b-6c7e52e88c89" containerName="storage-initializer"
Sep 08 13:50:50.524657 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:50:50.524532    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b1eb5ea-e6ca-4280-a512-155e22159bef" containerName="main"
Sep 08 13:50:50.524657 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:50:50.524543    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="c33c72a9-0a05-4de7-b63b-6c7e52e88c89" containerName="main"
Sep 08 13:50:50.527447 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:50:50.527432    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5897d7c8b8-v84qr"
Sep 08 13:50:50.529987 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:50:50.529967    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"seaweedfs-s3-creds\""
Sep 08 13:50:50.530126 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:50:50.530024    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"odh-kserve-custom-ca-bundle\""
Sep 08 13:50:50.530589 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:50:50.530571    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"openshift-service-ca.crt\""
Sep 08 13:50:50.530704 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:50:50.530592    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"kube-root-ca.crt\""
Sep 08 13:50:50.530860 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:50:50.530843    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"default-dockercfg-6r98f\""
Sep 08 13:50:50.530932 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:50:50.530845    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"tokenizer-migration-path-test-kserve-self-signed-certs\""
Sep 08 13:50:50.543185 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:50:50.543165    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5897d7c8b8-v84qr"]
Sep 08 13:50:50.573104 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:50:50.573080    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:50:50.588992 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:50:50.588965    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc3ac180-6bbd-49dd-81a6-082f6d61ea59-dshm\") pod \"tokenizer-migration-path-test-kserve-5897d7c8b8-v84qr\" (UID: \"fc3ac180-6bbd-49dd-81a6-082f6d61ea59\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5897d7c8b8-v84qr"
Sep 08 13:50:50.589093 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:50:50.589015    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fc3ac180-6bbd-49dd-81a6-082f6d61ea59-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-5897d7c8b8-v84qr\" (UID: \"fc3ac180-6bbd-49dd-81a6-082f6d61ea59\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5897d7c8b8-v84qr"
Sep 08 13:50:50.589093 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:50:50.589052    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2rz5\" (UniqueName: \"kubernetes.io/projected/fc3ac180-6bbd-49dd-81a6-082f6d61ea59-kube-api-access-q2rz5\") pod \"tokenizer-migration-path-test-kserve-5897d7c8b8-v84qr\" (UID: \"fc3ac180-6bbd-49dd-81a6-082f6d61ea59\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5897d7c8b8-v84qr"
Sep 08 13:50:50.589199 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:50:50.589103    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fc3ac180-6bbd-49dd-81a6-082f6d61ea59-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-5897d7c8b8-v84qr\" (UID: \"fc3ac180-6bbd-49dd-81a6-082f6d61ea59\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5897d7c8b8-v84qr"
Sep 08 13:50:50.589199 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:50:50.589141    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc3ac180-6bbd-49dd-81a6-082f6d61ea59-tls-certs\") pod \"tokenizer-migration-path-test-kserve-5897d7c8b8-v84qr\" (UID: \"fc3ac180-6bbd-49dd-81a6-082f6d61ea59\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5897d7c8b8-v84qr"
Sep 08 13:50:50.589292 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:50:50.589198    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc3ac180-6bbd-49dd-81a6-082f6d61ea59-model-cache\") pod \"tokenizer-migration-path-test-kserve-5897d7c8b8-v84qr\" (UID: \"fc3ac180-6bbd-49dd-81a6-082f6d61ea59\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5897d7c8b8-v84qr"
Sep 08 13:50:50.589292 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:50:50.589233    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc3ac180-6bbd-49dd-81a6-082f6d61ea59-home\") pod \"tokenizer-migration-path-test-kserve-5897d7c8b8-v84qr\" (UID: \"fc3ac180-6bbd-49dd-81a6-082f6d61ea59\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5897d7c8b8-v84qr"
Sep 08 13:50:50.589292 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:50:50.589273    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc3ac180-6bbd-49dd-81a6-082f6d61ea59-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-5897d7c8b8-v84qr\" (UID: \"fc3ac180-6bbd-49dd-81a6-082f6d61ea59\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5897d7c8b8-v84qr"
Sep 08 13:50:50.690212 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:50:50.690191    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc3ac180-6bbd-49dd-81a6-082f6d61ea59-tls-certs\") pod \"tokenizer-migration-path-test-kserve-5897d7c8b8-v84qr\" (UID: \"fc3ac180-6bbd-49dd-81a6-082f6d61ea59\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5897d7c8b8-v84qr"
Sep 08 13:50:50.690313 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:50:50.690230    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc3ac180-6bbd-49dd-81a6-082f6d61ea59-model-cache\") pod \"tokenizer-migration-path-test-kserve-5897d7c8b8-v84qr\" (UID: \"fc3ac180-6bbd-49dd-81a6-082f6d61ea59\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5897d7c8b8-v84qr"
Sep 08 13:50:50.690313 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:50:50.690249    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc3ac180-6bbd-49dd-81a6-082f6d61ea59-home\") pod \"tokenizer-migration-path-test-kserve-5897d7c8b8-v84qr\" (UID: \"fc3ac180-6bbd-49dd-81a6-082f6d61ea59\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5897d7c8b8-v84qr"
Sep 08 13:50:50.690313 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:50:50.690270    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc3ac180-6bbd-49dd-81a6-082f6d61ea59-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-5897d7c8b8-v84qr\" (UID: \"fc3ac180-6bbd-49dd-81a6-082f6d61ea59\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5897d7c8b8-v84qr"
Sep 08 13:50:50.690313 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:50:50.690299    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc3ac180-6bbd-49dd-81a6-082f6d61ea59-dshm\") pod \"tokenizer-migration-path-test-kserve-5897d7c8b8-v84qr\" (UID: \"fc3ac180-6bbd-49dd-81a6-082f6d61ea59\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5897d7c8b8-v84qr"
Sep 08 13:50:50.690515 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:50:50.690314    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fc3ac180-6bbd-49dd-81a6-082f6d61ea59-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-5897d7c8b8-v84qr\" (UID: \"fc3ac180-6bbd-49dd-81a6-082f6d61ea59\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5897d7c8b8-v84qr"
Sep 08 13:50:50.690515 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:50:50.690331    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2rz5\" (UniqueName: \"kubernetes.io/projected/fc3ac180-6bbd-49dd-81a6-082f6d61ea59-kube-api-access-q2rz5\") pod \"tokenizer-migration-path-test-kserve-5897d7c8b8-v84qr\" (UID: \"fc3ac180-6bbd-49dd-81a6-082f6d61ea59\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5897d7c8b8-v84qr"
Sep 08 13:50:50.690515 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:50:50.690362    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fc3ac180-6bbd-49dd-81a6-082f6d61ea59-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-5897d7c8b8-v84qr\" (UID: \"fc3ac180-6bbd-49dd-81a6-082f6d61ea59\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5897d7c8b8-v84qr"
Sep 08 13:50:50.690684 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:50:50.690647    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc3ac180-6bbd-49dd-81a6-082f6d61ea59-home\") pod \"tokenizer-migration-path-test-kserve-5897d7c8b8-v84qr\" (UID: \"fc3ac180-6bbd-49dd-81a6-082f6d61ea59\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5897d7c8b8-v84qr"
Sep 08 13:50:50.690737 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:50:50.690697    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc3ac180-6bbd-49dd-81a6-082f6d61ea59-model-cache\") pod \"tokenizer-migration-path-test-kserve-5897d7c8b8-v84qr\" (UID: \"fc3ac180-6bbd-49dd-81a6-082f6d61ea59\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5897d7c8b8-v84qr"
Sep 08 13:50:50.690797 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:50:50.690744    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fc3ac180-6bbd-49dd-81a6-082f6d61ea59-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-5897d7c8b8-v84qr\" (UID: \"fc3ac180-6bbd-49dd-81a6-082f6d61ea59\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5897d7c8b8-v84qr"
Sep 08 13:50:50.690797 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:50:50.690779    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc3ac180-6bbd-49dd-81a6-082f6d61ea59-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-5897d7c8b8-v84qr\" (UID: \"fc3ac180-6bbd-49dd-81a6-082f6d61ea59\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5897d7c8b8-v84qr"
Sep 08 13:50:50.691024 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:50:50.691005    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fc3ac180-6bbd-49dd-81a6-082f6d61ea59-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-5897d7c8b8-v84qr\" (UID: \"fc3ac180-6bbd-49dd-81a6-082f6d61ea59\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5897d7c8b8-v84qr"
Sep 08 13:50:50.692652 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:50:50.692632    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc3ac180-6bbd-49dd-81a6-082f6d61ea59-dshm\") pod \"tokenizer-migration-path-test-kserve-5897d7c8b8-v84qr\" (UID: \"fc3ac180-6bbd-49dd-81a6-082f6d61ea59\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5897d7c8b8-v84qr"
Sep 08 13:50:50.692820 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:50:50.692803    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc3ac180-6bbd-49dd-81a6-082f6d61ea59-tls-certs\") pod \"tokenizer-migration-path-test-kserve-5897d7c8b8-v84qr\" (UID: \"fc3ac180-6bbd-49dd-81a6-082f6d61ea59\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5897d7c8b8-v84qr"
Sep 08 13:50:50.701300 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:50:50.701277    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2rz5\" (UniqueName: \"kubernetes.io/projected/fc3ac180-6bbd-49dd-81a6-082f6d61ea59-kube-api-access-q2rz5\") pod \"tokenizer-migration-path-test-kserve-5897d7c8b8-v84qr\" (UID: \"fc3ac180-6bbd-49dd-81a6-082f6d61ea59\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5897d7c8b8-v84qr"
Sep 08 13:50:50.837554 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:50:50.837477    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5897d7c8b8-v84qr"
Sep 08 13:50:50.861908 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:50:50.861883    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:50:50.972874 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:50:50.972843    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:50:51.117957 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:50:51.117886    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5897d7c8b8-v84qr"]
Sep 08 13:50:51.122430 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:50:51.122397    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc3ac180_6bbd_49dd_81a6_082f6d61ea59.slice/crio-4e62aa89c5e8ac3a47d34b324eb9b2259603d972c34285715ad788f6ea197f76 WatchSource:0}: Error finding container 4e62aa89c5e8ac3a47d34b324eb9b2259603d972c34285715ad788f6ea197f76: Status 404 returned error can't find the container with id 4e62aa89c5e8ac3a47d34b324eb9b2259603d972c34285715ad788f6ea197f76
Sep 08 13:50:51.585395 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:50:51.585349    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5897d7c8b8-v84qr" event={"ID":"fc3ac180-6bbd-49dd-81a6-082f6d61ea59","Type":"ContainerStarted","Data":"1c0be2f4036d23886b39e907b907558b685555b7cec252a46eebe06ffb557249"}
Sep 08 13:50:51.585894 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:50:51.585413    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5897d7c8b8-v84qr" event={"ID":"fc3ac180-6bbd-49dd-81a6-082f6d61ea59","Type":"ContainerStarted","Data":"4e62aa89c5e8ac3a47d34b324eb9b2259603d972c34285715ad788f6ea197f76"}
Sep 08 13:50:53.595121 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:50:53.595039    2573 generic.go:356] "Generic (PLEG): container finished" podID="fc3ac180-6bbd-49dd-81a6-082f6d61ea59" containerID="1c0be2f4036d23886b39e907b907558b685555b7cec252a46eebe06ffb557249" exitCode=0
Sep 08 13:50:53.595464 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:50:53.595115    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5897d7c8b8-v84qr" event={"ID":"fc3ac180-6bbd-49dd-81a6-082f6d61ea59","Type":"ContainerDied","Data":"1c0be2f4036d23886b39e907b907558b685555b7cec252a46eebe06ffb557249"}
Sep 08 13:50:54.481046 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:50:54.481012    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:50:54.601010 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:50:54.600979    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5897d7c8b8-v84qr" event={"ID":"fc3ac180-6bbd-49dd-81a6-082f6d61ea59","Type":"ContainerStarted","Data":"4f4dad53c6fb7e90661e63d7a6de16109a1cf01890f5f588092059e52affc1a6"}
Sep 08 13:50:54.625643 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:50:54.625602    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5897d7c8b8-v84qr" podStartSLOduration=2.576450843 podStartE2EDuration="4.625588128s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.049137285s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:50:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:50:54.623542704 +0000 UTC m=+2104.894990660" watchObservedRunningTime="2026-09-08 13:50:54.625588128 +0000 UTC m=+2104.897036059"
Sep 08 13:50:54.838142 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:50:54.838071    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5897d7c8b8-v84qr"
Sep 08 13:51:00.838053 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:51:00.838020    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5897d7c8b8-v84qr"
Sep 08 13:51:00.850338 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:51:00.850316    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5897d7c8b8-v84qr"
Sep 08 13:51:01.637440 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:51:01.637409    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5897d7c8b8-v84qr"
Sep 08 13:51:07.540453 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:51:07.540416    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:51:07.605166 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:51:07.605140    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:51:09.944071 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:51:09.944032    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:51:10.058388 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:51:10.058329    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:51:10.165492 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:51:10.165456    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:51:15.009338 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:51:15.009302    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:51:21.872710 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:51:21.872677    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:51:23.613205 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:51:23.613175    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:51:40.411965 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:51:40.411928    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:51:50.397463 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:51:50.397380    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-rxhmm"]
Sep 08 13:51:50.400724 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:51:50.400701    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-rxhmm"
Sep 08 13:51:50.404643 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:51:50.404624    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-leave-group-5b69d57a\"/\"leave-v2-kserve-self-signed-certs\""
Sep 08 13:51:50.405495 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:51:50.405475    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-leave-group-5b69d57a\"/\"kube-root-ca.crt\""
Sep 08 13:51:50.405604 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:51:50.405494    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-leave-group-5b69d57a\"/\"openshift-service-ca.crt\""
Sep 08 13:51:50.405604 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:51:50.405534    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-leave-group-5b69d57a\"/\"default-dockercfg-ht47h\""
Sep 08 13:51:50.412998 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:51:50.412980    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-rxhmm"]
Sep 08 13:51:50.441521 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:51:50.441473    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:51:50.534330 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:51:50.534301    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef47c15b-c0e6-4b9b-9432-445d0cd4aefb-home\") pod \"leave-v2-kserve-556b4dddf9-rxhmm\" (UID: \"ef47c15b-c0e6-4b9b-9432-445d0cd4aefb\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-rxhmm"
Sep 08 13:51:50.534469 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:51:50.534337    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef47c15b-c0e6-4b9b-9432-445d0cd4aefb-dshm\") pod \"leave-v2-kserve-556b4dddf9-rxhmm\" (UID: \"ef47c15b-c0e6-4b9b-9432-445d0cd4aefb\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-rxhmm"
Sep 08 13:51:50.534469 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:51:50.534355    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef47c15b-c0e6-4b9b-9432-445d0cd4aefb-tmp-dir\") pod \"leave-v2-kserve-556b4dddf9-rxhmm\" (UID: \"ef47c15b-c0e6-4b9b-9432-445d0cd4aefb\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-rxhmm"
Sep 08 13:51:50.534469 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:51:50.534423    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef47c15b-c0e6-4b9b-9432-445d0cd4aefb-model-cache\") pod \"leave-v2-kserve-556b4dddf9-rxhmm\" (UID: \"ef47c15b-c0e6-4b9b-9432-445d0cd4aefb\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-rxhmm"
Sep 08 13:51:50.534469 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:51:50.534455    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smcxv\" (UniqueName: \"kubernetes.io/projected/ef47c15b-c0e6-4b9b-9432-445d0cd4aefb-kube-api-access-smcxv\") pod \"leave-v2-kserve-556b4dddf9-rxhmm\" (UID: \"ef47c15b-c0e6-4b9b-9432-445d0cd4aefb\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-rxhmm"
Sep 08 13:51:50.534662 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:51:50.534483    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef47c15b-c0e6-4b9b-9432-445d0cd4aefb-tls-certs\") pod \"leave-v2-kserve-556b4dddf9-rxhmm\" (UID: \"ef47c15b-c0e6-4b9b-9432-445d0cd4aefb\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-rxhmm"
Sep 08 13:51:50.635620 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:51:50.635595    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef47c15b-c0e6-4b9b-9432-445d0cd4aefb-model-cache\") pod \"leave-v2-kserve-556b4dddf9-rxhmm\" (UID: \"ef47c15b-c0e6-4b9b-9432-445d0cd4aefb\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-rxhmm"
Sep 08 13:51:50.635768 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:51:50.635629    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smcxv\" (UniqueName: \"kubernetes.io/projected/ef47c15b-c0e6-4b9b-9432-445d0cd4aefb-kube-api-access-smcxv\") pod \"leave-v2-kserve-556b4dddf9-rxhmm\" (UID: \"ef47c15b-c0e6-4b9b-9432-445d0cd4aefb\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-rxhmm"
Sep 08 13:51:50.635768 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:51:50.635659    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef47c15b-c0e6-4b9b-9432-445d0cd4aefb-tls-certs\") pod \"leave-v2-kserve-556b4dddf9-rxhmm\" (UID: \"ef47c15b-c0e6-4b9b-9432-445d0cd4aefb\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-rxhmm"
Sep 08 13:51:50.635768 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:51:50.635727    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef47c15b-c0e6-4b9b-9432-445d0cd4aefb-home\") pod \"leave-v2-kserve-556b4dddf9-rxhmm\" (UID: \"ef47c15b-c0e6-4b9b-9432-445d0cd4aefb\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-rxhmm"
Sep 08 13:51:50.635768 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:51:50.635765    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef47c15b-c0e6-4b9b-9432-445d0cd4aefb-dshm\") pod \"leave-v2-kserve-556b4dddf9-rxhmm\" (UID: \"ef47c15b-c0e6-4b9b-9432-445d0cd4aefb\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-rxhmm"
Sep 08 13:51:50.635980 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:51:50.635791    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef47c15b-c0e6-4b9b-9432-445d0cd4aefb-tmp-dir\") pod \"leave-v2-kserve-556b4dddf9-rxhmm\" (UID: \"ef47c15b-c0e6-4b9b-9432-445d0cd4aefb\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-rxhmm"
Sep 08 13:51:50.636038 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:51:50.635997    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef47c15b-c0e6-4b9b-9432-445d0cd4aefb-model-cache\") pod \"leave-v2-kserve-556b4dddf9-rxhmm\" (UID: \"ef47c15b-c0e6-4b9b-9432-445d0cd4aefb\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-rxhmm"
Sep 08 13:51:50.636179 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:51:50.636160    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef47c15b-c0e6-4b9b-9432-445d0cd4aefb-tmp-dir\") pod \"leave-v2-kserve-556b4dddf9-rxhmm\" (UID: \"ef47c15b-c0e6-4b9b-9432-445d0cd4aefb\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-rxhmm"
Sep 08 13:51:50.636289 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:51:50.636264    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef47c15b-c0e6-4b9b-9432-445d0cd4aefb-home\") pod \"leave-v2-kserve-556b4dddf9-rxhmm\" (UID: \"ef47c15b-c0e6-4b9b-9432-445d0cd4aefb\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-rxhmm"
Sep 08 13:51:50.638199 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:51:50.638177    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef47c15b-c0e6-4b9b-9432-445d0cd4aefb-dshm\") pod \"leave-v2-kserve-556b4dddf9-rxhmm\" (UID: \"ef47c15b-c0e6-4b9b-9432-445d0cd4aefb\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-rxhmm"
Sep 08 13:51:50.638199 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:51:50.638193    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef47c15b-c0e6-4b9b-9432-445d0cd4aefb-tls-certs\") pod \"leave-v2-kserve-556b4dddf9-rxhmm\" (UID: \"ef47c15b-c0e6-4b9b-9432-445d0cd4aefb\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-rxhmm"
Sep 08 13:51:50.644730 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:51:50.644705    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-smcxv\" (UniqueName: \"kubernetes.io/projected/ef47c15b-c0e6-4b9b-9432-445d0cd4aefb-kube-api-access-smcxv\") pod \"leave-v2-kserve-556b4dddf9-rxhmm\" (UID: \"ef47c15b-c0e6-4b9b-9432-445d0cd4aefb\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-rxhmm"
Sep 08 13:51:50.710236 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:51:50.710183    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-rxhmm"
Sep 08 13:51:50.986509 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:51:50.986472    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-rxhmm"]
Sep 08 13:51:50.988246 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:51:50.988211    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef47c15b_c0e6_4b9b_9432_445d0cd4aefb.slice/crio-64dcced1fad76eb9bf8614da69283671b5f80b0e022dee55b5be09160335374e WatchSource:0}: Error finding container 64dcced1fad76eb9bf8614da69283671b5f80b0e022dee55b5be09160335374e: Status 404 returned error can't find the container with id 64dcced1fad76eb9bf8614da69283671b5f80b0e022dee55b5be09160335374e
Sep 08 13:51:51.819346 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:51:51.819307    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-rxhmm" event={"ID":"ef47c15b-c0e6-4b9b-9432-445d0cd4aefb","Type":"ContainerStarted","Data":"724b90e7b2f6df8c588d91e3b300b5d130371a6c34d1c8cb020dff21bbd82638"}
Sep 08 13:51:51.819768 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:51:51.819374    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-rxhmm" event={"ID":"ef47c15b-c0e6-4b9b-9432-445d0cd4aefb","Type":"ContainerStarted","Data":"64dcced1fad76eb9bf8614da69283671b5f80b0e022dee55b5be09160335374e"}
Sep 08 13:51:51.843195 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:51:51.843140    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-rxhmm" podStartSLOduration=1.843121348 podStartE2EDuration="1.843121348s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:51:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:51:51.842359431 +0000 UTC m=+2162.113807375" watchObservedRunningTime="2026-09-08 13:51:51.843121348 +0000 UTC m=+2162.114569280"
Sep 08 13:51:52.711285 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:51:52.711242    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-rxhmm"
Sep 08 13:51:53.376068 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:51:53.376037    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:51:59.309240 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:51:59.309202    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:51:59.410402 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:51:59.410365    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:52:00.710829 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:52:00.710786    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-rxhmm"
Sep 08 13:52:00.723238 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:52:00.723212    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-rxhmm"
Sep 08 13:52:00.870145 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:52:00.870118    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-rxhmm"
Sep 08 13:52:04.728762 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:52:04.728723    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:52:04.740344 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:52:04.740316    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5897d7c8b8-v84qr"]
Sep 08 13:52:04.749342 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:52:04.749297    2573 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 08 13:52:04.749342 ip-10-0-132-254 kubenswrapper[2573]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 08 13:52:04.749342 ip-10-0-132-254 kubenswrapper[2573]:         , stdout: , stderr: , exit code -1
Sep 08 13:52:04.749342 ip-10-0-132-254 kubenswrapper[2573]:  > containerID="4f4dad53c6fb7e90661e63d7a6de16109a1cf01890f5f588092059e52affc1a6" cmd=["/bin/sleep","15"]
Sep 08 13:52:04.749653 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:52:04.749367    2573 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5897d7c8b8-v84qr" message="" err=<
Sep 08 13:52:04.749653 ip-10-0-132-254 kubenswrapper[2573]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 08 13:52:04.749653 ip-10-0-132-254 kubenswrapper[2573]:         , stdout: , stderr: , exit code -1
Sep 08 13:52:04.749653 ip-10-0-132-254 kubenswrapper[2573]:  >
Sep 08 13:52:04.749653 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:52:04.749387    2573 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 08 13:52:04.749653 ip-10-0-132-254 kubenswrapper[2573]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 08 13:52:04.749653 ip-10-0-132-254 kubenswrapper[2573]:         , stdout: , stderr: , exit code -1
Sep 08 13:52:04.749653 ip-10-0-132-254 kubenswrapper[2573]:  > pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5897d7c8b8-v84qr" podUID="fc3ac180-6bbd-49dd-81a6-082f6d61ea59" containerName="main" containerID="cri-o://4f4dad53c6fb7e90661e63d7a6de16109a1cf01890f5f588092059e52affc1a6"
Sep 08 13:52:04.749653 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:52:04.749441    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5897d7c8b8-v84qr" podUID="fc3ac180-6bbd-49dd-81a6-082f6d61ea59" containerName="main" containerID="cri-o://4f4dad53c6fb7e90661e63d7a6de16109a1cf01890f5f588092059e52affc1a6" gracePeriod=60
Sep 08 13:52:04.796629 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:52:04.796602    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:52:05.086158 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:52:05.086136    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5897d7c8b8-v84qr"
Sep 08 13:52:05.251945 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:52:05.251911    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc3ac180-6bbd-49dd-81a6-082f6d61ea59-tls-certs\") pod \"fc3ac180-6bbd-49dd-81a6-082f6d61ea59\" (UID: \"fc3ac180-6bbd-49dd-81a6-082f6d61ea59\") "
Sep 08 13:52:05.252099 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:52:05.251951    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc3ac180-6bbd-49dd-81a6-082f6d61ea59-home\") pod \"fc3ac180-6bbd-49dd-81a6-082f6d61ea59\" (UID: \"fc3ac180-6bbd-49dd-81a6-082f6d61ea59\") "
Sep 08 13:52:05.252099 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:52:05.251969    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc3ac180-6bbd-49dd-81a6-082f6d61ea59-model-cache\") pod \"fc3ac180-6bbd-49dd-81a6-082f6d61ea59\" (UID: \"fc3ac180-6bbd-49dd-81a6-082f6d61ea59\") "
Sep 08 13:52:05.252099 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:52:05.251989    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fc3ac180-6bbd-49dd-81a6-082f6d61ea59-cabundle-cert\") pod \"fc3ac180-6bbd-49dd-81a6-082f6d61ea59\" (UID: \"fc3ac180-6bbd-49dd-81a6-082f6d61ea59\") "
Sep 08 13:52:05.252268 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:52:05.252112    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q2rz5\" (UniqueName: \"kubernetes.io/projected/fc3ac180-6bbd-49dd-81a6-082f6d61ea59-kube-api-access-q2rz5\") pod \"fc3ac180-6bbd-49dd-81a6-082f6d61ea59\" (UID: \"fc3ac180-6bbd-49dd-81a6-082f6d61ea59\") "
Sep 08 13:52:05.252268 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:52:05.252154    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fc3ac180-6bbd-49dd-81a6-082f6d61ea59-kserve-provision-location\") pod \"fc3ac180-6bbd-49dd-81a6-082f6d61ea59\" (UID: \"fc3ac180-6bbd-49dd-81a6-082f6d61ea59\") "
Sep 08 13:52:05.252268 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:52:05.252207    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc3ac180-6bbd-49dd-81a6-082f6d61ea59-dshm\") pod \"fc3ac180-6bbd-49dd-81a6-082f6d61ea59\" (UID: \"fc3ac180-6bbd-49dd-81a6-082f6d61ea59\") "
Sep 08 13:52:05.252268 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:52:05.252221    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc3ac180-6bbd-49dd-81a6-082f6d61ea59-model-cache" (OuterVolumeSpecName: "model-cache") pod "fc3ac180-6bbd-49dd-81a6-082f6d61ea59" (UID: "fc3ac180-6bbd-49dd-81a6-082f6d61ea59"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:52:05.252268 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:52:05.252233    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc3ac180-6bbd-49dd-81a6-082f6d61ea59-home" (OuterVolumeSpecName: "home") pod "fc3ac180-6bbd-49dd-81a6-082f6d61ea59" (UID: "fc3ac180-6bbd-49dd-81a6-082f6d61ea59"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:52:05.252531 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:52:05.252302    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc3ac180-6bbd-49dd-81a6-082f6d61ea59-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "fc3ac180-6bbd-49dd-81a6-082f6d61ea59" (UID: "fc3ac180-6bbd-49dd-81a6-082f6d61ea59"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:52:05.252531 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:52:05.252318    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc3ac180-6bbd-49dd-81a6-082f6d61ea59-tmp-dir\") pod \"fc3ac180-6bbd-49dd-81a6-082f6d61ea59\" (UID: \"fc3ac180-6bbd-49dd-81a6-082f6d61ea59\") "
Sep 08 13:52:05.252635 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:52:05.252610    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc3ac180-6bbd-49dd-81a6-082f6d61ea59-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fc3ac180-6bbd-49dd-81a6-082f6d61ea59" (UID: "fc3ac180-6bbd-49dd-81a6-082f6d61ea59"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:52:05.252635 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:52:05.252629    2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc3ac180-6bbd-49dd-81a6-082f6d61ea59-home\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 13:52:05.252779 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:52:05.252646    2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc3ac180-6bbd-49dd-81a6-082f6d61ea59-model-cache\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 13:52:05.252779 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:52:05.252656    2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fc3ac180-6bbd-49dd-81a6-082f6d61ea59-cabundle-cert\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 13:52:05.254356 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:52:05.254330    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc3ac180-6bbd-49dd-81a6-082f6d61ea59-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fc3ac180-6bbd-49dd-81a6-082f6d61ea59" (UID: "fc3ac180-6bbd-49dd-81a6-082f6d61ea59"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:52:05.254356 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:52:05.254338    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc3ac180-6bbd-49dd-81a6-082f6d61ea59-kube-api-access-q2rz5" (OuterVolumeSpecName: "kube-api-access-q2rz5") pod "fc3ac180-6bbd-49dd-81a6-082f6d61ea59" (UID: "fc3ac180-6bbd-49dd-81a6-082f6d61ea59"). InnerVolumeSpecName "kube-api-access-q2rz5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:52:05.254521 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:52:05.254493    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc3ac180-6bbd-49dd-81a6-082f6d61ea59-dshm" (OuterVolumeSpecName: "dshm") pod "fc3ac180-6bbd-49dd-81a6-082f6d61ea59" (UID: "fc3ac180-6bbd-49dd-81a6-082f6d61ea59"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:52:05.273638 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:52:05.273580    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc3ac180-6bbd-49dd-81a6-082f6d61ea59-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fc3ac180-6bbd-49dd-81a6-082f6d61ea59" (UID: "fc3ac180-6bbd-49dd-81a6-082f6d61ea59"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:52:05.353565 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:52:05.353541    2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc3ac180-6bbd-49dd-81a6-082f6d61ea59-tmp-dir\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 13:52:05.353565 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:52:05.353560    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc3ac180-6bbd-49dd-81a6-082f6d61ea59-tls-certs\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 13:52:05.353687 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:52:05.353570    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2rz5\" (UniqueName: \"kubernetes.io/projected/fc3ac180-6bbd-49dd-81a6-082f6d61ea59-kube-api-access-q2rz5\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 13:52:05.353687 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:52:05.353578    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fc3ac180-6bbd-49dd-81a6-082f6d61ea59-kserve-provision-location\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 13:52:05.353687 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:52:05.353586    2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc3ac180-6bbd-49dd-81a6-082f6d61ea59-dshm\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 13:52:05.882190 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:52:05.882156    2573 generic.go:356] "Generic (PLEG): container finished" podID="fc3ac180-6bbd-49dd-81a6-082f6d61ea59" containerID="4f4dad53c6fb7e90661e63d7a6de16109a1cf01890f5f588092059e52affc1a6" exitCode=0
Sep 08 13:52:05.882605 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:52:05.882213    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5897d7c8b8-v84qr"
Sep 08 13:52:05.882605 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:52:05.882239    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5897d7c8b8-v84qr" event={"ID":"fc3ac180-6bbd-49dd-81a6-082f6d61ea59","Type":"ContainerDied","Data":"4f4dad53c6fb7e90661e63d7a6de16109a1cf01890f5f588092059e52affc1a6"}
Sep 08 13:52:05.882605 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:52:05.882300    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5897d7c8b8-v84qr" event={"ID":"fc3ac180-6bbd-49dd-81a6-082f6d61ea59","Type":"ContainerDied","Data":"4e62aa89c5e8ac3a47d34b324eb9b2259603d972c34285715ad788f6ea197f76"}
Sep 08 13:52:05.882605 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:52:05.882323    2573 scope.go:117] "RemoveContainer" containerID="4f4dad53c6fb7e90661e63d7a6de16109a1cf01890f5f588092059e52affc1a6"
Sep 08 13:52:05.892077 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:52:05.892058    2573 scope.go:117] "RemoveContainer" containerID="1c0be2f4036d23886b39e907b907558b685555b7cec252a46eebe06ffb557249"
Sep 08 13:52:05.899257 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:52:05.899240    2573 scope.go:117] "RemoveContainer" containerID="4f4dad53c6fb7e90661e63d7a6de16109a1cf01890f5f588092059e52affc1a6"
Sep 08 13:52:05.899566 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:52:05.899544    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f4dad53c6fb7e90661e63d7a6de16109a1cf01890f5f588092059e52affc1a6\": container with ID starting with 4f4dad53c6fb7e90661e63d7a6de16109a1cf01890f5f588092059e52affc1a6 not found: ID does not exist" containerID="4f4dad53c6fb7e90661e63d7a6de16109a1cf01890f5f588092059e52affc1a6"
Sep 08 13:52:05.899650 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:52:05.899578    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f4dad53c6fb7e90661e63d7a6de16109a1cf01890f5f588092059e52affc1a6"} err="failed to get container status \"4f4dad53c6fb7e90661e63d7a6de16109a1cf01890f5f588092059e52affc1a6\": rpc error: code = NotFound desc = could not find container \"4f4dad53c6fb7e90661e63d7a6de16109a1cf01890f5f588092059e52affc1a6\": container with ID starting with 4f4dad53c6fb7e90661e63d7a6de16109a1cf01890f5f588092059e52affc1a6 not found: ID does not exist"
Sep 08 13:52:05.899650 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:52:05.899602    2573 scope.go:117] "RemoveContainer" containerID="1c0be2f4036d23886b39e907b907558b685555b7cec252a46eebe06ffb557249"
Sep 08 13:52:05.899896 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:52:05.899876    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c0be2f4036d23886b39e907b907558b685555b7cec252a46eebe06ffb557249\": container with ID starting with 1c0be2f4036d23886b39e907b907558b685555b7cec252a46eebe06ffb557249 not found: ID does not exist" containerID="1c0be2f4036d23886b39e907b907558b685555b7cec252a46eebe06ffb557249"
Sep 08 13:52:05.899949 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:52:05.899901    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c0be2f4036d23886b39e907b907558b685555b7cec252a46eebe06ffb557249"} err="failed to get container status \"1c0be2f4036d23886b39e907b907558b685555b7cec252a46eebe06ffb557249\": rpc error: code = NotFound desc = could not find container \"1c0be2f4036d23886b39e907b907558b685555b7cec252a46eebe06ffb557249\": container with ID starting with 1c0be2f4036d23886b39e907b907558b685555b7cec252a46eebe06ffb557249 not found: ID does not exist"
Sep 08 13:52:05.905061 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:52:05.905039    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5897d7c8b8-v84qr"]
Sep 08 13:52:05.911181 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:52:05.911151    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5897d7c8b8-v84qr"]
Sep 08 13:52:06.174421 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:52:06.174348    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc3ac180-6bbd-49dd-81a6-082f6d61ea59" path="/var/lib/kubelet/pods/fc3ac180-6bbd-49dd-81a6-082f6d61ea59/volumes"
Sep 08 13:52:09.836026 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:52:09.835989    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:52:09.942409 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:52:09.942375    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:52:10.044474 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:52:10.044447    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:52:10.329629 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:52:10.329597    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-rxhmm"]
Sep 08 13:52:10.358555 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:52:10.358524    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:52:10.733063 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:52:10.733034    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:52:16.129937 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:52:16.129908    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:52:18.931887 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:52:18.931852    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:52:25.348869 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:52:25.348825    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-rxhmm" podUID="ef47c15b-c0e6-4b9b-9432-445d0cd4aefb" containerName="main" containerID="cri-o://724b90e7b2f6df8c588d91e3b300b5d130371a6c34d1c8cb020dff21bbd82638" gracePeriod=45
Sep 08 13:52:25.591745 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:52:25.591723    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-rxhmm"
Sep 08 13:52:25.604601 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:52:25.604552    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef47c15b-c0e6-4b9b-9432-445d0cd4aefb-dshm\") pod \"ef47c15b-c0e6-4b9b-9432-445d0cd4aefb\" (UID: \"ef47c15b-c0e6-4b9b-9432-445d0cd4aefb\") "
Sep 08 13:52:25.604601 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:52:25.604582    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-smcxv\" (UniqueName: \"kubernetes.io/projected/ef47c15b-c0e6-4b9b-9432-445d0cd4aefb-kube-api-access-smcxv\") pod \"ef47c15b-c0e6-4b9b-9432-445d0cd4aefb\" (UID: \"ef47c15b-c0e6-4b9b-9432-445d0cd4aefb\") "
Sep 08 13:52:25.604601 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:52:25.604601    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef47c15b-c0e6-4b9b-9432-445d0cd4aefb-home\") pod \"ef47c15b-c0e6-4b9b-9432-445d0cd4aefb\" (UID: \"ef47c15b-c0e6-4b9b-9432-445d0cd4aefb\") "
Sep 08 13:52:25.604811 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:52:25.604635    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef47c15b-c0e6-4b9b-9432-445d0cd4aefb-tmp-dir\") pod \"ef47c15b-c0e6-4b9b-9432-445d0cd4aefb\" (UID: \"ef47c15b-c0e6-4b9b-9432-445d0cd4aefb\") "
Sep 08 13:52:25.604811 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:52:25.604691    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef47c15b-c0e6-4b9b-9432-445d0cd4aefb-model-cache\") pod \"ef47c15b-c0e6-4b9b-9432-445d0cd4aefb\" (UID: \"ef47c15b-c0e6-4b9b-9432-445d0cd4aefb\") "
Sep 08 13:52:25.604811 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:52:25.604706    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef47c15b-c0e6-4b9b-9432-445d0cd4aefb-tls-certs\") pod \"ef47c15b-c0e6-4b9b-9432-445d0cd4aefb\" (UID: \"ef47c15b-c0e6-4b9b-9432-445d0cd4aefb\") "
Sep 08 13:52:25.604978 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:52:25.604932    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef47c15b-c0e6-4b9b-9432-445d0cd4aefb-home" (OuterVolumeSpecName: "home") pod "ef47c15b-c0e6-4b9b-9432-445d0cd4aefb" (UID: "ef47c15b-c0e6-4b9b-9432-445d0cd4aefb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:52:25.604978 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:52:25.604945    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef47c15b-c0e6-4b9b-9432-445d0cd4aefb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ef47c15b-c0e6-4b9b-9432-445d0cd4aefb" (UID: "ef47c15b-c0e6-4b9b-9432-445d0cd4aefb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:52:25.605057 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:52:25.604976    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef47c15b-c0e6-4b9b-9432-445d0cd4aefb-model-cache" (OuterVolumeSpecName: "model-cache") pod "ef47c15b-c0e6-4b9b-9432-445d0cd4aefb" (UID: "ef47c15b-c0e6-4b9b-9432-445d0cd4aefb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:52:25.606748 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:52:25.606721    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef47c15b-c0e6-4b9b-9432-445d0cd4aefb-dshm" (OuterVolumeSpecName: "dshm") pod "ef47c15b-c0e6-4b9b-9432-445d0cd4aefb" (UID: "ef47c15b-c0e6-4b9b-9432-445d0cd4aefb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:52:25.606859 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:52:25.606768    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef47c15b-c0e6-4b9b-9432-445d0cd4aefb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ef47c15b-c0e6-4b9b-9432-445d0cd4aefb" (UID: "ef47c15b-c0e6-4b9b-9432-445d0cd4aefb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:52:25.606859 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:52:25.606817    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef47c15b-c0e6-4b9b-9432-445d0cd4aefb-kube-api-access-smcxv" (OuterVolumeSpecName: "kube-api-access-smcxv") pod "ef47c15b-c0e6-4b9b-9432-445d0cd4aefb" (UID: "ef47c15b-c0e6-4b9b-9432-445d0cd4aefb"). InnerVolumeSpecName "kube-api-access-smcxv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:52:25.705233 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:52:25.705202    2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef47c15b-c0e6-4b9b-9432-445d0cd4aefb-dshm\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 13:52:25.705233 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:52:25.705227    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smcxv\" (UniqueName: \"kubernetes.io/projected/ef47c15b-c0e6-4b9b-9432-445d0cd4aefb-kube-api-access-smcxv\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 13:52:25.705233 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:52:25.705238    2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef47c15b-c0e6-4b9b-9432-445d0cd4aefb-home\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 13:52:25.705484 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:52:25.705263    2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef47c15b-c0e6-4b9b-9432-445d0cd4aefb-tmp-dir\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 13:52:25.705484 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:52:25.705272    2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef47c15b-c0e6-4b9b-9432-445d0cd4aefb-model-cache\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 13:52:25.705484 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:52:25.705280    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef47c15b-c0e6-4b9b-9432-445d0cd4aefb-tls-certs\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 13:52:25.967046 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:52:25.967013    2573 generic.go:356] "Generic (PLEG): container finished" podID="ef47c15b-c0e6-4b9b-9432-445d0cd4aefb" containerID="724b90e7b2f6df8c588d91e3b300b5d130371a6c34d1c8cb020dff21bbd82638" exitCode=0
Sep 08 13:52:25.967209 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:52:25.967062    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-rxhmm"
Sep 08 13:52:25.967209 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:52:25.967074    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-rxhmm" event={"ID":"ef47c15b-c0e6-4b9b-9432-445d0cd4aefb","Type":"ContainerDied","Data":"724b90e7b2f6df8c588d91e3b300b5d130371a6c34d1c8cb020dff21bbd82638"}
Sep 08 13:52:25.967209 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:52:25.967129    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-rxhmm" event={"ID":"ef47c15b-c0e6-4b9b-9432-445d0cd4aefb","Type":"ContainerDied","Data":"64dcced1fad76eb9bf8614da69283671b5f80b0e022dee55b5be09160335374e"}
Sep 08 13:52:25.967209 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:52:25.967160    2573 scope.go:117] "RemoveContainer" containerID="724b90e7b2f6df8c588d91e3b300b5d130371a6c34d1c8cb020dff21bbd82638"
Sep 08 13:52:25.975935 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:52:25.975909    2573 scope.go:117] "RemoveContainer" containerID="724b90e7b2f6df8c588d91e3b300b5d130371a6c34d1c8cb020dff21bbd82638"
Sep 08 13:52:25.976197 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:52:25.976177    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"724b90e7b2f6df8c588d91e3b300b5d130371a6c34d1c8cb020dff21bbd82638\": container with ID starting with 724b90e7b2f6df8c588d91e3b300b5d130371a6c34d1c8cb020dff21bbd82638 not found: ID does not exist" containerID="724b90e7b2f6df8c588d91e3b300b5d130371a6c34d1c8cb020dff21bbd82638"
Sep 08 13:52:25.976236 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:52:25.976205    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"724b90e7b2f6df8c588d91e3b300b5d130371a6c34d1c8cb020dff21bbd82638"} err="failed to get container status \"724b90e7b2f6df8c588d91e3b300b5d130371a6c34d1c8cb020dff21bbd82638\": rpc error: code = NotFound desc = could not find container \"724b90e7b2f6df8c588d91e3b300b5d130371a6c34d1c8cb020dff21bbd82638\": container with ID starting with 724b90e7b2f6df8c588d91e3b300b5d130371a6c34d1c8cb020dff21bbd82638 not found: ID does not exist"
Sep 08 13:52:25.989540 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:52:25.989492    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-rxhmm"]
Sep 08 13:52:25.993049 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:52:25.993027    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-rxhmm"]
Sep 08 13:52:26.175169 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:52:26.175143    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ef47c15b-c0e6-4b9b-9432-445d0cd4aefb" path="/var/lib/kubelet/pods/ef47c15b-c0e6-4b9b-9432-445d0cd4aefb/volumes"
Sep 08 13:52:28.343145 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:52:28.343117    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:52:34.614442 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:52:34.614412    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:52:52.487263 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:52:52.487228    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:52:54.375859 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:52:54.375823    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:52:54.473924 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:52:54.473897    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:52:57.594016 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:52:57.593987    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:52:57.691387 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:52:57.691359    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:52:58.124133 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:52:58.124104    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:52:58.224624 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:52:58.224594    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:53:00.957895 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:53:00.957868    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:53:06.850043 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:53:06.850006    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:53:06.953744 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:53:06.953711    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:53:37.310871 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:53:37.310836    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57ff94957b-846r5"]
Sep 08 13:53:37.311278 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:53:37.311163    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef47c15b-c0e6-4b9b-9432-445d0cd4aefb" containerName="main"
Sep 08 13:53:37.311278 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:53:37.311176    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef47c15b-c0e6-4b9b-9432-445d0cd4aefb" containerName="main"
Sep 08 13:53:37.311278 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:53:37.311192    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc3ac180-6bbd-49dd-81a6-082f6d61ea59" containerName="main"
Sep 08 13:53:37.311278 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:53:37.311198    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc3ac180-6bbd-49dd-81a6-082f6d61ea59" containerName="main"
Sep 08 13:53:37.311278 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:53:37.311211    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc3ac180-6bbd-49dd-81a6-082f6d61ea59" containerName="storage-initializer"
Sep 08 13:53:37.311278 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:53:37.311217    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc3ac180-6bbd-49dd-81a6-082f6d61ea59" containerName="storage-initializer"
Sep 08 13:53:37.311471 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:53:37.311280    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef47c15b-c0e6-4b9b-9432-445d0cd4aefb" containerName="main"
Sep 08 13:53:37.311471 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:53:37.311289    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc3ac180-6bbd-49dd-81a6-082f6d61ea59" containerName="main"
Sep 08 13:53:37.314593 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:53:37.314572    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57ff94957b-846r5"
Sep 08 13:53:37.317484 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:53:37.317462    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"kube-root-ca.crt\""
Sep 08 13:53:37.318208 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:53:37.318191    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"default-dockercfg-pwm66\""
Sep 08 13:53:37.318318 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:53:37.318258    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"seaweedfs-s3-creds\""
Sep 08 13:53:37.318318 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:53:37.318309    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"openshift-service-ca.crt\""
Sep 08 13:53:37.318599 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:53:37.318578    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"tokenizer-template-args-test-kserve-self-signed-certs\""
Sep 08 13:53:37.318701 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:53:37.318642    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"odh-kserve-custom-ca-bundle\""
Sep 08 13:53:37.332669 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:53:37.332649    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57ff94957b-846r5"]
Sep 08 13:53:37.363410 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:53:37.363387    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:53:37.431635 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:53:37.431609    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a435301f-52e5-4ad0-be11-fe8d070676c5-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-57ff94957b-846r5\" (UID: \"a435301f-52e5-4ad0-be11-fe8d070676c5\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57ff94957b-846r5"
Sep 08 13:53:37.431741 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:53:37.431648    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a435301f-52e5-4ad0-be11-fe8d070676c5-model-cache\") pod \"tokenizer-template-args-test-kserve-57ff94957b-846r5\" (UID: \"a435301f-52e5-4ad0-be11-fe8d070676c5\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57ff94957b-846r5"
Sep 08 13:53:37.431741 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:53:37.431666    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a435301f-52e5-4ad0-be11-fe8d070676c5-tmp-dir\") pod \"tokenizer-template-args-test-kserve-57ff94957b-846r5\" (UID: \"a435301f-52e5-4ad0-be11-fe8d070676c5\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57ff94957b-846r5"
Sep 08 13:53:37.431741 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:53:37.431726    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6j4r\" (UniqueName: \"kubernetes.io/projected/a435301f-52e5-4ad0-be11-fe8d070676c5-kube-api-access-g6j4r\") pod \"tokenizer-template-args-test-kserve-57ff94957b-846r5\" (UID: \"a435301f-52e5-4ad0-be11-fe8d070676c5\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57ff94957b-846r5"
Sep 08 13:53:37.431882 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:53:37.431769    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a435301f-52e5-4ad0-be11-fe8d070676c5-tls-certs\") pod \"tokenizer-template-args-test-kserve-57ff94957b-846r5\" (UID: \"a435301f-52e5-4ad0-be11-fe8d070676c5\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57ff94957b-846r5"
Sep 08 13:53:37.431882 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:53:37.431812    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a435301f-52e5-4ad0-be11-fe8d070676c5-dshm\") pod \"tokenizer-template-args-test-kserve-57ff94957b-846r5\" (UID: \"a435301f-52e5-4ad0-be11-fe8d070676c5\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57ff94957b-846r5"
Sep 08 13:53:37.432011 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:53:37.431869    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a435301f-52e5-4ad0-be11-fe8d070676c5-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-57ff94957b-846r5\" (UID: \"a435301f-52e5-4ad0-be11-fe8d070676c5\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57ff94957b-846r5"
Sep 08 13:53:37.432011 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:53:37.431936    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a435301f-52e5-4ad0-be11-fe8d070676c5-home\") pod \"tokenizer-template-args-test-kserve-57ff94957b-846r5\" (UID: \"a435301f-52e5-4ad0-be11-fe8d070676c5\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57ff94957b-846r5"
Sep 08 13:53:37.532594 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:53:37.532569    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a435301f-52e5-4ad0-be11-fe8d070676c5-dshm\") pod \"tokenizer-template-args-test-kserve-57ff94957b-846r5\" (UID: \"a435301f-52e5-4ad0-be11-fe8d070676c5\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57ff94957b-846r5"
Sep 08 13:53:37.532710 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:53:37.532600    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a435301f-52e5-4ad0-be11-fe8d070676c5-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-57ff94957b-846r5\" (UID: \"a435301f-52e5-4ad0-be11-fe8d070676c5\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57ff94957b-846r5"
Sep 08 13:53:37.532710 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:53:37.532626    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a435301f-52e5-4ad0-be11-fe8d070676c5-home\") pod \"tokenizer-template-args-test-kserve-57ff94957b-846r5\" (UID: \"a435301f-52e5-4ad0-be11-fe8d070676c5\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57ff94957b-846r5"
Sep 08 13:53:37.532821 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:53:37.532741    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a435301f-52e5-4ad0-be11-fe8d070676c5-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-57ff94957b-846r5\" (UID: \"a435301f-52e5-4ad0-be11-fe8d070676c5\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57ff94957b-846r5"
Sep 08 13:53:37.532821 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:53:37.532796    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a435301f-52e5-4ad0-be11-fe8d070676c5-model-cache\") pod \"tokenizer-template-args-test-kserve-57ff94957b-846r5\" (UID: \"a435301f-52e5-4ad0-be11-fe8d070676c5\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57ff94957b-846r5"
Sep 08 13:53:37.532927 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:53:37.532822    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a435301f-52e5-4ad0-be11-fe8d070676c5-tmp-dir\") pod \"tokenizer-template-args-test-kserve-57ff94957b-846r5\" (UID: \"a435301f-52e5-4ad0-be11-fe8d070676c5\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57ff94957b-846r5"
Sep 08 13:53:37.532927 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:53:37.532848    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6j4r\" (UniqueName: \"kubernetes.io/projected/a435301f-52e5-4ad0-be11-fe8d070676c5-kube-api-access-g6j4r\") pod \"tokenizer-template-args-test-kserve-57ff94957b-846r5\" (UID: \"a435301f-52e5-4ad0-be11-fe8d070676c5\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57ff94957b-846r5"
Sep 08 13:53:37.533495 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:53:37.533317    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a435301f-52e5-4ad0-be11-fe8d070676c5-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-57ff94957b-846r5\" (UID: \"a435301f-52e5-4ad0-be11-fe8d070676c5\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57ff94957b-846r5"
Sep 08 13:53:37.533495 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:53:37.533371    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a435301f-52e5-4ad0-be11-fe8d070676c5-tls-certs\") pod \"tokenizer-template-args-test-kserve-57ff94957b-846r5\" (UID: \"a435301f-52e5-4ad0-be11-fe8d070676c5\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57ff94957b-846r5"
Sep 08 13:53:37.533687 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:53:37.533562    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a435301f-52e5-4ad0-be11-fe8d070676c5-home\") pod \"tokenizer-template-args-test-kserve-57ff94957b-846r5\" (UID: \"a435301f-52e5-4ad0-be11-fe8d070676c5\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57ff94957b-846r5"
Sep 08 13:53:37.533753 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:53:37.533728    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a435301f-52e5-4ad0-be11-fe8d070676c5-model-cache\") pod \"tokenizer-template-args-test-kserve-57ff94957b-846r5\" (UID: \"a435301f-52e5-4ad0-be11-fe8d070676c5\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57ff94957b-846r5"
Sep 08 13:53:37.533813 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:53:37.533595    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a435301f-52e5-4ad0-be11-fe8d070676c5-tmp-dir\") pod \"tokenizer-template-args-test-kserve-57ff94957b-846r5\" (UID: \"a435301f-52e5-4ad0-be11-fe8d070676c5\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57ff94957b-846r5"
Sep 08 13:53:37.533904 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:53:37.533869    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a435301f-52e5-4ad0-be11-fe8d070676c5-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-57ff94957b-846r5\" (UID: \"a435301f-52e5-4ad0-be11-fe8d070676c5\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57ff94957b-846r5"
Sep 08 13:53:37.539254 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:53:37.536022    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a435301f-52e5-4ad0-be11-fe8d070676c5-dshm\") pod \"tokenizer-template-args-test-kserve-57ff94957b-846r5\" (UID: \"a435301f-52e5-4ad0-be11-fe8d070676c5\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57ff94957b-846r5"
Sep 08 13:53:37.540334 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:53:37.540311    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a435301f-52e5-4ad0-be11-fe8d070676c5-tls-certs\") pod \"tokenizer-template-args-test-kserve-57ff94957b-846r5\" (UID: \"a435301f-52e5-4ad0-be11-fe8d070676c5\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57ff94957b-846r5"
Sep 08 13:53:37.542103 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:53:37.542083    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6j4r\" (UniqueName: \"kubernetes.io/projected/a435301f-52e5-4ad0-be11-fe8d070676c5-kube-api-access-g6j4r\") pod \"tokenizer-template-args-test-kserve-57ff94957b-846r5\" (UID: \"a435301f-52e5-4ad0-be11-fe8d070676c5\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57ff94957b-846r5"
Sep 08 13:53:37.624842 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:53:37.624815    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57ff94957b-846r5"
Sep 08 13:53:37.680978 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:53:37.680943    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:53:37.806508 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:53:37.806476    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:53:37.897864 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:53:37.897840    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57ff94957b-846r5"]
Sep 08 13:53:37.900125 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:53:37.900088    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda435301f_52e5_4ad0_be11_fe8d070676c5.slice/crio-34dba74bdf3c2115866da724ba5557eae39c4bf699f7a794a86f7fcc695d17ef WatchSource:0}: Error finding container 34dba74bdf3c2115866da724ba5557eae39c4bf699f7a794a86f7fcc695d17ef: Status 404 returned error can't find the container with id 34dba74bdf3c2115866da724ba5557eae39c4bf699f7a794a86f7fcc695d17ef
Sep 08 13:53:38.255912 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:53:38.255820    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57ff94957b-846r5" event={"ID":"a435301f-52e5-4ad0-be11-fe8d070676c5","Type":"ContainerStarted","Data":"62ec2fc76e74ba2ff1f69eb722119a66757aed549ec0cc7a52330d77bcdea682"}
Sep 08 13:53:38.255912 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:53:38.255879    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57ff94957b-846r5" event={"ID":"a435301f-52e5-4ad0-be11-fe8d070676c5","Type":"ContainerStarted","Data":"34dba74bdf3c2115866da724ba5557eae39c4bf699f7a794a86f7fcc695d17ef"}
Sep 08 13:53:40.266410 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:53:40.266379    2573 generic.go:356] "Generic (PLEG): container finished" podID="a435301f-52e5-4ad0-be11-fe8d070676c5" containerID="62ec2fc76e74ba2ff1f69eb722119a66757aed549ec0cc7a52330d77bcdea682" exitCode=0
Sep 08 13:53:40.266795 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:53:40.266435    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57ff94957b-846r5" event={"ID":"a435301f-52e5-4ad0-be11-fe8d070676c5","Type":"ContainerDied","Data":"62ec2fc76e74ba2ff1f69eb722119a66757aed549ec0cc7a52330d77bcdea682"}
Sep 08 13:53:41.272414 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:53:41.272377    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57ff94957b-846r5" event={"ID":"a435301f-52e5-4ad0-be11-fe8d070676c5","Type":"ContainerStarted","Data":"1c6d54f4a5f0242f8bb85a18e39d808a5ea7192175fd795fea7a7dbd183fe135"}
Sep 08 13:53:41.297219 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:53:41.297169    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57ff94957b-846r5" podStartSLOduration=2.206409697 podStartE2EDuration="4.297151321s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.090741624s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:53:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:53:41.296919664 +0000 UTC m=+2271.568367605" watchObservedRunningTime="2026-09-08 13:53:41.297151321 +0000 UTC m=+2271.568599253"
Sep 08 13:53:41.625881 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:53:41.625852    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57ff94957b-846r5"
Sep 08 13:53:47.625444 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:53:47.625407    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57ff94957b-846r5"
Sep 08 13:53:47.637796 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:53:47.637772    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57ff94957b-846r5"
Sep 08 13:53:48.309208 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:53:48.309178    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57ff94957b-846r5"
Sep 08 13:54:00.959167 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:54:00.959133    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:54:14.543608 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:54:14.543573    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:54:14.561758 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:54:14.561730    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57ff94957b-846r5"]
Sep 08 13:54:14.570412 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:54:14.570372    2573 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 08 13:54:14.570412 ip-10-0-132-254 kubenswrapper[2573]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 08 13:54:14.570412 ip-10-0-132-254 kubenswrapper[2573]:         , stdout: , stderr: , exit code -1
Sep 08 13:54:14.570412 ip-10-0-132-254 kubenswrapper[2573]:  > containerID="1c6d54f4a5f0242f8bb85a18e39d808a5ea7192175fd795fea7a7dbd183fe135" cmd=["/bin/sleep","15"]
Sep 08 13:54:14.570684 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:54:14.570440    2573 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57ff94957b-846r5" message="" err=<
Sep 08 13:54:14.570684 ip-10-0-132-254 kubenswrapper[2573]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 08 13:54:14.570684 ip-10-0-132-254 kubenswrapper[2573]:         , stdout: , stderr: , exit code -1
Sep 08 13:54:14.570684 ip-10-0-132-254 kubenswrapper[2573]:  >
Sep 08 13:54:14.570684 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:54:14.570458    2573 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 08 13:54:14.570684 ip-10-0-132-254 kubenswrapper[2573]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 08 13:54:14.570684 ip-10-0-132-254 kubenswrapper[2573]:         , stdout: , stderr: , exit code -1
Sep 08 13:54:14.570684 ip-10-0-132-254 kubenswrapper[2573]:  > pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57ff94957b-846r5" podUID="a435301f-52e5-4ad0-be11-fe8d070676c5" containerName="main" containerID="cri-o://1c6d54f4a5f0242f8bb85a18e39d808a5ea7192175fd795fea7a7dbd183fe135"
Sep 08 13:54:14.570684 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:54:14.570526    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57ff94957b-846r5" podUID="a435301f-52e5-4ad0-be11-fe8d070676c5" containerName="main" containerID="cri-o://1c6d54f4a5f0242f8bb85a18e39d808a5ea7192175fd795fea7a7dbd183fe135" gracePeriod=60
Sep 08 13:54:14.614035 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:54:14.614010    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:54:14.911836 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:54:14.911815    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57ff94957b-846r5"
Sep 08 13:54:15.015794 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:54:15.015771    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a435301f-52e5-4ad0-be11-fe8d070676c5-tmp-dir\") pod \"a435301f-52e5-4ad0-be11-fe8d070676c5\" (UID: \"a435301f-52e5-4ad0-be11-fe8d070676c5\") "
Sep 08 13:54:15.015973 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:54:15.015800    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a435301f-52e5-4ad0-be11-fe8d070676c5-home\") pod \"a435301f-52e5-4ad0-be11-fe8d070676c5\" (UID: \"a435301f-52e5-4ad0-be11-fe8d070676c5\") "
Sep 08 13:54:15.015973 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:54:15.015820    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a435301f-52e5-4ad0-be11-fe8d070676c5-kserve-provision-location\") pod \"a435301f-52e5-4ad0-be11-fe8d070676c5\" (UID: \"a435301f-52e5-4ad0-be11-fe8d070676c5\") "
Sep 08 13:54:15.015973 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:54:15.015839    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a435301f-52e5-4ad0-be11-fe8d070676c5-dshm\") pod \"a435301f-52e5-4ad0-be11-fe8d070676c5\" (UID: \"a435301f-52e5-4ad0-be11-fe8d070676c5\") "
Sep 08 13:54:15.015973 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:54:15.015864    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a435301f-52e5-4ad0-be11-fe8d070676c5-cabundle-cert\") pod \"a435301f-52e5-4ad0-be11-fe8d070676c5\" (UID: \"a435301f-52e5-4ad0-be11-fe8d070676c5\") "
Sep 08 13:54:15.015973 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:54:15.015913    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a435301f-52e5-4ad0-be11-fe8d070676c5-model-cache\") pod \"a435301f-52e5-4ad0-be11-fe8d070676c5\" (UID: \"a435301f-52e5-4ad0-be11-fe8d070676c5\") "
Sep 08 13:54:15.015973 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:54:15.015931    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g6j4r\" (UniqueName: \"kubernetes.io/projected/a435301f-52e5-4ad0-be11-fe8d070676c5-kube-api-access-g6j4r\") pod \"a435301f-52e5-4ad0-be11-fe8d070676c5\" (UID: \"a435301f-52e5-4ad0-be11-fe8d070676c5\") "
Sep 08 13:54:15.016322 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:54:15.015982    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a435301f-52e5-4ad0-be11-fe8d070676c5-tls-certs\") pod \"a435301f-52e5-4ad0-be11-fe8d070676c5\" (UID: \"a435301f-52e5-4ad0-be11-fe8d070676c5\") "
Sep 08 13:54:15.016322 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:54:15.016079    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a435301f-52e5-4ad0-be11-fe8d070676c5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a435301f-52e5-4ad0-be11-fe8d070676c5" (UID: "a435301f-52e5-4ad0-be11-fe8d070676c5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:54:15.016322 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:54:15.016097    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a435301f-52e5-4ad0-be11-fe8d070676c5-home" (OuterVolumeSpecName: "home") pod "a435301f-52e5-4ad0-be11-fe8d070676c5" (UID: "a435301f-52e5-4ad0-be11-fe8d070676c5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:54:15.016322 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:54:15.016261    2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a435301f-52e5-4ad0-be11-fe8d070676c5-tmp-dir\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 13:54:15.016322 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:54:15.016279    2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a435301f-52e5-4ad0-be11-fe8d070676c5-home\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 13:54:15.016592 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:54:15.016332    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a435301f-52e5-4ad0-be11-fe8d070676c5-model-cache" (OuterVolumeSpecName: "model-cache") pod "a435301f-52e5-4ad0-be11-fe8d070676c5" (UID: "a435301f-52e5-4ad0-be11-fe8d070676c5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:54:15.016660 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:54:15.016589    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a435301f-52e5-4ad0-be11-fe8d070676c5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a435301f-52e5-4ad0-be11-fe8d070676c5" (UID: "a435301f-52e5-4ad0-be11-fe8d070676c5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:54:15.018324 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:54:15.018299    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a435301f-52e5-4ad0-be11-fe8d070676c5-kube-api-access-g6j4r" (OuterVolumeSpecName: "kube-api-access-g6j4r") pod "a435301f-52e5-4ad0-be11-fe8d070676c5" (UID: "a435301f-52e5-4ad0-be11-fe8d070676c5"). InnerVolumeSpecName "kube-api-access-g6j4r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:54:15.018429 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:54:15.018391    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a435301f-52e5-4ad0-be11-fe8d070676c5-dshm" (OuterVolumeSpecName: "dshm") pod "a435301f-52e5-4ad0-be11-fe8d070676c5" (UID: "a435301f-52e5-4ad0-be11-fe8d070676c5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:54:15.018429 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:54:15.018412    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a435301f-52e5-4ad0-be11-fe8d070676c5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a435301f-52e5-4ad0-be11-fe8d070676c5" (UID: "a435301f-52e5-4ad0-be11-fe8d070676c5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:54:15.043062 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:54:15.043030    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a435301f-52e5-4ad0-be11-fe8d070676c5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a435301f-52e5-4ad0-be11-fe8d070676c5" (UID: "a435301f-52e5-4ad0-be11-fe8d070676c5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:54:15.117380 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:54:15.117329    2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a435301f-52e5-4ad0-be11-fe8d070676c5-model-cache\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 13:54:15.117380 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:54:15.117349    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6j4r\" (UniqueName: \"kubernetes.io/projected/a435301f-52e5-4ad0-be11-fe8d070676c5-kube-api-access-g6j4r\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 13:54:15.117380 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:54:15.117358    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a435301f-52e5-4ad0-be11-fe8d070676c5-tls-certs\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 13:54:15.117380 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:54:15.117368    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a435301f-52e5-4ad0-be11-fe8d070676c5-kserve-provision-location\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 13:54:15.117380 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:54:15.117376    2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a435301f-52e5-4ad0-be11-fe8d070676c5-dshm\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 13:54:15.117604 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:54:15.117384    2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a435301f-52e5-4ad0-be11-fe8d070676c5-cabundle-cert\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 13:54:15.402636 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:54:15.402604    2573 generic.go:356] "Generic (PLEG): container finished" podID="a435301f-52e5-4ad0-be11-fe8d070676c5" containerID="1c6d54f4a5f0242f8bb85a18e39d808a5ea7192175fd795fea7a7dbd183fe135" exitCode=0
Sep 08 13:54:15.402790 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:54:15.402658    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57ff94957b-846r5" event={"ID":"a435301f-52e5-4ad0-be11-fe8d070676c5","Type":"ContainerDied","Data":"1c6d54f4a5f0242f8bb85a18e39d808a5ea7192175fd795fea7a7dbd183fe135"}
Sep 08 13:54:15.402790 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:54:15.402698    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57ff94957b-846r5" event={"ID":"a435301f-52e5-4ad0-be11-fe8d070676c5","Type":"ContainerDied","Data":"34dba74bdf3c2115866da724ba5557eae39c4bf699f7a794a86f7fcc695d17ef"}
Sep 08 13:54:15.402790 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:54:15.402659    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57ff94957b-846r5"
Sep 08 13:54:15.402906 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:54:15.402729    2573 scope.go:117] "RemoveContainer" containerID="1c6d54f4a5f0242f8bb85a18e39d808a5ea7192175fd795fea7a7dbd183fe135"
Sep 08 13:54:15.411634 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:54:15.411617    2573 scope.go:117] "RemoveContainer" containerID="62ec2fc76e74ba2ff1f69eb722119a66757aed549ec0cc7a52330d77bcdea682"
Sep 08 13:54:15.418708 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:54:15.418689    2573 scope.go:117] "RemoveContainer" containerID="1c6d54f4a5f0242f8bb85a18e39d808a5ea7192175fd795fea7a7dbd183fe135"
Sep 08 13:54:15.418981 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:54:15.418961    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c6d54f4a5f0242f8bb85a18e39d808a5ea7192175fd795fea7a7dbd183fe135\": container with ID starting with 1c6d54f4a5f0242f8bb85a18e39d808a5ea7192175fd795fea7a7dbd183fe135 not found: ID does not exist" containerID="1c6d54f4a5f0242f8bb85a18e39d808a5ea7192175fd795fea7a7dbd183fe135"
Sep 08 13:54:15.419035 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:54:15.418988    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c6d54f4a5f0242f8bb85a18e39d808a5ea7192175fd795fea7a7dbd183fe135"} err="failed to get container status \"1c6d54f4a5f0242f8bb85a18e39d808a5ea7192175fd795fea7a7dbd183fe135\": rpc error: code = NotFound desc = could not find container \"1c6d54f4a5f0242f8bb85a18e39d808a5ea7192175fd795fea7a7dbd183fe135\": container with ID starting with 1c6d54f4a5f0242f8bb85a18e39d808a5ea7192175fd795fea7a7dbd183fe135 not found: ID does not exist"
Sep 08 13:54:15.419035 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:54:15.419005    2573 scope.go:117] "RemoveContainer" containerID="62ec2fc76e74ba2ff1f69eb722119a66757aed549ec0cc7a52330d77bcdea682"
Sep 08 13:54:15.419255 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:54:15.419237    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62ec2fc76e74ba2ff1f69eb722119a66757aed549ec0cc7a52330d77bcdea682\": container with ID starting with 62ec2fc76e74ba2ff1f69eb722119a66757aed549ec0cc7a52330d77bcdea682 not found: ID does not exist" containerID="62ec2fc76e74ba2ff1f69eb722119a66757aed549ec0cc7a52330d77bcdea682"
Sep 08 13:54:15.419311 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:54:15.419265    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62ec2fc76e74ba2ff1f69eb722119a66757aed549ec0cc7a52330d77bcdea682"} err="failed to get container status \"62ec2fc76e74ba2ff1f69eb722119a66757aed549ec0cc7a52330d77bcdea682\": rpc error: code = NotFound desc = could not find container \"62ec2fc76e74ba2ff1f69eb722119a66757aed549ec0cc7a52330d77bcdea682\": container with ID starting with 62ec2fc76e74ba2ff1f69eb722119a66757aed549ec0cc7a52330d77bcdea682 not found: ID does not exist"
Sep 08 13:54:15.425988 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:54:15.425963    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57ff94957b-846r5"]
Sep 08 13:54:15.429839 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:54:15.429819    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57ff94957b-846r5"]
Sep 08 13:54:16.174660 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:54:16.174626    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a435301f-52e5-4ad0-be11-fe8d070676c5" path="/var/lib/kubelet/pods/a435301f-52e5-4ad0-be11-fe8d070676c5/volumes"
Sep 08 13:54:40.243727 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:54:40.243692    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:54:42.513543 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:54:42.513493    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:54:45.325516 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:54:45.325480    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:54:45.509037 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:54:45.509003    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:55:04.482051 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:55:04.482017    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:55:04.739358 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:55:04.739278    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:55:04.843739 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:55:04.843709    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:55:11.577361 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:55:11.577310    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:55:16.924489 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:55:16.924453    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:55:17.434715 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:55:17.434677    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:55:18.500522 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:55:18.500475    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:55:18.598915 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:55:18.598884    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:55:22.863478 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:55:22.863450    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:55:29.541550 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:55:29.541495    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:55:35.641633 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:55:35.641600    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:55:48.503679 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:55:48.503649    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-late-join-483adbac/late-v2-kserve-86766546f-698wv"]
Sep 08 13:55:48.504078 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:55:48.503948    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a435301f-52e5-4ad0-be11-fe8d070676c5" containerName="storage-initializer"
Sep 08 13:55:48.504078 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:55:48.503966    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="a435301f-52e5-4ad0-be11-fe8d070676c5" containerName="storage-initializer"
Sep 08 13:55:48.504078 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:55:48.503983    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a435301f-52e5-4ad0-be11-fe8d070676c5" containerName="main"
Sep 08 13:55:48.504078 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:55:48.504014    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="a435301f-52e5-4ad0-be11-fe8d070676c5" containerName="main"
Sep 08 13:55:48.504078 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:55:48.504077    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="a435301f-52e5-4ad0-be11-fe8d070676c5" containerName="main"
Sep 08 13:55:48.506905 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:55:48.506888    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-698wv"
Sep 08 13:55:48.511025 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:55:48.511000    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-late-join-483adbac\"/\"default-dockercfg-7mq9x\""
Sep 08 13:55:48.511025 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:55:48.511018    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-late-join-483adbac\"/\"openshift-service-ca.crt\""
Sep 08 13:55:48.511248 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:55:48.511033    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-late-join-483adbac\"/\"late-v2-kserve-self-signed-certs\""
Sep 08 13:55:48.511248 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:55:48.511110    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-late-join-483adbac\"/\"kube-root-ca.crt\""
Sep 08 13:55:48.519288 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:55:48.519265    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-late-join-483adbac/late-v2-kserve-86766546f-698wv"]
Sep 08 13:55:48.534561 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:55:48.534541    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:55:48.585053 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:55:48.585024    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21ea1778-9280-4f27-b4d2-5316a75545a8-model-cache\") pod \"late-v2-kserve-86766546f-698wv\" (UID: \"21ea1778-9280-4f27-b4d2-5316a75545a8\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-698wv"
Sep 08 13:55:48.585182 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:55:48.585074    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21ea1778-9280-4f27-b4d2-5316a75545a8-dshm\") pod \"late-v2-kserve-86766546f-698wv\" (UID: \"21ea1778-9280-4f27-b4d2-5316a75545a8\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-698wv"
Sep 08 13:55:48.585182 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:55:48.585162    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21ea1778-9280-4f27-b4d2-5316a75545a8-tls-certs\") pod \"late-v2-kserve-86766546f-698wv\" (UID: \"21ea1778-9280-4f27-b4d2-5316a75545a8\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-698wv"
Sep 08 13:55:48.585277 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:55:48.585196    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21ea1778-9280-4f27-b4d2-5316a75545a8-tmp-dir\") pod \"late-v2-kserve-86766546f-698wv\" (UID: \"21ea1778-9280-4f27-b4d2-5316a75545a8\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-698wv"
Sep 08 13:55:48.585277 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:55:48.585228    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjjmp\" (UniqueName: \"kubernetes.io/projected/21ea1778-9280-4f27-b4d2-5316a75545a8-kube-api-access-gjjmp\") pod \"late-v2-kserve-86766546f-698wv\" (UID: \"21ea1778-9280-4f27-b4d2-5316a75545a8\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-698wv"
Sep 08 13:55:48.585277 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:55:48.585264    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21ea1778-9280-4f27-b4d2-5316a75545a8-home\") pod \"late-v2-kserve-86766546f-698wv\" (UID: \"21ea1778-9280-4f27-b4d2-5316a75545a8\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-698wv"
Sep 08 13:55:48.686164 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:55:48.686125    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21ea1778-9280-4f27-b4d2-5316a75545a8-dshm\") pod \"late-v2-kserve-86766546f-698wv\" (UID: \"21ea1778-9280-4f27-b4d2-5316a75545a8\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-698wv"
Sep 08 13:55:48.686295 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:55:48.686209    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21ea1778-9280-4f27-b4d2-5316a75545a8-tls-certs\") pod \"late-v2-kserve-86766546f-698wv\" (UID: \"21ea1778-9280-4f27-b4d2-5316a75545a8\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-698wv"
Sep 08 13:55:48.686295 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:55:48.686236    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21ea1778-9280-4f27-b4d2-5316a75545a8-tmp-dir\") pod \"late-v2-kserve-86766546f-698wv\" (UID: \"21ea1778-9280-4f27-b4d2-5316a75545a8\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-698wv"
Sep 08 13:55:48.686295 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:55:48.686259    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjjmp\" (UniqueName: \"kubernetes.io/projected/21ea1778-9280-4f27-b4d2-5316a75545a8-kube-api-access-gjjmp\") pod \"late-v2-kserve-86766546f-698wv\" (UID: \"21ea1778-9280-4f27-b4d2-5316a75545a8\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-698wv"
Sep 08 13:55:48.686460 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:55:48.686394    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21ea1778-9280-4f27-b4d2-5316a75545a8-home\") pod \"late-v2-kserve-86766546f-698wv\" (UID: \"21ea1778-9280-4f27-b4d2-5316a75545a8\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-698wv"
Sep 08 13:55:48.686460 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:55:48.686428    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21ea1778-9280-4f27-b4d2-5316a75545a8-model-cache\") pod \"late-v2-kserve-86766546f-698wv\" (UID: \"21ea1778-9280-4f27-b4d2-5316a75545a8\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-698wv"
Sep 08 13:55:48.686668 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:55:48.686647    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21ea1778-9280-4f27-b4d2-5316a75545a8-tmp-dir\") pod \"late-v2-kserve-86766546f-698wv\" (UID: \"21ea1778-9280-4f27-b4d2-5316a75545a8\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-698wv"
Sep 08 13:55:48.686750 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:55:48.686732    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21ea1778-9280-4f27-b4d2-5316a75545a8-home\") pod \"late-v2-kserve-86766546f-698wv\" (UID: \"21ea1778-9280-4f27-b4d2-5316a75545a8\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-698wv"
Sep 08 13:55:48.686796 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:55:48.686759    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21ea1778-9280-4f27-b4d2-5316a75545a8-model-cache\") pod \"late-v2-kserve-86766546f-698wv\" (UID: \"21ea1778-9280-4f27-b4d2-5316a75545a8\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-698wv"
Sep 08 13:55:48.688790 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:55:48.688769    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21ea1778-9280-4f27-b4d2-5316a75545a8-dshm\") pod \"late-v2-kserve-86766546f-698wv\" (UID: \"21ea1778-9280-4f27-b4d2-5316a75545a8\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-698wv"
Sep 08 13:55:48.688912 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:55:48.688893    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21ea1778-9280-4f27-b4d2-5316a75545a8-tls-certs\") pod \"late-v2-kserve-86766546f-698wv\" (UID: \"21ea1778-9280-4f27-b4d2-5316a75545a8\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-698wv"
Sep 08 13:55:48.696305 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:55:48.696287    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjjmp\" (UniqueName: \"kubernetes.io/projected/21ea1778-9280-4f27-b4d2-5316a75545a8-kube-api-access-gjjmp\") pod \"late-v2-kserve-86766546f-698wv\" (UID: \"21ea1778-9280-4f27-b4d2-5316a75545a8\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-698wv"
Sep 08 13:55:48.816899 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:55:48.816838    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-698wv"
Sep 08 13:55:49.089241 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:55:49.089165    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-late-join-483adbac/late-v2-kserve-86766546f-698wv"]
Sep 08 13:55:49.092098 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:55:49.092073    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21ea1778_9280_4f27_b4d2_5316a75545a8.slice/crio-7929de027cad354b6e31910c8dedc57eac159ca0eaee085f7d3e6ec2ba8c075e WatchSource:0}: Error finding container 7929de027cad354b6e31910c8dedc57eac159ca0eaee085f7d3e6ec2ba8c075e: Status 404 returned error can't find the container with id 7929de027cad354b6e31910c8dedc57eac159ca0eaee085f7d3e6ec2ba8c075e
Sep 08 13:55:49.752841 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:55:49.752807    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-698wv" event={"ID":"21ea1778-9280-4f27-b4d2-5316a75545a8","Type":"ContainerStarted","Data":"2dfdfbf5a957e01050ebdd10b8f2b289a9e90678da2bd844adc457a5cf419a38"}
Sep 08 13:55:49.753289 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:55:49.752856    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-698wv" event={"ID":"21ea1778-9280-4f27-b4d2-5316a75545a8","Type":"ContainerStarted","Data":"7929de027cad354b6e31910c8dedc57eac159ca0eaee085f7d3e6ec2ba8c075e"}
Sep 08 13:55:49.774788 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:55:49.774744    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-698wv" podStartSLOduration=1.77473017 podStartE2EDuration="1.77473017s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:55:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:55:49.773093619 +0000 UTC m=+2400.044541550" watchObservedRunningTime="2026-09-08 13:55:49.77473017 +0000 UTC m=+2400.046178101"
Sep 08 13:55:49.817910 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:55:49.817883    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-698wv"
Sep 08 13:55:50.301489 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:55:50.301458    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8h8cn_b936238e-bdf1-4dac-8bf8-6417c313597c/ovn-acl-logging/0.log"
Sep 08 13:55:50.307913 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:55:50.307891    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8h8cn_b936238e-bdf1-4dac-8bf8-6417c313597c/ovn-acl-logging/0.log"
Sep 08 13:55:58.817251 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:55:58.817212    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-698wv"
Sep 08 13:55:58.829250 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:55:58.829230    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-698wv"
Sep 08 13:55:59.800863 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:55:59.800833    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-698wv"
Sep 08 13:56:04.187144 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:56:04.187110    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:56:04.250680 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:56:04.250647    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:56:31.462371 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:56:31.462323    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:56:31.697945 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:56:31.697916    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:56:31.729032 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:56:31.728960    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-late-join-483adbac/late-v2-kserve-86766546f-698wv"]
Sep 08 13:56:31.957109 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:56:31.957079    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:56:32.110827 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:56:32.110745    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:56:36.452382 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:56:36.452353    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:56:45.390104 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:56:45.390075    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:56:46.748812 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:56:46.748746    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-698wv" podUID="21ea1778-9280-4f27-b4d2-5316a75545a8" containerName="main" containerID="cri-o://2dfdfbf5a957e01050ebdd10b8f2b289a9e90678da2bd844adc457a5cf419a38" gracePeriod=45
Sep 08 13:56:46.791079 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:56:46.791043    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-698wv" podUID="21ea1778-9280-4f27-b4d2-5316a75545a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/health\": dial tcp 10.133.0.37:8000: connect: connection refused"
Sep 08 13:56:46.975869 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:56:46.975842    2573 generic.go:356] "Generic (PLEG): container finished" podID="21ea1778-9280-4f27-b4d2-5316a75545a8" containerID="2dfdfbf5a957e01050ebdd10b8f2b289a9e90678da2bd844adc457a5cf419a38" exitCode=0
Sep 08 13:56:46.975953 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:56:46.975913    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-698wv" event={"ID":"21ea1778-9280-4f27-b4d2-5316a75545a8","Type":"ContainerDied","Data":"2dfdfbf5a957e01050ebdd10b8f2b289a9e90678da2bd844adc457a5cf419a38"}
Sep 08 13:56:46.976011 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:56:46.975966    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-698wv" event={"ID":"21ea1778-9280-4f27-b4d2-5316a75545a8","Type":"ContainerDied","Data":"7929de027cad354b6e31910c8dedc57eac159ca0eaee085f7d3e6ec2ba8c075e"}
Sep 08 13:56:46.976011 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:56:46.975994    2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7929de027cad354b6e31910c8dedc57eac159ca0eaee085f7d3e6ec2ba8c075e"
Sep 08 13:56:46.988919 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:56:46.988895    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-698wv"
Sep 08 13:56:47.020075 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:56:47.020013    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21ea1778-9280-4f27-b4d2-5316a75545a8-tmp-dir\") pod \"21ea1778-9280-4f27-b4d2-5316a75545a8\" (UID: \"21ea1778-9280-4f27-b4d2-5316a75545a8\") "
Sep 08 13:56:47.020075 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:56:47.020056    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21ea1778-9280-4f27-b4d2-5316a75545a8-home\") pod \"21ea1778-9280-4f27-b4d2-5316a75545a8\" (UID: \"21ea1778-9280-4f27-b4d2-5316a75545a8\") "
Sep 08 13:56:47.020240 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:56:47.020090    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21ea1778-9280-4f27-b4d2-5316a75545a8-model-cache\") pod \"21ea1778-9280-4f27-b4d2-5316a75545a8\" (UID: \"21ea1778-9280-4f27-b4d2-5316a75545a8\") "
Sep 08 13:56:47.020240 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:56:47.020128    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gjjmp\" (UniqueName: \"kubernetes.io/projected/21ea1778-9280-4f27-b4d2-5316a75545a8-kube-api-access-gjjmp\") pod \"21ea1778-9280-4f27-b4d2-5316a75545a8\" (UID: \"21ea1778-9280-4f27-b4d2-5316a75545a8\") "
Sep 08 13:56:47.020240 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:56:47.020158    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21ea1778-9280-4f27-b4d2-5316a75545a8-tls-certs\") pod \"21ea1778-9280-4f27-b4d2-5316a75545a8\" (UID: \"21ea1778-9280-4f27-b4d2-5316a75545a8\") "
Sep 08 13:56:47.020240 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:56:47.020220    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21ea1778-9280-4f27-b4d2-5316a75545a8-dshm\") pod \"21ea1778-9280-4f27-b4d2-5316a75545a8\" (UID: \"21ea1778-9280-4f27-b4d2-5316a75545a8\") "
Sep 08 13:56:47.020240 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:56:47.020221    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21ea1778-9280-4f27-b4d2-5316a75545a8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "21ea1778-9280-4f27-b4d2-5316a75545a8" (UID: "21ea1778-9280-4f27-b4d2-5316a75545a8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:56:47.020488 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:56:47.020372    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21ea1778-9280-4f27-b4d2-5316a75545a8-model-cache" (OuterVolumeSpecName: "model-cache") pod "21ea1778-9280-4f27-b4d2-5316a75545a8" (UID: "21ea1778-9280-4f27-b4d2-5316a75545a8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:56:47.020488 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:56:47.020380    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21ea1778-9280-4f27-b4d2-5316a75545a8-home" (OuterVolumeSpecName: "home") pod "21ea1778-9280-4f27-b4d2-5316a75545a8" (UID: "21ea1778-9280-4f27-b4d2-5316a75545a8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:56:47.020488 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:56:47.020439    2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21ea1778-9280-4f27-b4d2-5316a75545a8-tmp-dir\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 13:56:47.020488 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:56:47.020456    2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21ea1778-9280-4f27-b4d2-5316a75545a8-home\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 13:56:47.020488 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:56:47.020470    2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21ea1778-9280-4f27-b4d2-5316a75545a8-model-cache\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 13:56:47.022356 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:56:47.022328    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21ea1778-9280-4f27-b4d2-5316a75545a8-dshm" (OuterVolumeSpecName: "dshm") pod "21ea1778-9280-4f27-b4d2-5316a75545a8" (UID: "21ea1778-9280-4f27-b4d2-5316a75545a8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:56:47.022453 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:56:47.022371    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21ea1778-9280-4f27-b4d2-5316a75545a8-kube-api-access-gjjmp" (OuterVolumeSpecName: "kube-api-access-gjjmp") pod "21ea1778-9280-4f27-b4d2-5316a75545a8" (UID: "21ea1778-9280-4f27-b4d2-5316a75545a8"). InnerVolumeSpecName "kube-api-access-gjjmp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:56:47.022521 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:56:47.022451    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21ea1778-9280-4f27-b4d2-5316a75545a8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "21ea1778-9280-4f27-b4d2-5316a75545a8" (UID: "21ea1778-9280-4f27-b4d2-5316a75545a8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:56:47.121396 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:56:47.121367    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjjmp\" (UniqueName: \"kubernetes.io/projected/21ea1778-9280-4f27-b4d2-5316a75545a8-kube-api-access-gjjmp\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 13:56:47.121396 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:56:47.121393    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21ea1778-9280-4f27-b4d2-5316a75545a8-tls-certs\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 13:56:47.121396 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:56:47.121403    2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21ea1778-9280-4f27-b4d2-5316a75545a8-dshm\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 13:56:47.980056 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:56:47.980028    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-698wv"
Sep 08 13:56:48.009090 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:56:48.009065    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-late-join-483adbac/late-v2-kserve-86766546f-698wv"]
Sep 08 13:56:48.013165 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:56:48.013142    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-late-join-483adbac/late-v2-kserve-86766546f-698wv"]
Sep 08 13:56:48.174616 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:56:48.174579    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="21ea1778-9280-4f27-b4d2-5316a75545a8" path="/var/lib/kubelet/pods/21ea1778-9280-4f27-b4d2-5316a75545a8/volumes"
Sep 08 13:56:51.742983 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:56:51.742947    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:56:51.836152 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:56:51.836122    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:56:52.972233 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:56:52.972200    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:56:58.190738 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:56:58.190676    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:57:04.591935 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:57:04.591897    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:57:04.675521 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:57:04.675469    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:57:05.626410 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:57:05.626377    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-k4vst"]
Sep 08 13:57:05.626792 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:57:05.626707    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21ea1778-9280-4f27-b4d2-5316a75545a8" containerName="main"
Sep 08 13:57:05.626792 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:57:05.626720    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="21ea1778-9280-4f27-b4d2-5316a75545a8" containerName="main"
Sep 08 13:57:05.626792 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:57:05.626777    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="21ea1778-9280-4f27-b4d2-5316a75545a8" containerName="main"
Sep 08 13:57:05.629562 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:57:05.629546    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-k4vst"
Sep 08 13:57:05.632020 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:57:05.632001    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"default-dockercfg-l7glm\""
Sep 08 13:57:05.632748 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:57:05.632732    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"kube-root-ca.crt\""
Sep 08 13:57:05.632913 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:57:05.632898    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"openshift-service-ca.crt\""
Sep 08 13:57:05.632986 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:57:05.632968    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs\""
Sep 08 13:57:05.645684 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:57:05.645665    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-k4vst"]
Sep 08 13:57:05.655716 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:57:05.655693    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:57:05.764729 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:57:05.764701    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76f65727-300d-4713-b94b-f44c20118d62-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-k4vst\" (UID: \"76f65727-300d-4713-b94b-f44c20118d62\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-k4vst"
Sep 08 13:57:05.764828 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:57:05.764741    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76f65727-300d-4713-b94b-f44c20118d62-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-k4vst\" (UID: \"76f65727-300d-4713-b94b-f44c20118d62\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-k4vst"
Sep 08 13:57:05.764828 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:57:05.764769    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76f65727-300d-4713-b94b-f44c20118d62-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-k4vst\" (UID: \"76f65727-300d-4713-b94b-f44c20118d62\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-k4vst"
Sep 08 13:57:05.764906 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:57:05.764833    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fklrs\" (UniqueName: \"kubernetes.io/projected/76f65727-300d-4713-b94b-f44c20118d62-kube-api-access-fklrs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-k4vst\" (UID: \"76f65727-300d-4713-b94b-f44c20118d62\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-k4vst"
Sep 08 13:57:05.764906 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:57:05.764868    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76f65727-300d-4713-b94b-f44c20118d62-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-k4vst\" (UID: \"76f65727-300d-4713-b94b-f44c20118d62\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-k4vst"
Sep 08 13:57:05.764906 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:57:05.764899    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76f65727-300d-4713-b94b-f44c20118d62-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-k4vst\" (UID: \"76f65727-300d-4713-b94b-f44c20118d62\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-k4vst"
Sep 08 13:57:05.866078 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:57:05.866040    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76f65727-300d-4713-b94b-f44c20118d62-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-k4vst\" (UID: \"76f65727-300d-4713-b94b-f44c20118d62\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-k4vst"
Sep 08 13:57:05.866204 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:57:05.866096    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76f65727-300d-4713-b94b-f44c20118d62-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-k4vst\" (UID: \"76f65727-300d-4713-b94b-f44c20118d62\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-k4vst"
Sep 08 13:57:05.866204 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:57:05.866121    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76f65727-300d-4713-b94b-f44c20118d62-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-k4vst\" (UID: \"76f65727-300d-4713-b94b-f44c20118d62\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-k4vst"
Sep 08 13:57:05.866204 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:57:05.866140    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76f65727-300d-4713-b94b-f44c20118d62-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-k4vst\" (UID: \"76f65727-300d-4713-b94b-f44c20118d62\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-k4vst"
Sep 08 13:57:05.866204 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:57:05.866165    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fklrs\" (UniqueName: \"kubernetes.io/projected/76f65727-300d-4713-b94b-f44c20118d62-kube-api-access-fklrs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-k4vst\" (UID: \"76f65727-300d-4713-b94b-f44c20118d62\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-k4vst"
Sep 08 13:57:05.866204 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:57:05.866193    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76f65727-300d-4713-b94b-f44c20118d62-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-k4vst\" (UID: \"76f65727-300d-4713-b94b-f44c20118d62\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-k4vst"
Sep 08 13:57:05.866556 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:57:05.866492    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76f65727-300d-4713-b94b-f44c20118d62-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-k4vst\" (UID: \"76f65727-300d-4713-b94b-f44c20118d62\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-k4vst"
Sep 08 13:57:05.866556 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:57:05.866544    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76f65727-300d-4713-b94b-f44c20118d62-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-k4vst\" (UID: \"76f65727-300d-4713-b94b-f44c20118d62\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-k4vst"
Sep 08 13:57:05.866723 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:57:05.866585    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76f65727-300d-4713-b94b-f44c20118d62-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-k4vst\" (UID: \"76f65727-300d-4713-b94b-f44c20118d62\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-k4vst"
Sep 08 13:57:05.868460 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:57:05.868440    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76f65727-300d-4713-b94b-f44c20118d62-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-k4vst\" (UID: \"76f65727-300d-4713-b94b-f44c20118d62\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-k4vst"
Sep 08 13:57:05.868660 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:57:05.868643    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76f65727-300d-4713-b94b-f44c20118d62-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-k4vst\" (UID: \"76f65727-300d-4713-b94b-f44c20118d62\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-k4vst"
Sep 08 13:57:05.887808 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:57:05.887750    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fklrs\" (UniqueName: \"kubernetes.io/projected/76f65727-300d-4713-b94b-f44c20118d62-kube-api-access-fklrs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-k4vst\" (UID: \"76f65727-300d-4713-b94b-f44c20118d62\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-k4vst"
Sep 08 13:57:05.939041 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:57:05.939016    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-k4vst"
Sep 08 13:57:05.946666 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:57:05.946638    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:57:06.210459 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:57:06.210386    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-k4vst"]
Sep 08 13:57:06.213208 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:57:06.213180    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76f65727_300d_4713_b94b_f44c20118d62.slice/crio-a6831bd2497e02e58f36f404780af90fd6b027ed210ed6490b2675c972b0b3ba WatchSource:0}: Error finding container a6831bd2497e02e58f36f404780af90fd6b027ed210ed6490b2675c972b0b3ba: Status 404 returned error can't find the container with id a6831bd2497e02e58f36f404780af90fd6b027ed210ed6490b2675c972b0b3ba
Sep 08 13:57:07.056258 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:57:07.056225    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-k4vst" event={"ID":"76f65727-300d-4713-b94b-f44c20118d62","Type":"ContainerStarted","Data":"0a5c3c38986ad19385c1c5a825f219770900aa98dadcff2624c5fb3d3df6fede"}
Sep 08 13:57:07.056654 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:57:07.056275    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-k4vst" event={"ID":"76f65727-300d-4713-b94b-f44c20118d62","Type":"ContainerStarted","Data":"a6831bd2497e02e58f36f404780af90fd6b027ed210ed6490b2675c972b0b3ba"}
Sep 08 13:57:07.085765 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:57:07.085724    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-k4vst" podStartSLOduration=2.085709414 podStartE2EDuration="2.085709414s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:57:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:57:07.082212542 +0000 UTC m=+2477.353660474" watchObservedRunningTime="2026-09-08 13:57:07.085709414 +0000 UTC m=+2477.357157345"
Sep 08 13:57:07.939140 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:57:07.939098    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-k4vst"
Sep 08 13:57:15.069809 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:57:15.069747    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:57:15.939626 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:57:15.939583    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-k4vst"
Sep 08 13:57:15.951868 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:57:15.951840    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-k4vst"
Sep 08 13:57:16.103150 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:57:16.103121    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-k4vst"
Sep 08 13:57:21.966146 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:57:21.966109    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:57:22.100590 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:57:22.100544    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:57:28.429497 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:57:28.429466    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:57:33.913371 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:57:33.913336    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:57:44.113536 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:57:44.113397    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:57:44.393581 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:57:44.393553    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:57:48.520570 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:57:48.520533    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:58:05.444743 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:58:05.444706    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:58:05.556534 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:58:05.556484    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:58:05.662165 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:58:05.662124    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:58:10.469470 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:58:10.469433    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:58:22.439633 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:58:22.439600    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:58:23.933657 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:58:23.933600    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:58:26.609451 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:58:26.609396    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:58:35.660851 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:58:35.660818    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:58:40.539622 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:58:40.539567    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:58:42.275748 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:58:42.275702    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:58:42.376847 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:58:42.376798    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:58:43.118182 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:58:43.118152    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:58:43.217829 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:58:43.217789    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:59:03.599750 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:59:03.599715    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-k8kqp"]
Sep 08 13:59:03.602881 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:59:03.602865    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-k8kqp"
Sep 08 13:59:03.605900 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:59:03.605877    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rollback-b8fdb441\"/\"openshift-service-ca.crt\""
Sep 08 13:59:03.606046 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:59:03.605879    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rollback-b8fdb441\"/\"default-dockercfg-g727d\""
Sep 08 13:59:03.606046 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:59:03.605879    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rollback-b8fdb441\"/\"roll-v2-kserve-self-signed-certs\""
Sep 08 13:59:03.606046 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:59:03.605926    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rollback-b8fdb441\"/\"kube-root-ca.crt\""
Sep 08 13:59:03.617847 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:59:03.617826    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-k8kqp"]
Sep 08 13:59:03.768190 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:59:03.768159    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgr7j\" (UniqueName: \"kubernetes.io/projected/e4c6a19b-868e-4418-8c70-67dd6d0a491e-kube-api-access-pgr7j\") pod \"roll-v2-kserve-d4dbf9d67-k8kqp\" (UID: \"e4c6a19b-868e-4418-8c70-67dd6d0a491e\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-k8kqp"
Sep 08 13:59:03.768190 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:59:03.768192    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4c6a19b-868e-4418-8c70-67dd6d0a491e-home\") pod \"roll-v2-kserve-d4dbf9d67-k8kqp\" (UID: \"e4c6a19b-868e-4418-8c70-67dd6d0a491e\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-k8kqp"
Sep 08 13:59:03.768431 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:59:03.768223    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4c6a19b-868e-4418-8c70-67dd6d0a491e-tls-certs\") pod \"roll-v2-kserve-d4dbf9d67-k8kqp\" (UID: \"e4c6a19b-868e-4418-8c70-67dd6d0a491e\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-k8kqp"
Sep 08 13:59:03.768431 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:59:03.768304    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4c6a19b-868e-4418-8c70-67dd6d0a491e-model-cache\") pod \"roll-v2-kserve-d4dbf9d67-k8kqp\" (UID: \"e4c6a19b-868e-4418-8c70-67dd6d0a491e\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-k8kqp"
Sep 08 13:59:03.768431 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:59:03.768342    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4c6a19b-868e-4418-8c70-67dd6d0a491e-dshm\") pod \"roll-v2-kserve-d4dbf9d67-k8kqp\" (UID: \"e4c6a19b-868e-4418-8c70-67dd6d0a491e\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-k8kqp"
Sep 08 13:59:03.768431 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:59:03.768363    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4c6a19b-868e-4418-8c70-67dd6d0a491e-tmp-dir\") pod \"roll-v2-kserve-d4dbf9d67-k8kqp\" (UID: \"e4c6a19b-868e-4418-8c70-67dd6d0a491e\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-k8kqp"
Sep 08 13:59:03.868838 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:59:03.868770    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4c6a19b-868e-4418-8c70-67dd6d0a491e-tls-certs\") pod \"roll-v2-kserve-d4dbf9d67-k8kqp\" (UID: \"e4c6a19b-868e-4418-8c70-67dd6d0a491e\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-k8kqp"
Sep 08 13:59:03.868838 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:59:03.868803    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4c6a19b-868e-4418-8c70-67dd6d0a491e-model-cache\") pod \"roll-v2-kserve-d4dbf9d67-k8kqp\" (UID: \"e4c6a19b-868e-4418-8c70-67dd6d0a491e\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-k8kqp"
Sep 08 13:59:03.868838 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:59:03.868823    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4c6a19b-868e-4418-8c70-67dd6d0a491e-dshm\") pod \"roll-v2-kserve-d4dbf9d67-k8kqp\" (UID: \"e4c6a19b-868e-4418-8c70-67dd6d0a491e\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-k8kqp"
Sep 08 13:59:03.868838 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:59:03.868839    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4c6a19b-868e-4418-8c70-67dd6d0a491e-tmp-dir\") pod \"roll-v2-kserve-d4dbf9d67-k8kqp\" (UID: \"e4c6a19b-868e-4418-8c70-67dd6d0a491e\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-k8kqp"
Sep 08 13:59:03.869171 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:59:03.868888    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgr7j\" (UniqueName: \"kubernetes.io/projected/e4c6a19b-868e-4418-8c70-67dd6d0a491e-kube-api-access-pgr7j\") pod \"roll-v2-kserve-d4dbf9d67-k8kqp\" (UID: \"e4c6a19b-868e-4418-8c70-67dd6d0a491e\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-k8kqp"
Sep 08 13:59:03.869171 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:59:03.868918    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4c6a19b-868e-4418-8c70-67dd6d0a491e-home\") pod \"roll-v2-kserve-d4dbf9d67-k8kqp\" (UID: \"e4c6a19b-868e-4418-8c70-67dd6d0a491e\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-k8kqp"
Sep 08 13:59:03.869292 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:59:03.869226    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4c6a19b-868e-4418-8c70-67dd6d0a491e-model-cache\") pod \"roll-v2-kserve-d4dbf9d67-k8kqp\" (UID: \"e4c6a19b-868e-4418-8c70-67dd6d0a491e\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-k8kqp"
Sep 08 13:59:03.869351 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:59:03.869294    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4c6a19b-868e-4418-8c70-67dd6d0a491e-tmp-dir\") pod \"roll-v2-kserve-d4dbf9d67-k8kqp\" (UID: \"e4c6a19b-868e-4418-8c70-67dd6d0a491e\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-k8kqp"
Sep 08 13:59:03.869351 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:59:03.869320    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4c6a19b-868e-4418-8c70-67dd6d0a491e-home\") pod \"roll-v2-kserve-d4dbf9d67-k8kqp\" (UID: \"e4c6a19b-868e-4418-8c70-67dd6d0a491e\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-k8kqp"
Sep 08 13:59:03.871229 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:59:03.871207    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4c6a19b-868e-4418-8c70-67dd6d0a491e-dshm\") pod \"roll-v2-kserve-d4dbf9d67-k8kqp\" (UID: \"e4c6a19b-868e-4418-8c70-67dd6d0a491e\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-k8kqp"
Sep 08 13:59:03.871324 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:59:03.871305    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4c6a19b-868e-4418-8c70-67dd6d0a491e-tls-certs\") pod \"roll-v2-kserve-d4dbf9d67-k8kqp\" (UID: \"e4c6a19b-868e-4418-8c70-67dd6d0a491e\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-k8kqp"
Sep 08 13:59:03.877994 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:59:03.877970    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgr7j\" (UniqueName: \"kubernetes.io/projected/e4c6a19b-868e-4418-8c70-67dd6d0a491e-kube-api-access-pgr7j\") pod \"roll-v2-kserve-d4dbf9d67-k8kqp\" (UID: \"e4c6a19b-868e-4418-8c70-67dd6d0a491e\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-k8kqp"
Sep 08 13:59:03.912370 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:59:03.912347    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-k8kqp"
Sep 08 13:59:04.184279 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:59:04.184248    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-k8kqp"]
Sep 08 13:59:04.188418 ip-10-0-132-254 kubenswrapper[2573]: W0908 13:59:04.188390    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4c6a19b_868e_4418_8c70_67dd6d0a491e.slice/crio-2ee90a1333f26e70857f55cfb2a04ba40cc48f09cb4dde06becda36530c00582 WatchSource:0}: Error finding container 2ee90a1333f26e70857f55cfb2a04ba40cc48f09cb4dde06becda36530c00582: Status 404 returned error can't find the container with id 2ee90a1333f26e70857f55cfb2a04ba40cc48f09cb4dde06becda36530c00582
Sep 08 13:59:04.544620 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:59:04.544540    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-k8kqp" event={"ID":"e4c6a19b-868e-4418-8c70-67dd6d0a491e","Type":"ContainerStarted","Data":"e7a2775ccfa226d49667735c6306ed320a59a380a465105e45a2ced8dc40d624"}
Sep 08 13:59:04.544620 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:59:04.544595    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-k8kqp" event={"ID":"e4c6a19b-868e-4418-8c70-67dd6d0a491e","Type":"ContainerStarted","Data":"2ee90a1333f26e70857f55cfb2a04ba40cc48f09cb4dde06becda36530c00582"}
Sep 08 13:59:04.561951 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:59:04.561900    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-k8kqp" podStartSLOduration=1.5618846290000001 podStartE2EDuration="1.561884629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:59:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:59:04.560606929 +0000 UTC m=+2594.832054881" watchObservedRunningTime="2026-09-08 13:59:04.561884629 +0000 UTC m=+2594.833332559"
Sep 08 13:59:04.913378 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:59:04.913349    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-k8kqp"
Sep 08 13:59:05.631373 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:59:05.631339    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:59:05.731413 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:59:05.731385    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:59:13.913482 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:59:13.913401    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-k8kqp"
Sep 08 13:59:13.925689 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:59:13.925666    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-k8kqp"
Sep 08 13:59:14.595943 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:59:14.595911    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-k8kqp"
Sep 08 13:59:52.871790 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:59:52.871752    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-k4vst"]
Sep 08 13:59:52.880582 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:59:52.880545    2573 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 08 13:59:52.880582 ip-10-0-132-254 kubenswrapper[2573]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 08 13:59:52.880582 ip-10-0-132-254 kubenswrapper[2573]:         , stdout: , stderr: , exit code -1
Sep 08 13:59:52.880582 ip-10-0-132-254 kubenswrapper[2573]:  > containerID="0a5c3c38986ad19385c1c5a825f219770900aa98dadcff2624c5fb3d3df6fede" cmd=["/bin/sleep","15"]
Sep 08 13:59:52.880819 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:59:52.880607    2573 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-k4vst" message="" err=<
Sep 08 13:59:52.880819 ip-10-0-132-254 kubenswrapper[2573]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 08 13:59:52.880819 ip-10-0-132-254 kubenswrapper[2573]:         , stdout: , stderr: , exit code -1
Sep 08 13:59:52.880819 ip-10-0-132-254 kubenswrapper[2573]:  >
Sep 08 13:59:52.880819 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:59:52.880620    2573 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 08 13:59:52.880819 ip-10-0-132-254 kubenswrapper[2573]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 08 13:59:52.880819 ip-10-0-132-254 kubenswrapper[2573]:         , stdout: , stderr: , exit code -1
Sep 08 13:59:52.880819 ip-10-0-132-254 kubenswrapper[2573]:  > pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-k4vst" podUID="76f65727-300d-4713-b94b-f44c20118d62" containerName="main" containerID="cri-o://0a5c3c38986ad19385c1c5a825f219770900aa98dadcff2624c5fb3d3df6fede"
Sep 08 13:59:52.880819 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:59:52.880655    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-k4vst" podUID="76f65727-300d-4713-b94b-f44c20118d62" containerName="main" containerID="cri-o://0a5c3c38986ad19385c1c5a825f219770900aa98dadcff2624c5fb3d3df6fede" gracePeriod=60
Sep 08 13:59:52.892874 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:59:52.892841    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:59:53.092656 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:59:53.092627    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-k4vst" podUID="76f65727-300d-4713-b94b-f44c20118d62" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/health\": dial tcp 10.133.0.38:8000: connect: connection refused"
Sep 08 13:59:53.213040 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:59:53.213018    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-k4vst"
Sep 08 13:59:53.280293 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:59:53.280264    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76f65727-300d-4713-b94b-f44c20118d62-dshm\") pod \"76f65727-300d-4713-b94b-f44c20118d62\" (UID: \"76f65727-300d-4713-b94b-f44c20118d62\") "
Sep 08 13:59:53.280451 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:59:53.280323    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76f65727-300d-4713-b94b-f44c20118d62-tls-certs\") pod \"76f65727-300d-4713-b94b-f44c20118d62\" (UID: \"76f65727-300d-4713-b94b-f44c20118d62\") "
Sep 08 13:59:53.280451 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:59:53.280348    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76f65727-300d-4713-b94b-f44c20118d62-home\") pod \"76f65727-300d-4713-b94b-f44c20118d62\" (UID: \"76f65727-300d-4713-b94b-f44c20118d62\") "
Sep 08 13:59:53.280451 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:59:53.280380    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76f65727-300d-4713-b94b-f44c20118d62-model-cache\") pod \"76f65727-300d-4713-b94b-f44c20118d62\" (UID: \"76f65727-300d-4713-b94b-f44c20118d62\") "
Sep 08 13:59:53.280451 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:59:53.280419    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fklrs\" (UniqueName: \"kubernetes.io/projected/76f65727-300d-4713-b94b-f44c20118d62-kube-api-access-fklrs\") pod \"76f65727-300d-4713-b94b-f44c20118d62\" (UID: \"76f65727-300d-4713-b94b-f44c20118d62\") "
Sep 08 13:59:53.280711 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:59:53.280454    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76f65727-300d-4713-b94b-f44c20118d62-tmp-dir\") pod \"76f65727-300d-4713-b94b-f44c20118d62\" (UID: \"76f65727-300d-4713-b94b-f44c20118d62\") "
Sep 08 13:59:53.280768 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:59:53.280704    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76f65727-300d-4713-b94b-f44c20118d62-model-cache" (OuterVolumeSpecName: "model-cache") pod "76f65727-300d-4713-b94b-f44c20118d62" (UID: "76f65727-300d-4713-b94b-f44c20118d62"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:59:53.280768 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:59:53.280722    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76f65727-300d-4713-b94b-f44c20118d62-home" (OuterVolumeSpecName: "home") pod "76f65727-300d-4713-b94b-f44c20118d62" (UID: "76f65727-300d-4713-b94b-f44c20118d62"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:59:53.280768 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:59:53.280731    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76f65727-300d-4713-b94b-f44c20118d62-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "76f65727-300d-4713-b94b-f44c20118d62" (UID: "76f65727-300d-4713-b94b-f44c20118d62"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:59:53.280927 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:59:53.280804    2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76f65727-300d-4713-b94b-f44c20118d62-home\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 13:59:53.280927 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:59:53.280822    2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76f65727-300d-4713-b94b-f44c20118d62-model-cache\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 13:59:53.280927 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:59:53.280835    2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76f65727-300d-4713-b94b-f44c20118d62-tmp-dir\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 13:59:53.282724 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:59:53.282698    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76f65727-300d-4713-b94b-f44c20118d62-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "76f65727-300d-4713-b94b-f44c20118d62" (UID: "76f65727-300d-4713-b94b-f44c20118d62"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:59:53.282843 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:59:53.282829    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76f65727-300d-4713-b94b-f44c20118d62-dshm" (OuterVolumeSpecName: "dshm") pod "76f65727-300d-4713-b94b-f44c20118d62" (UID: "76f65727-300d-4713-b94b-f44c20118d62"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:59:53.282889 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:59:53.282855    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76f65727-300d-4713-b94b-f44c20118d62-kube-api-access-fklrs" (OuterVolumeSpecName: "kube-api-access-fklrs") pod "76f65727-300d-4713-b94b-f44c20118d62" (UID: "76f65727-300d-4713-b94b-f44c20118d62"). InnerVolumeSpecName "kube-api-access-fklrs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:59:53.381544 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:59:53.381517    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76f65727-300d-4713-b94b-f44c20118d62-tls-certs\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 13:59:53.381544 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:59:53.381543    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fklrs\" (UniqueName: \"kubernetes.io/projected/76f65727-300d-4713-b94b-f44c20118d62-kube-api-access-fklrs\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 13:59:53.381715 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:59:53.381553    2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76f65727-300d-4713-b94b-f44c20118d62-dshm\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 13:59:53.749539 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:59:53.749494    2573 generic.go:356] "Generic (PLEG): container finished" podID="76f65727-300d-4713-b94b-f44c20118d62" containerID="0a5c3c38986ad19385c1c5a825f219770900aa98dadcff2624c5fb3d3df6fede" exitCode=0
Sep 08 13:59:53.749742 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:59:53.749568    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-k4vst"
Sep 08 13:59:53.749742 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:59:53.749577    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-k4vst" event={"ID":"76f65727-300d-4713-b94b-f44c20118d62","Type":"ContainerDied","Data":"0a5c3c38986ad19385c1c5a825f219770900aa98dadcff2624c5fb3d3df6fede"}
Sep 08 13:59:53.749742 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:59:53.749657    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-k4vst" event={"ID":"76f65727-300d-4713-b94b-f44c20118d62","Type":"ContainerDied","Data":"a6831bd2497e02e58f36f404780af90fd6b027ed210ed6490b2675c972b0b3ba"}
Sep 08 13:59:53.749742 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:59:53.749681    2573 scope.go:117] "RemoveContainer" containerID="0a5c3c38986ad19385c1c5a825f219770900aa98dadcff2624c5fb3d3df6fede"
Sep 08 13:59:53.758159 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:59:53.758140    2573 scope.go:117] "RemoveContainer" containerID="0a5c3c38986ad19385c1c5a825f219770900aa98dadcff2624c5fb3d3df6fede"
Sep 08 13:59:53.758434 ip-10-0-132-254 kubenswrapper[2573]: E0908 13:59:53.758418    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a5c3c38986ad19385c1c5a825f219770900aa98dadcff2624c5fb3d3df6fede\": container with ID starting with 0a5c3c38986ad19385c1c5a825f219770900aa98dadcff2624c5fb3d3df6fede not found: ID does not exist" containerID="0a5c3c38986ad19385c1c5a825f219770900aa98dadcff2624c5fb3d3df6fede"
Sep 08 13:59:53.758487 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:59:53.758445    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a5c3c38986ad19385c1c5a825f219770900aa98dadcff2624c5fb3d3df6fede"} err="failed to get container status \"0a5c3c38986ad19385c1c5a825f219770900aa98dadcff2624c5fb3d3df6fede\": rpc error: code = NotFound desc = could not find container \"0a5c3c38986ad19385c1c5a825f219770900aa98dadcff2624c5fb3d3df6fede\": container with ID starting with 0a5c3c38986ad19385c1c5a825f219770900aa98dadcff2624c5fb3d3df6fede not found: ID does not exist"
Sep 08 13:59:53.773904 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:59:53.770548    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-k4vst"]
Sep 08 13:59:53.776610 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:59:53.776590    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-k4vst"]
Sep 08 13:59:54.114419 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:59:54.114342    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 13:59:54.174473 ip-10-0-132-254 kubenswrapper[2573]: I0908 13:59:54.174444    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="76f65727-300d-4713-b94b-f44c20118d62" path="/var/lib/kubelet/pods/76f65727-300d-4713-b94b-f44c20118d62/volumes"
Sep 08 14:00:16.835947 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:00:16.835903    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-k8kqp"]
Sep 08 14:00:18.090342 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:00:18.090308    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:00:18.217576 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:00:18.217543    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:00:21.904410 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:00:21.904368    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:00:23.555423 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:00:23.555387    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:00:28.238419 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:00:28.238381    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:00:31.855214 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:00:31.855145    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-k8kqp" podUID="e4c6a19b-868e-4418-8c70-67dd6d0a491e" containerName="main" containerID="cri-o://e7a2775ccfa226d49667735c6306ed320a59a380a465105e45a2ced8dc40d624" gracePeriod=45
Sep 08 14:00:32.094220 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:00:32.094192    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-k8kqp"
Sep 08 14:00:32.193930 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:00:32.193902    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4c6a19b-868e-4418-8c70-67dd6d0a491e-model-cache\") pod \"e4c6a19b-868e-4418-8c70-67dd6d0a491e\" (UID: \"e4c6a19b-868e-4418-8c70-67dd6d0a491e\") "
Sep 08 14:00:32.194098 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:00:32.193971    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4c6a19b-868e-4418-8c70-67dd6d0a491e-tls-certs\") pod \"e4c6a19b-868e-4418-8c70-67dd6d0a491e\" (UID: \"e4c6a19b-868e-4418-8c70-67dd6d0a491e\") "
Sep 08 14:00:32.194098 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:00:32.193988    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4c6a19b-868e-4418-8c70-67dd6d0a491e-tmp-dir\") pod \"e4c6a19b-868e-4418-8c70-67dd6d0a491e\" (UID: \"e4c6a19b-868e-4418-8c70-67dd6d0a491e\") "
Sep 08 14:00:32.194098 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:00:32.194026    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pgr7j\" (UniqueName: \"kubernetes.io/projected/e4c6a19b-868e-4418-8c70-67dd6d0a491e-kube-api-access-pgr7j\") pod \"e4c6a19b-868e-4418-8c70-67dd6d0a491e\" (UID: \"e4c6a19b-868e-4418-8c70-67dd6d0a491e\") "
Sep 08 14:00:32.194098 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:00:32.194044    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4c6a19b-868e-4418-8c70-67dd6d0a491e-home\") pod \"e4c6a19b-868e-4418-8c70-67dd6d0a491e\" (UID: \"e4c6a19b-868e-4418-8c70-67dd6d0a491e\") "
Sep 08 14:00:32.194098 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:00:32.194071    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4c6a19b-868e-4418-8c70-67dd6d0a491e-dshm\") pod \"e4c6a19b-868e-4418-8c70-67dd6d0a491e\" (UID: \"e4c6a19b-868e-4418-8c70-67dd6d0a491e\") "
Sep 08 14:00:32.194365 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:00:32.194190    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4c6a19b-868e-4418-8c70-67dd6d0a491e-model-cache" (OuterVolumeSpecName: "model-cache") pod "e4c6a19b-868e-4418-8c70-67dd6d0a491e" (UID: "e4c6a19b-868e-4418-8c70-67dd6d0a491e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:00:32.194365 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:00:32.194292    2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4c6a19b-868e-4418-8c70-67dd6d0a491e-model-cache\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 14:00:32.194365 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:00:32.194302    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4c6a19b-868e-4418-8c70-67dd6d0a491e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e4c6a19b-868e-4418-8c70-67dd6d0a491e" (UID: "e4c6a19b-868e-4418-8c70-67dd6d0a491e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:00:32.194365 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:00:32.194313    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4c6a19b-868e-4418-8c70-67dd6d0a491e-home" (OuterVolumeSpecName: "home") pod "e4c6a19b-868e-4418-8c70-67dd6d0a491e" (UID: "e4c6a19b-868e-4418-8c70-67dd6d0a491e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:00:32.196321 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:00:32.196294    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4c6a19b-868e-4418-8c70-67dd6d0a491e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e4c6a19b-868e-4418-8c70-67dd6d0a491e" (UID: "e4c6a19b-868e-4418-8c70-67dd6d0a491e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 14:00:32.196321 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:00:32.196310    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4c6a19b-868e-4418-8c70-67dd6d0a491e-dshm" (OuterVolumeSpecName: "dshm") pod "e4c6a19b-868e-4418-8c70-67dd6d0a491e" (UID: "e4c6a19b-868e-4418-8c70-67dd6d0a491e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:00:32.196455 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:00:32.196322    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4c6a19b-868e-4418-8c70-67dd6d0a491e-kube-api-access-pgr7j" (OuterVolumeSpecName: "kube-api-access-pgr7j") pod "e4c6a19b-868e-4418-8c70-67dd6d0a491e" (UID: "e4c6a19b-868e-4418-8c70-67dd6d0a491e"). InnerVolumeSpecName "kube-api-access-pgr7j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 14:00:32.294704 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:00:32.294671    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4c6a19b-868e-4418-8c70-67dd6d0a491e-tls-certs\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 14:00:32.294866 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:00:32.294715    2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4c6a19b-868e-4418-8c70-67dd6d0a491e-tmp-dir\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 14:00:32.294866 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:00:32.294726    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pgr7j\" (UniqueName: \"kubernetes.io/projected/e4c6a19b-868e-4418-8c70-67dd6d0a491e-kube-api-access-pgr7j\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 14:00:32.294866 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:00:32.294735    2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4c6a19b-868e-4418-8c70-67dd6d0a491e-home\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 14:00:32.294866 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:00:32.294744    2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4c6a19b-868e-4418-8c70-67dd6d0a491e-dshm\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 14:00:32.912837 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:00:32.912799    2573 generic.go:356] "Generic (PLEG): container finished" podID="e4c6a19b-868e-4418-8c70-67dd6d0a491e" containerID="e7a2775ccfa226d49667735c6306ed320a59a380a465105e45a2ced8dc40d624" exitCode=0
Sep 08 14:00:32.913351 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:00:32.912848    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-k8kqp"
Sep 08 14:00:32.913351 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:00:32.912889    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-k8kqp" event={"ID":"e4c6a19b-868e-4418-8c70-67dd6d0a491e","Type":"ContainerDied","Data":"e7a2775ccfa226d49667735c6306ed320a59a380a465105e45a2ced8dc40d624"}
Sep 08 14:00:32.913351 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:00:32.912949    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-k8kqp" event={"ID":"e4c6a19b-868e-4418-8c70-67dd6d0a491e","Type":"ContainerDied","Data":"2ee90a1333f26e70857f55cfb2a04ba40cc48f09cb4dde06becda36530c00582"}
Sep 08 14:00:32.913351 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:00:32.912983    2573 scope.go:117] "RemoveContainer" containerID="e7a2775ccfa226d49667735c6306ed320a59a380a465105e45a2ced8dc40d624"
Sep 08 14:00:32.921572 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:00:32.921555    2573 scope.go:117] "RemoveContainer" containerID="e7a2775ccfa226d49667735c6306ed320a59a380a465105e45a2ced8dc40d624"
Sep 08 14:00:32.921918 ip-10-0-132-254 kubenswrapper[2573]: E0908 14:00:32.921897    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7a2775ccfa226d49667735c6306ed320a59a380a465105e45a2ced8dc40d624\": container with ID starting with e7a2775ccfa226d49667735c6306ed320a59a380a465105e45a2ced8dc40d624 not found: ID does not exist" containerID="e7a2775ccfa226d49667735c6306ed320a59a380a465105e45a2ced8dc40d624"
Sep 08 14:00:32.921980 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:00:32.921930    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7a2775ccfa226d49667735c6306ed320a59a380a465105e45a2ced8dc40d624"} err="failed to get container status \"e7a2775ccfa226d49667735c6306ed320a59a380a465105e45a2ced8dc40d624\": rpc error: code = NotFound desc = could not find container \"e7a2775ccfa226d49667735c6306ed320a59a380a465105e45a2ced8dc40d624\": container with ID starting with e7a2775ccfa226d49667735c6306ed320a59a380a465105e45a2ced8dc40d624 not found: ID does not exist"
Sep 08 14:00:32.934736 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:00:32.934710    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-k8kqp"]
Sep 08 14:00:32.938769 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:00:32.938731    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-k8kqp"]
Sep 08 14:00:34.174215 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:00:34.174181    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e4c6a19b-868e-4418-8c70-67dd6d0a491e" path="/var/lib/kubelet/pods/e4c6a19b-868e-4418-8c70-67dd6d0a491e/volumes"
Sep 08 14:00:42.883352 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:00:42.883234    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:00:42.976875 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:00:42.976839    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:00:49.606834 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:00:49.606781    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:00:50.325354 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:00:50.325329    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8h8cn_b936238e-bdf1-4dac-8bf8-6417c313597c/ovn-acl-logging/0.log"
Sep 08 14:00:50.332322 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:00:50.332300    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8h8cn_b936238e-bdf1-4dac-8bf8-6417c313597c/ovn-acl-logging/0.log"
Sep 08 14:00:55.455864 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:00:55.455833    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:00:55.610483 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:00:55.610423    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:00:58.967831 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:00:58.967781    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:01:02.591712 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:01:02.591677    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vgsvl"]
Sep 08 14:01:02.592165 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:01:02.591969    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4c6a19b-868e-4418-8c70-67dd6d0a491e" containerName="main"
Sep 08 14:01:02.592165 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:01:02.591982    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4c6a19b-868e-4418-8c70-67dd6d0a491e" containerName="main"
Sep 08 14:01:02.592165 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:01:02.591998    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76f65727-300d-4713-b94b-f44c20118d62" containerName="main"
Sep 08 14:01:02.592165 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:01:02.592003    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="76f65727-300d-4713-b94b-f44c20118d62" containerName="main"
Sep 08 14:01:02.592165 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:01:02.592055    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="76f65727-300d-4713-b94b-f44c20118d62" containerName="main"
Sep 08 14:01:02.592165 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:01:02.592062    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4c6a19b-868e-4418-8c70-67dd6d0a491e" containerName="main"
Sep 08 14:01:02.594843 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:01:02.594824    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vgsvl"
Sep 08 14:01:02.597572 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:01:02.597554    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"kube-root-ca.crt\""
Sep 08 14:01:02.598372 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:01:02.598351    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvca2d2d7d499abb359505529ebe02c136-kserve-self-signed-certs\""
Sep 08 14:01:02.598372 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:01:02.598368    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"default-dockercfg-8cxq9\""
Sep 08 14:01:02.598526 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:01:02.598359    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"openshift-service-ca.crt\""
Sep 08 14:01:02.608380 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:01:02.608359    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vgsvl"]
Sep 08 14:01:02.626648 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:01:02.626614    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:01:02.761735 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:01:02.761696    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c00e1298-16fa-4bfd-b746-c030454ad65c-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vgsvl\" (UID: \"c00e1298-16fa-4bfd-b746-c030454ad65c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vgsvl"
Sep 08 14:01:02.761911 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:01:02.761751    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c00e1298-16fa-4bfd-b746-c030454ad65c-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vgsvl\" (UID: \"c00e1298-16fa-4bfd-b746-c030454ad65c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vgsvl"
Sep 08 14:01:02.761911 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:01:02.761825    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8bn2\" (UniqueName: \"kubernetes.io/projected/c00e1298-16fa-4bfd-b746-c030454ad65c-kube-api-access-k8bn2\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vgsvl\" (UID: \"c00e1298-16fa-4bfd-b746-c030454ad65c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vgsvl"
Sep 08 14:01:02.761911 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:01:02.761856    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c00e1298-16fa-4bfd-b746-c030454ad65c-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vgsvl\" (UID: \"c00e1298-16fa-4bfd-b746-c030454ad65c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vgsvl"
Sep 08 14:01:02.761911 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:01:02.761880    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c00e1298-16fa-4bfd-b746-c030454ad65c-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vgsvl\" (UID: \"c00e1298-16fa-4bfd-b746-c030454ad65c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vgsvl"
Sep 08 14:01:02.762073 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:01:02.761949    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c00e1298-16fa-4bfd-b746-c030454ad65c-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vgsvl\" (UID: \"c00e1298-16fa-4bfd-b746-c030454ad65c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vgsvl"
Sep 08 14:01:02.863179 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:01:02.863093    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c00e1298-16fa-4bfd-b746-c030454ad65c-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vgsvl\" (UID: \"c00e1298-16fa-4bfd-b746-c030454ad65c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vgsvl"
Sep 08 14:01:02.863179 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:01:02.863139    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8bn2\" (UniqueName: \"kubernetes.io/projected/c00e1298-16fa-4bfd-b746-c030454ad65c-kube-api-access-k8bn2\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vgsvl\" (UID: \"c00e1298-16fa-4bfd-b746-c030454ad65c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vgsvl"
Sep 08 14:01:02.863403 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:01:02.863261    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c00e1298-16fa-4bfd-b746-c030454ad65c-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vgsvl\" (UID: \"c00e1298-16fa-4bfd-b746-c030454ad65c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vgsvl"
Sep 08 14:01:02.863403 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:01:02.863308    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c00e1298-16fa-4bfd-b746-c030454ad65c-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vgsvl\" (UID: \"c00e1298-16fa-4bfd-b746-c030454ad65c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vgsvl"
Sep 08 14:01:02.863403 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:01:02.863363    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c00e1298-16fa-4bfd-b746-c030454ad65c-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vgsvl\" (UID: \"c00e1298-16fa-4bfd-b746-c030454ad65c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vgsvl"
Sep 08 14:01:02.863590 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:01:02.863419    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c00e1298-16fa-4bfd-b746-c030454ad65c-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vgsvl\" (UID: \"c00e1298-16fa-4bfd-b746-c030454ad65c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vgsvl"
Sep 08 14:01:02.863590 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:01:02.863474    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c00e1298-16fa-4bfd-b746-c030454ad65c-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vgsvl\" (UID: \"c00e1298-16fa-4bfd-b746-c030454ad65c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vgsvl"
Sep 08 14:01:02.863705 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:01:02.863645    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c00e1298-16fa-4bfd-b746-c030454ad65c-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vgsvl\" (UID: \"c00e1298-16fa-4bfd-b746-c030454ad65c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vgsvl"
Sep 08 14:01:02.863798 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:01:02.863780    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c00e1298-16fa-4bfd-b746-c030454ad65c-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vgsvl\" (UID: \"c00e1298-16fa-4bfd-b746-c030454ad65c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vgsvl"
Sep 08 14:01:02.865585 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:01:02.865560    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c00e1298-16fa-4bfd-b746-c030454ad65c-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vgsvl\" (UID: \"c00e1298-16fa-4bfd-b746-c030454ad65c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vgsvl"
Sep 08 14:01:02.865803 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:01:02.865786    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c00e1298-16fa-4bfd-b746-c030454ad65c-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vgsvl\" (UID: \"c00e1298-16fa-4bfd-b746-c030454ad65c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vgsvl"
Sep 08 14:01:02.866088 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:01:02.866070    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:01:02.873123 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:01:02.873099    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8bn2\" (UniqueName: \"kubernetes.io/projected/c00e1298-16fa-4bfd-b746-c030454ad65c-kube-api-access-k8bn2\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vgsvl\" (UID: \"c00e1298-16fa-4bfd-b746-c030454ad65c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vgsvl"
Sep 08 14:01:02.904485 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:01:02.904463    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vgsvl"
Sep 08 14:01:03.174235 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:01:03.174209    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vgsvl"]
Sep 08 14:01:03.177041 ip-10-0-132-254 kubenswrapper[2573]: W0908 14:01:03.177017    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc00e1298_16fa_4bfd_b746_c030454ad65c.slice/crio-07f30b2be8f1a908bad7702827692df156465757ec53beb93d202c476071a764 WatchSource:0}: Error finding container 07f30b2be8f1a908bad7702827692df156465757ec53beb93d202c476071a764: Status 404 returned error can't find the container with id 07f30b2be8f1a908bad7702827692df156465757ec53beb93d202c476071a764
Sep 08 14:01:04.045122 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:01:04.045082    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vgsvl" event={"ID":"c00e1298-16fa-4bfd-b746-c030454ad65c","Type":"ContainerStarted","Data":"a05661480e490c3279f34d38c53bdebe565b8ccc991a35566776c1ed09a8a7c7"}
Sep 08 14:01:04.045646 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:01:04.045172    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vgsvl" event={"ID":"c00e1298-16fa-4bfd-b746-c030454ad65c","Type":"ContainerStarted","Data":"07f30b2be8f1a908bad7702827692df156465757ec53beb93d202c476071a764"}
Sep 08 14:01:04.062067 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:01:04.062016    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vgsvl" podStartSLOduration=2.062001849 podStartE2EDuration="2.062001849s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 14:01:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:01:04.061945179 +0000 UTC m=+2714.333393133" watchObservedRunningTime="2026-09-08 14:01:04.062001849 +0000 UTC m=+2714.333449795"
Sep 08 14:01:04.905443 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:01:04.905405    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vgsvl"
Sep 08 14:01:05.242028 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:01:05.241954    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:01:05.344106 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:01:05.344077    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:01:06.410830 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:01:06.410798    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:01:12.904909 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:01:12.904865    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vgsvl"
Sep 08 14:01:12.917087 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:01:12.917057    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vgsvl"
Sep 08 14:01:13.098329 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:01:13.098295    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vgsvl"
Sep 08 14:01:23.565086 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:01:23.565049    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:01:27.537303 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:01:27.537267    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:01:27.634120 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:01:27.634087    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:01:29.227622 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:01:29.227587    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:01:30.802782 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:01:30.802694    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:01:31.108716 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:01:31.108678    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:01:32.612695 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:01:32.612660    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:01:38.702444 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:01:38.702412    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:01:48.039879 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:01:48.039840    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:01:53.361986 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:01:53.361951    2573 scope.go:117] "RemoveContainer" containerID="2dfdfbf5a957e01050ebdd10b8f2b289a9e90678da2bd844adc457a5cf419a38"
Sep 08 14:02:00.208680 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:02:00.208634    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:02:00.423357 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:02:00.423326    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scherktd4"]
Sep 08 14:02:00.426834 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:02:00.426818    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scherktd4"
Sep 08 14:02:00.430086 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:02:00.430066    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-epp-sa-dockercfg-2cht9\""
Sep 08 14:02:00.431705 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:02:00.431687    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisv08544b88a8d963ffd553cc1f3ed82d16-kserve-self-signed-certs\""
Sep 08 14:02:00.441896 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:02:00.441873    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scherktd4"]
Sep 08 14:02:00.483799 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:02:00.483727    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:02:00.539076 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:02:00.539044    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c31085af-73c8-441f-b7ac-ee519f0c1cec-tls-certs\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scherktd4\" (UID: \"c31085af-73c8-441f-b7ac-ee519f0c1cec\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scherktd4"
Sep 08 14:02:00.539210 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:02:00.539106    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzcxf\" (UniqueName: \"kubernetes.io/projected/c31085af-73c8-441f-b7ac-ee519f0c1cec-kube-api-access-qzcxf\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scherktd4\" (UID: \"c31085af-73c8-441f-b7ac-ee519f0c1cec\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scherktd4"
Sep 08 14:02:00.640059 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:02:00.640023    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c31085af-73c8-441f-b7ac-ee519f0c1cec-tls-certs\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scherktd4\" (UID: \"c31085af-73c8-441f-b7ac-ee519f0c1cec\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scherktd4"
Sep 08 14:02:00.640253 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:02:00.640081    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzcxf\" (UniqueName: \"kubernetes.io/projected/c31085af-73c8-441f-b7ac-ee519f0c1cec-kube-api-access-qzcxf\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scherktd4\" (UID: \"c31085af-73c8-441f-b7ac-ee519f0c1cec\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scherktd4"
Sep 08 14:02:00.642553 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:02:00.642530    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c31085af-73c8-441f-b7ac-ee519f0c1cec-tls-certs\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scherktd4\" (UID: \"c31085af-73c8-441f-b7ac-ee519f0c1cec\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scherktd4"
Sep 08 14:02:00.649514 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:02:00.649484    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzcxf\" (UniqueName: \"kubernetes.io/projected/c31085af-73c8-441f-b7ac-ee519f0c1cec-kube-api-access-qzcxf\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scherktd4\" (UID: \"c31085af-73c8-441f-b7ac-ee519f0c1cec\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scherktd4"
Sep 08 14:02:00.736296 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:02:00.736217    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scherktd4"
Sep 08 14:02:01.007775 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:02:01.007745    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scherktd4"]
Sep 08 14:02:01.010186 ip-10-0-132-254 kubenswrapper[2573]: W0908 14:02:01.010162    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc31085af_73c8_441f_b7ac_ee519f0c1cec.slice/crio-f84a46b616cece9d8f2cd6b264da04d1d59ad4fe7e23516f4885ecd932502c54 WatchSource:0}: Error finding container f84a46b616cece9d8f2cd6b264da04d1d59ad4fe7e23516f4885ecd932502c54: Status 404 returned error can't find the container with id f84a46b616cece9d8f2cd6b264da04d1d59ad4fe7e23516f4885ecd932502c54
Sep 08 14:02:01.296653 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:02:01.296513    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scherktd4" event={"ID":"c31085af-73c8-441f-b7ac-ee519f0c1cec","Type":"ContainerStarted","Data":"2837809cabb6000b0ad68011f67b8613582c523b6f081b7a13e72b37911940ca"}
Sep 08 14:02:01.296653 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:02:01.296562    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scherktd4" event={"ID":"c31085af-73c8-441f-b7ac-ee519f0c1cec","Type":"ContainerStarted","Data":"f84a46b616cece9d8f2cd6b264da04d1d59ad4fe7e23516f4885ecd932502c54"}
Sep 08 14:02:01.296653 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:02:01.296638    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scherktd4"
Sep 08 14:02:01.317932 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:02:01.317876    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scherktd4" podStartSLOduration=1.31785958 podStartE2EDuration="1.31785958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 14:02:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:02:01.317030654 +0000 UTC m=+2771.588478600" watchObservedRunningTime="2026-09-08 14:02:01.31785958 +0000 UTC m=+2771.589307511"
Sep 08 14:02:10.017242 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:02:10.017206    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:02:12.282717 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:02:12.282683    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:02:16.177994 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:02:16.177916    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:02:16.278516 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:02:16.278468    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:02:32.303057 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:02:32.303025    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scherktd4"
Sep 08 14:02:33.872078 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:02:33.872045    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scherktd4"]
Sep 08 14:02:33.879302 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:02:33.879275    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:02:34.024466 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:02:34.024432    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:02:37.340445 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:02:37.340416    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:02:42.302897 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:02:42.302858    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scherktd4" podUID="c31085af-73c8-441f-b7ac-ee519f0c1cec" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 08 14:02:42.304971 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:02:42.304946    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:02:42.400067 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:02:42.400033    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:02:47.385821 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:02:47.385789    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:02:48.891565 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:02:48.891532    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scherktd4" podUID="c31085af-73c8-441f-b7ac-ee519f0c1cec" containerName="main" containerID="cri-o://2837809cabb6000b0ad68011f67b8613582c523b6f081b7a13e72b37911940ca" gracePeriod=45
Sep 08 14:02:50.300918 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:02:50.300886    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:02:50.602767 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:02:50.602693    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:02:52.302928 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:02:52.302890    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scherktd4" podUID="c31085af-73c8-441f-b7ac-ee519f0c1cec" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 08 14:03:02.302808 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:03:02.302765    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scherktd4" podUID="c31085af-73c8-441f-b7ac-ee519f0c1cec" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 08 14:03:02.303289 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:03:02.302864    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scherktd4"
Sep 08 14:03:05.136703 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:03:05.136670    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:03:08.516583 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:03:08.516550    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:03:10.101973 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:03:10.101938    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:03:12.302769 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:03:12.302734    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scherktd4" podUID="c31085af-73c8-441f-b7ac-ee519f0c1cec" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 08 14:03:16.067180 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:03:16.067144    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:03:19.137815 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:03:19.137789    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scherktd4"
Sep 08 14:03:19.159940 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:03:19.159881    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qzcxf\" (UniqueName: \"kubernetes.io/projected/c31085af-73c8-441f-b7ac-ee519f0c1cec-kube-api-access-qzcxf\") pod \"c31085af-73c8-441f-b7ac-ee519f0c1cec\" (UID: \"c31085af-73c8-441f-b7ac-ee519f0c1cec\") "
Sep 08 14:03:19.160071 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:03:19.159972    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c31085af-73c8-441f-b7ac-ee519f0c1cec-tls-certs\") pod \"c31085af-73c8-441f-b7ac-ee519f0c1cec\" (UID: \"c31085af-73c8-441f-b7ac-ee519f0c1cec\") "
Sep 08 14:03:19.162041 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:03:19.162018    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c31085af-73c8-441f-b7ac-ee519f0c1cec-kube-api-access-qzcxf" (OuterVolumeSpecName: "kube-api-access-qzcxf") pod "c31085af-73c8-441f-b7ac-ee519f0c1cec" (UID: "c31085af-73c8-441f-b7ac-ee519f0c1cec"). InnerVolumeSpecName "kube-api-access-qzcxf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 14:03:19.162165 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:03:19.162053    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c31085af-73c8-441f-b7ac-ee519f0c1cec-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c31085af-73c8-441f-b7ac-ee519f0c1cec" (UID: "c31085af-73c8-441f-b7ac-ee519f0c1cec"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 14:03:19.260651 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:03:19.260625    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c31085af-73c8-441f-b7ac-ee519f0c1cec-tls-certs\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 14:03:19.260651 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:03:19.260647    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qzcxf\" (UniqueName: \"kubernetes.io/projected/c31085af-73c8-441f-b7ac-ee519f0c1cec-kube-api-access-qzcxf\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 14:03:19.612656 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:03:19.612563    2573 generic.go:356] "Generic (PLEG): container finished" podID="c31085af-73c8-441f-b7ac-ee519f0c1cec" containerID="2837809cabb6000b0ad68011f67b8613582c523b6f081b7a13e72b37911940ca" exitCode=0
Sep 08 14:03:19.612656 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:03:19.612611    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scherktd4"
Sep 08 14:03:19.612855 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:03:19.612611    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scherktd4" event={"ID":"c31085af-73c8-441f-b7ac-ee519f0c1cec","Type":"ContainerDied","Data":"2837809cabb6000b0ad68011f67b8613582c523b6f081b7a13e72b37911940ca"}
Sep 08 14:03:19.612855 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:03:19.612735    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scherktd4" event={"ID":"c31085af-73c8-441f-b7ac-ee519f0c1cec","Type":"ContainerDied","Data":"f84a46b616cece9d8f2cd6b264da04d1d59ad4fe7e23516f4885ecd932502c54"}
Sep 08 14:03:19.612855 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:03:19.612762    2573 scope.go:117] "RemoveContainer" containerID="2837809cabb6000b0ad68011f67b8613582c523b6f081b7a13e72b37911940ca"
Sep 08 14:03:19.621089 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:03:19.621073    2573 scope.go:117] "RemoveContainer" containerID="2837809cabb6000b0ad68011f67b8613582c523b6f081b7a13e72b37911940ca"
Sep 08 14:03:19.621337 ip-10-0-132-254 kubenswrapper[2573]: E0908 14:03:19.621320    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2837809cabb6000b0ad68011f67b8613582c523b6f081b7a13e72b37911940ca\": container with ID starting with 2837809cabb6000b0ad68011f67b8613582c523b6f081b7a13e72b37911940ca not found: ID does not exist" containerID="2837809cabb6000b0ad68011f67b8613582c523b6f081b7a13e72b37911940ca"
Sep 08 14:03:19.621389 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:03:19.621344    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2837809cabb6000b0ad68011f67b8613582c523b6f081b7a13e72b37911940ca"} err="failed to get container status \"2837809cabb6000b0ad68011f67b8613582c523b6f081b7a13e72b37911940ca\": rpc error: code = NotFound desc = could not find container \"2837809cabb6000b0ad68011f67b8613582c523b6f081b7a13e72b37911940ca\": container with ID starting with 2837809cabb6000b0ad68011f67b8613582c523b6f081b7a13e72b37911940ca not found: ID does not exist"
Sep 08 14:03:19.632495 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:03:19.632467    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scherktd4"]
Sep 08 14:03:19.636110 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:03:19.636088    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scherktd4"]
Sep 08 14:03:20.175366 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:03:20.175332    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c31085af-73c8-441f-b7ac-ee519f0c1cec" path="/var/lib/kubelet/pods/c31085af-73c8-441f-b7ac-ee519f0c1cec/volumes"
Sep 08 14:03:23.992187 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:03:23.992153    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vgsvl"]
Sep 08 14:03:24.000418 ip-10-0-132-254 kubenswrapper[2573]: E0908 14:03:24.000383    2573 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 08 14:03:24.000418 ip-10-0-132-254 kubenswrapper[2573]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 08 14:03:24.000418 ip-10-0-132-254 kubenswrapper[2573]:         , stdout: , stderr: , exit code -1
Sep 08 14:03:24.000418 ip-10-0-132-254 kubenswrapper[2573]:  > containerID="a05661480e490c3279f34d38c53bdebe565b8ccc991a35566776c1ed09a8a7c7" cmd=["/bin/sleep","15"]
Sep 08 14:03:24.000760 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:03:24.000433    2573 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vgsvl" message="" err=<
Sep 08 14:03:24.000760 ip-10-0-132-254 kubenswrapper[2573]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 08 14:03:24.000760 ip-10-0-132-254 kubenswrapper[2573]:         , stdout: , stderr: , exit code -1
Sep 08 14:03:24.000760 ip-10-0-132-254 kubenswrapper[2573]:  >
Sep 08 14:03:24.000760 ip-10-0-132-254 kubenswrapper[2573]: E0908 14:03:24.000445    2573 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 08 14:03:24.000760 ip-10-0-132-254 kubenswrapper[2573]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 08 14:03:24.000760 ip-10-0-132-254 kubenswrapper[2573]:         , stdout: , stderr: , exit code -1
Sep 08 14:03:24.000760 ip-10-0-132-254 kubenswrapper[2573]:  > pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vgsvl" podUID="c00e1298-16fa-4bfd-b746-c030454ad65c" containerName="main" containerID="cri-o://a05661480e490c3279f34d38c53bdebe565b8ccc991a35566776c1ed09a8a7c7"
Sep 08 14:03:24.000760 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:03:24.000481    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vgsvl" podUID="c00e1298-16fa-4bfd-b746-c030454ad65c" containerName="main" containerID="cri-o://a05661480e490c3279f34d38c53bdebe565b8ccc991a35566776c1ed09a8a7c7" gracePeriod=60
Sep 08 14:03:24.023517 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:03:24.023473    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:03:24.081930 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:03:24.081904    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:03:24.083837 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:03:24.083813    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vgsvl" podUID="c00e1298-16fa-4bfd-b746-c030454ad65c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": dial tcp 10.133.0.40:8000: connect: connection refused"
Sep 08 14:03:24.347579 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:03:24.347554    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vgsvl"
Sep 08 14:03:24.399416 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:03:24.399384    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k8bn2\" (UniqueName: \"kubernetes.io/projected/c00e1298-16fa-4bfd-b746-c030454ad65c-kube-api-access-k8bn2\") pod \"c00e1298-16fa-4bfd-b746-c030454ad65c\" (UID: \"c00e1298-16fa-4bfd-b746-c030454ad65c\") "
Sep 08 14:03:24.399612 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:03:24.399422    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c00e1298-16fa-4bfd-b746-c030454ad65c-dshm\") pod \"c00e1298-16fa-4bfd-b746-c030454ad65c\" (UID: \"c00e1298-16fa-4bfd-b746-c030454ad65c\") "
Sep 08 14:03:24.399612 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:03:24.399466    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c00e1298-16fa-4bfd-b746-c030454ad65c-tls-certs\") pod \"c00e1298-16fa-4bfd-b746-c030454ad65c\" (UID: \"c00e1298-16fa-4bfd-b746-c030454ad65c\") "
Sep 08 14:03:24.399612 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:03:24.399532    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c00e1298-16fa-4bfd-b746-c030454ad65c-tmp-dir\") pod \"c00e1298-16fa-4bfd-b746-c030454ad65c\" (UID: \"c00e1298-16fa-4bfd-b746-c030454ad65c\") "
Sep 08 14:03:24.399612 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:03:24.399564    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c00e1298-16fa-4bfd-b746-c030454ad65c-model-cache\") pod \"c00e1298-16fa-4bfd-b746-c030454ad65c\" (UID: \"c00e1298-16fa-4bfd-b746-c030454ad65c\") "
Sep 08 14:03:24.399612 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:03:24.399580    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c00e1298-16fa-4bfd-b746-c030454ad65c-home\") pod \"c00e1298-16fa-4bfd-b746-c030454ad65c\" (UID: \"c00e1298-16fa-4bfd-b746-c030454ad65c\") "
Sep 08 14:03:24.399905 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:03:24.399849    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c00e1298-16fa-4bfd-b746-c030454ad65c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c00e1298-16fa-4bfd-b746-c030454ad65c" (UID: "c00e1298-16fa-4bfd-b746-c030454ad65c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:03:24.399964 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:03:24.399879    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c00e1298-16fa-4bfd-b746-c030454ad65c-model-cache" (OuterVolumeSpecName: "model-cache") pod "c00e1298-16fa-4bfd-b746-c030454ad65c" (UID: "c00e1298-16fa-4bfd-b746-c030454ad65c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:03:24.400021 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:03:24.399948    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c00e1298-16fa-4bfd-b746-c030454ad65c-home" (OuterVolumeSpecName: "home") pod "c00e1298-16fa-4bfd-b746-c030454ad65c" (UID: "c00e1298-16fa-4bfd-b746-c030454ad65c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:03:24.401626 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:03:24.401604    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c00e1298-16fa-4bfd-b746-c030454ad65c-dshm" (OuterVolumeSpecName: "dshm") pod "c00e1298-16fa-4bfd-b746-c030454ad65c" (UID: "c00e1298-16fa-4bfd-b746-c030454ad65c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:03:24.401724 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:03:24.401700    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c00e1298-16fa-4bfd-b746-c030454ad65c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c00e1298-16fa-4bfd-b746-c030454ad65c" (UID: "c00e1298-16fa-4bfd-b746-c030454ad65c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 14:03:24.401769 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:03:24.401734    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c00e1298-16fa-4bfd-b746-c030454ad65c-kube-api-access-k8bn2" (OuterVolumeSpecName: "kube-api-access-k8bn2") pod "c00e1298-16fa-4bfd-b746-c030454ad65c" (UID: "c00e1298-16fa-4bfd-b746-c030454ad65c"). InnerVolumeSpecName "kube-api-access-k8bn2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 14:03:24.500932 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:03:24.500893    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c00e1298-16fa-4bfd-b746-c030454ad65c-tls-certs\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 14:03:24.500932 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:03:24.500926    2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c00e1298-16fa-4bfd-b746-c030454ad65c-tmp-dir\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 14:03:24.500932 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:03:24.500934    2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c00e1298-16fa-4bfd-b746-c030454ad65c-model-cache\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 14:03:24.500932 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:03:24.500942    2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c00e1298-16fa-4bfd-b746-c030454ad65c-home\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 14:03:24.501192 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:03:24.500950    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8bn2\" (UniqueName: \"kubernetes.io/projected/c00e1298-16fa-4bfd-b746-c030454ad65c-kube-api-access-k8bn2\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 14:03:24.501192 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:03:24.500959    2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c00e1298-16fa-4bfd-b746-c030454ad65c-dshm\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 14:03:24.635100 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:03:24.635062    2573 generic.go:356] "Generic (PLEG): container finished" podID="c00e1298-16fa-4bfd-b746-c030454ad65c" containerID="a05661480e490c3279f34d38c53bdebe565b8ccc991a35566776c1ed09a8a7c7" exitCode=0
Sep 08 14:03:24.635264 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:03:24.635111    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vgsvl"
Sep 08 14:03:24.635264 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:03:24.635108    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vgsvl" event={"ID":"c00e1298-16fa-4bfd-b746-c030454ad65c","Type":"ContainerDied","Data":"a05661480e490c3279f34d38c53bdebe565b8ccc991a35566776c1ed09a8a7c7"}
Sep 08 14:03:24.635264 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:03:24.635244    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vgsvl" event={"ID":"c00e1298-16fa-4bfd-b746-c030454ad65c","Type":"ContainerDied","Data":"07f30b2be8f1a908bad7702827692df156465757ec53beb93d202c476071a764"}
Sep 08 14:03:24.635379 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:03:24.635275    2573 scope.go:117] "RemoveContainer" containerID="a05661480e490c3279f34d38c53bdebe565b8ccc991a35566776c1ed09a8a7c7"
Sep 08 14:03:24.644841 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:03:24.644822    2573 scope.go:117] "RemoveContainer" containerID="a05661480e490c3279f34d38c53bdebe565b8ccc991a35566776c1ed09a8a7c7"
Sep 08 14:03:24.645082 ip-10-0-132-254 kubenswrapper[2573]: E0908 14:03:24.645064    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a05661480e490c3279f34d38c53bdebe565b8ccc991a35566776c1ed09a8a7c7\": container with ID starting with a05661480e490c3279f34d38c53bdebe565b8ccc991a35566776c1ed09a8a7c7 not found: ID does not exist" containerID="a05661480e490c3279f34d38c53bdebe565b8ccc991a35566776c1ed09a8a7c7"
Sep 08 14:03:24.645133 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:03:24.645090    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a05661480e490c3279f34d38c53bdebe565b8ccc991a35566776c1ed09a8a7c7"} err="failed to get container status \"a05661480e490c3279f34d38c53bdebe565b8ccc991a35566776c1ed09a8a7c7\": rpc error: code = NotFound desc = could not find container \"a05661480e490c3279f34d38c53bdebe565b8ccc991a35566776c1ed09a8a7c7\": container with ID starting with a05661480e490c3279f34d38c53bdebe565b8ccc991a35566776c1ed09a8a7c7 not found: ID does not exist"
Sep 08 14:03:24.661260 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:03:24.661232    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vgsvl"]
Sep 08 14:03:24.665290 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:03:24.665267    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vgsvl"]
Sep 08 14:03:26.174307 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:03:26.174270    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c00e1298-16fa-4bfd-b746-c030454ad65c" path="/var/lib/kubelet/pods/c00e1298-16fa-4bfd-b746-c030454ad65c/volumes"
Sep 08 14:03:29.870989 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:03:29.870957    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:03:30.729761 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:03:30.729729    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:03:30.831728 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:03:30.831648    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:04:14.134850 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:04:14.134816    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:04:14.277033 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:04:14.276998    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:04:20.476752 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:04:20.476719    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:04:21.330290 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:04:21.330250    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:04:22.428549 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:04:22.428516    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:04:23.748749 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:04:23.748715    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-8dfd58b86-dtn9b"]
Sep 08 14:04:23.749215 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:04:23.749037    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c00e1298-16fa-4bfd-b746-c030454ad65c" containerName="main"
Sep 08 14:04:23.749215 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:04:23.749052    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="c00e1298-16fa-4bfd-b746-c030454ad65c" containerName="main"
Sep 08 14:04:23.749215 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:04:23.749071    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c31085af-73c8-441f-b7ac-ee519f0c1cec" containerName="main"
Sep 08 14:04:23.749215 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:04:23.749076    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="c31085af-73c8-441f-b7ac-ee519f0c1cec" containerName="main"
Sep 08 14:04:23.749215 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:04:23.749125    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="c00e1298-16fa-4bfd-b746-c030454ad65c" containerName="main"
Sep 08 14:04:23.749215 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:04:23.749135    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="c31085af-73c8-441f-b7ac-ee519f0c1cec" containerName="main"
Sep 08 14:04:23.752114 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:04:23.752100    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-8dfd58b86-dtn9b"
Sep 08 14:04:23.754911 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:04:23.754890    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"odh-kserve-custom-ca-bundle\""
Sep 08 14:04:23.755783 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:04:23.755763    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"default-dockercfg-9825j\""
Sep 08 14:04:23.755876 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:04:23.755796    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"kube-root-ca.crt\""
Sep 08 14:04:23.755876 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:04:23.755798    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"seaweedfs-s3-creds\""
Sep 08 14:04:23.756033 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:04:23.756012    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"openshift-service-ca.crt\""
Sep 08 14:04:23.756428 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:04:23.756410    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"llmisv3e414c2ba058a022dfd694dbcbac5b51-kserve-self-signed-certs\""
Sep 08 14:04:23.772876 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:04:23.772857    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-8dfd58b86-dtn9b"]
Sep 08 14:04:23.777064 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:04:23.777011    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:04:23.850347 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:04:23.850324    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ee41a3ed-54ab-4e79-aa42-c549ab9d97f4-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-8dfd58b86-dtn9b\" (UID: \"ee41a3ed-54ab-4e79-aa42-c549ab9d97f4\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-8dfd58b86-dtn9b"
Sep 08 14:04:23.850477 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:04:23.850356    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee41a3ed-54ab-4e79-aa42-c549ab9d97f4-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-8dfd58b86-dtn9b\" (UID: \"ee41a3ed-54ab-4e79-aa42-c549ab9d97f4\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-8dfd58b86-dtn9b"
Sep 08 14:04:23.850477 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:04:23.850379    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee41a3ed-54ab-4e79-aa42-c549ab9d97f4-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-8dfd58b86-dtn9b\" (UID: \"ee41a3ed-54ab-4e79-aa42-c549ab9d97f4\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-8dfd58b86-dtn9b"
Sep 08 14:04:23.850477 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:04:23.850398    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ee41a3ed-54ab-4e79-aa42-c549ab9d97f4-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-8dfd58b86-dtn9b\" (UID: \"ee41a3ed-54ab-4e79-aa42-c549ab9d97f4\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-8dfd58b86-dtn9b"
Sep 08 14:04:23.850477 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:04:23.850442    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee41a3ed-54ab-4e79-aa42-c549ab9d97f4-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-8dfd58b86-dtn9b\" (UID: \"ee41a3ed-54ab-4e79-aa42-c549ab9d97f4\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-8dfd58b86-dtn9b"
Sep 08 14:04:23.850477 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:04:23.850470    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee41a3ed-54ab-4e79-aa42-c549ab9d97f4-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-8dfd58b86-dtn9b\" (UID: \"ee41a3ed-54ab-4e79-aa42-c549ab9d97f4\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-8dfd58b86-dtn9b"
Sep 08 14:04:23.850681 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:04:23.850492    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t69zc\" (UniqueName: \"kubernetes.io/projected/ee41a3ed-54ab-4e79-aa42-c549ab9d97f4-kube-api-access-t69zc\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-8dfd58b86-dtn9b\" (UID: \"ee41a3ed-54ab-4e79-aa42-c549ab9d97f4\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-8dfd58b86-dtn9b"
Sep 08 14:04:23.850681 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:04:23.850526    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee41a3ed-54ab-4e79-aa42-c549ab9d97f4-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-8dfd58b86-dtn9b\" (UID: \"ee41a3ed-54ab-4e79-aa42-c549ab9d97f4\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-8dfd58b86-dtn9b"
Sep 08 14:04:23.951224 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:04:23.951198    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee41a3ed-54ab-4e79-aa42-c549ab9d97f4-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-8dfd58b86-dtn9b\" (UID: \"ee41a3ed-54ab-4e79-aa42-c549ab9d97f4\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-8dfd58b86-dtn9b"
Sep 08 14:04:23.951372 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:04:23.951240    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee41a3ed-54ab-4e79-aa42-c549ab9d97f4-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-8dfd58b86-dtn9b\" (UID: \"ee41a3ed-54ab-4e79-aa42-c549ab9d97f4\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-8dfd58b86-dtn9b"
Sep 08 14:04:23.951372 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:04:23.951262    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t69zc\" (UniqueName: \"kubernetes.io/projected/ee41a3ed-54ab-4e79-aa42-c549ab9d97f4-kube-api-access-t69zc\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-8dfd58b86-dtn9b\" (UID: \"ee41a3ed-54ab-4e79-aa42-c549ab9d97f4\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-8dfd58b86-dtn9b"
Sep 08 14:04:23.951476 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:04:23.951377    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee41a3ed-54ab-4e79-aa42-c549ab9d97f4-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-8dfd58b86-dtn9b\" (UID: \"ee41a3ed-54ab-4e79-aa42-c549ab9d97f4\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-8dfd58b86-dtn9b"
Sep 08 14:04:23.951476 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:04:23.951429    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ee41a3ed-54ab-4e79-aa42-c549ab9d97f4-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-8dfd58b86-dtn9b\" (UID: \"ee41a3ed-54ab-4e79-aa42-c549ab9d97f4\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-8dfd58b86-dtn9b"
Sep 08 14:04:23.951476 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:04:23.951459    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee41a3ed-54ab-4e79-aa42-c549ab9d97f4-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-8dfd58b86-dtn9b\" (UID: \"ee41a3ed-54ab-4e79-aa42-c549ab9d97f4\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-8dfd58b86-dtn9b"
Sep 08 14:04:23.951662 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:04:23.951496    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee41a3ed-54ab-4e79-aa42-c549ab9d97f4-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-8dfd58b86-dtn9b\" (UID: \"ee41a3ed-54ab-4e79-aa42-c549ab9d97f4\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-8dfd58b86-dtn9b"
Sep 08 14:04:23.951662 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:04:23.951548    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ee41a3ed-54ab-4e79-aa42-c549ab9d97f4-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-8dfd58b86-dtn9b\" (UID: \"ee41a3ed-54ab-4e79-aa42-c549ab9d97f4\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-8dfd58b86-dtn9b"
Sep 08 14:04:23.951777 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:04:23.951733    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee41a3ed-54ab-4e79-aa42-c549ab9d97f4-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-8dfd58b86-dtn9b\" (UID: \"ee41a3ed-54ab-4e79-aa42-c549ab9d97f4\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-8dfd58b86-dtn9b"
Sep 08 14:04:23.951878 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:04:23.951859    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee41a3ed-54ab-4e79-aa42-c549ab9d97f4-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-8dfd58b86-dtn9b\" (UID: \"ee41a3ed-54ab-4e79-aa42-c549ab9d97f4\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-8dfd58b86-dtn9b"
Sep 08 14:04:23.951937 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:04:23.951878    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ee41a3ed-54ab-4e79-aa42-c549ab9d97f4-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-8dfd58b86-dtn9b\" (UID: \"ee41a3ed-54ab-4e79-aa42-c549ab9d97f4\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-8dfd58b86-dtn9b"
Sep 08 14:04:23.951999 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:04:23.951962    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee41a3ed-54ab-4e79-aa42-c549ab9d97f4-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-8dfd58b86-dtn9b\" (UID: \"ee41a3ed-54ab-4e79-aa42-c549ab9d97f4\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-8dfd58b86-dtn9b"
Sep 08 14:04:23.952182 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:04:23.952165    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ee41a3ed-54ab-4e79-aa42-c549ab9d97f4-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-8dfd58b86-dtn9b\" (UID: \"ee41a3ed-54ab-4e79-aa42-c549ab9d97f4\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-8dfd58b86-dtn9b"
Sep 08 14:04:23.953711 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:04:23.953695    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee41a3ed-54ab-4e79-aa42-c549ab9d97f4-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-8dfd58b86-dtn9b\" (UID: \"ee41a3ed-54ab-4e79-aa42-c549ab9d97f4\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-8dfd58b86-dtn9b"
Sep 08 14:04:23.953999 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:04:23.953979    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee41a3ed-54ab-4e79-aa42-c549ab9d97f4-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-8dfd58b86-dtn9b\" (UID: \"ee41a3ed-54ab-4e79-aa42-c549ab9d97f4\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-8dfd58b86-dtn9b"
Sep 08 14:04:23.965018 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:04:23.964992    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t69zc\" (UniqueName: \"kubernetes.io/projected/ee41a3ed-54ab-4e79-aa42-c549ab9d97f4-kube-api-access-t69zc\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-8dfd58b86-dtn9b\" (UID: \"ee41a3ed-54ab-4e79-aa42-c549ab9d97f4\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-8dfd58b86-dtn9b"
Sep 08 14:04:24.034118 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:04:24.034043    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:04:24.062391 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:04:24.062364    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-8dfd58b86-dtn9b"
Sep 08 14:04:24.342555 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:04:24.342530    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-8dfd58b86-dtn9b"]
Sep 08 14:04:24.344608 ip-10-0-132-254 kubenswrapper[2573]: W0908 14:04:24.344584    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee41a3ed_54ab_4e79_aa42_c549ab9d97f4.slice/crio-943fb573ad58e3f8b688b9d26779ae2f40f56e1dd95c154301afb8c7af6e9f76 WatchSource:0}: Error finding container 943fb573ad58e3f8b688b9d26779ae2f40f56e1dd95c154301afb8c7af6e9f76: Status 404 returned error can't find the container with id 943fb573ad58e3f8b688b9d26779ae2f40f56e1dd95c154301afb8c7af6e9f76
Sep 08 14:04:24.881662 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:04:24.881629    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-8dfd58b86-dtn9b" event={"ID":"ee41a3ed-54ab-4e79-aa42-c549ab9d97f4","Type":"ContainerStarted","Data":"9ba564e1f3a5bdf165079ea1a8855e57da4143c09a044973f4869bc8950ec59a"}
Sep 08 14:04:24.882136 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:04:24.881699    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-8dfd58b86-dtn9b" event={"ID":"ee41a3ed-54ab-4e79-aa42-c549ab9d97f4","Type":"ContainerStarted","Data":"943fb573ad58e3f8b688b9d26779ae2f40f56e1dd95c154301afb8c7af6e9f76"}
Sep 08 14:04:27.464352 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:04:27.464265    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:04:27.565569 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:04:27.565542    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:04:27.898988 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:04:27.898952    2573 generic.go:356] "Generic (PLEG): container finished" podID="ee41a3ed-54ab-4e79-aa42-c549ab9d97f4" containerID="9ba564e1f3a5bdf165079ea1a8855e57da4143c09a044973f4869bc8950ec59a" exitCode=0
Sep 08 14:04:27.899137 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:04:27.899028    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-8dfd58b86-dtn9b" event={"ID":"ee41a3ed-54ab-4e79-aa42-c549ab9d97f4","Type":"ContainerDied","Data":"9ba564e1f3a5bdf165079ea1a8855e57da4143c09a044973f4869bc8950ec59a"}
Sep 08 14:04:28.905595 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:04:28.905561    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-8dfd58b86-dtn9b" event={"ID":"ee41a3ed-54ab-4e79-aa42-c549ab9d97f4","Type":"ContainerStarted","Data":"205110bf4195508e926357d01949cf200d50c0954ee6a51f67d10f27a8240de9"}
Sep 08 14:04:28.933205 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:04:28.933150    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-8dfd58b86-dtn9b" podStartSLOduration=3.175408022 podStartE2EDuration="5.933134876s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.757726854s" isStatefulPod=false podCreationTimestamp="2026-09-08 14:04:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:04:28.931389593 +0000 UTC m=+2919.202837537" watchObservedRunningTime="2026-09-08 14:04:28.933134876 +0000 UTC m=+2919.204582811"
Sep 08 14:04:29.062836 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:04:29.062804    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-8dfd58b86-dtn9b"
Sep 08 14:04:34.062993 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:04:34.062960    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-8dfd58b86-dtn9b"
Sep 08 14:04:34.064538 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:04:34.064491    2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-8dfd58b86-dtn9b" podUID="ee41a3ed-54ab-4e79-aa42-c549ab9d97f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/health\": dial tcp 10.133.0.42:8000: connect: connection refused"
Sep 08 14:04:34.832686 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:04:34.832657    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:04:38.845299 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:04:38.845266    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:04:44.062962 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:04:44.062907    2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-8dfd58b86-dtn9b" podUID="ee41a3ed-54ab-4e79-aa42-c549ab9d97f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/health\": dial tcp 10.133.0.42:8000: connect: connection refused"
Sep 08 14:04:54.063046 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:04:54.062999    2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-8dfd58b86-dtn9b" podUID="ee41a3ed-54ab-4e79-aa42-c549ab9d97f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/health\": dial tcp 10.133.0.42:8000: connect: connection refused"
Sep 08 14:04:56.636510 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:04:56.636479    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:04:58.552133 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:04:58.552095    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:05:04.063248 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:05:04.063210    2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-8dfd58b86-dtn9b" podUID="ee41a3ed-54ab-4e79-aa42-c549ab9d97f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/health\": dial tcp 10.133.0.42:8000: connect: connection refused"
Sep 08 14:05:04.869047 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:05:04.869011    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:05:14.063361 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:05:14.063270    2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-8dfd58b86-dtn9b" podUID="ee41a3ed-54ab-4e79-aa42-c549ab9d97f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/health\": dial tcp 10.133.0.42:8000: connect: connection refused"
Sep 08 14:05:24.063516 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:05:24.063453    2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-8dfd58b86-dtn9b" podUID="ee41a3ed-54ab-4e79-aa42-c549ab9d97f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/health\": dial tcp 10.133.0.42:8000: connect: connection refused"
Sep 08 14:05:34.063734 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:05:34.063694    2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-8dfd58b86-dtn9b" podUID="ee41a3ed-54ab-4e79-aa42-c549ab9d97f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/health\": dial tcp 10.133.0.42:8000: connect: connection refused"
Sep 08 14:05:44.072552 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:05:44.072495    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-8dfd58b86-dtn9b"
Sep 08 14:05:44.080016 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:05:44.079993    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-8dfd58b86-dtn9b"
Sep 08 14:05:45.683712 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:05:45.683684    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:05:45.777351 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:05:45.777321    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:05:50.350489 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:05:50.350462    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8h8cn_b936238e-bdf1-4dac-8bf8-6417c313597c/ovn-acl-logging/0.log"
Sep 08 14:05:50.357429 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:05:50.357408    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8h8cn_b936238e-bdf1-4dac-8bf8-6417c313597c/ovn-acl-logging/0.log"
Sep 08 14:05:51.943960 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:05:51.943924    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:05:57.726775 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:05:57.726744    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:05:57.828038 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:05:57.828010    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:06:03.975778 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:06:03.975742    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:06:04.314440 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:06:04.314368    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:06:15.072951 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:06:15.072911    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:06:18.379115 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:06:18.379087    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:06:20.015777 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:06:20.015738    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:06:20.029710 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:06:20.029683    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-8dfd58b86-dtn9b"]
Sep 08 14:06:20.116231 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:06:20.116197    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:06:35.052627 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:06:35.052579    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-8dfd58b86-dtn9b" podUID="ee41a3ed-54ab-4e79-aa42-c549ab9d97f4" containerName="main" containerID="cri-o://205110bf4195508e926357d01949cf200d50c0954ee6a51f67d10f27a8240de9" gracePeriod=45
Sep 08 14:06:35.697288 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:06:35.697264    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-8dfd58b86-dtn9b"
Sep 08 14:06:35.731820 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:06:35.731748    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee41a3ed-54ab-4e79-aa42-c549ab9d97f4-dshm\") pod \"ee41a3ed-54ab-4e79-aa42-c549ab9d97f4\" (UID: \"ee41a3ed-54ab-4e79-aa42-c549ab9d97f4\") "
Sep 08 14:06:35.731820 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:06:35.731799    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee41a3ed-54ab-4e79-aa42-c549ab9d97f4-home\") pod \"ee41a3ed-54ab-4e79-aa42-c549ab9d97f4\" (UID: \"ee41a3ed-54ab-4e79-aa42-c549ab9d97f4\") "
Sep 08 14:06:35.732005 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:06:35.731846    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee41a3ed-54ab-4e79-aa42-c549ab9d97f4-tmp-dir\") pod \"ee41a3ed-54ab-4e79-aa42-c549ab9d97f4\" (UID: \"ee41a3ed-54ab-4e79-aa42-c549ab9d97f4\") "
Sep 08 14:06:35.732005 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:06:35.731884    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee41a3ed-54ab-4e79-aa42-c549ab9d97f4-model-cache\") pod \"ee41a3ed-54ab-4e79-aa42-c549ab9d97f4\" (UID: \"ee41a3ed-54ab-4e79-aa42-c549ab9d97f4\") "
Sep 08 14:06:35.732005 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:06:35.731910    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t69zc\" (UniqueName: \"kubernetes.io/projected/ee41a3ed-54ab-4e79-aa42-c549ab9d97f4-kube-api-access-t69zc\") pod \"ee41a3ed-54ab-4e79-aa42-c549ab9d97f4\" (UID: \"ee41a3ed-54ab-4e79-aa42-c549ab9d97f4\") "
Sep 08 14:06:35.732005 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:06:35.731964    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ee41a3ed-54ab-4e79-aa42-c549ab9d97f4-cabundle-cert\") pod \"ee41a3ed-54ab-4e79-aa42-c549ab9d97f4\" (UID: \"ee41a3ed-54ab-4e79-aa42-c549ab9d97f4\") "
Sep 08 14:06:35.732005 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:06:35.732002    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee41a3ed-54ab-4e79-aa42-c549ab9d97f4-tls-certs\") pod \"ee41a3ed-54ab-4e79-aa42-c549ab9d97f4\" (UID: \"ee41a3ed-54ab-4e79-aa42-c549ab9d97f4\") "
Sep 08 14:06:35.732250 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:06:35.732041    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ee41a3ed-54ab-4e79-aa42-c549ab9d97f4-kserve-provision-location\") pod \"ee41a3ed-54ab-4e79-aa42-c549ab9d97f4\" (UID: \"ee41a3ed-54ab-4e79-aa42-c549ab9d97f4\") "
Sep 08 14:06:35.732303 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:06:35.732267    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee41a3ed-54ab-4e79-aa42-c549ab9d97f4-model-cache" (OuterVolumeSpecName: "model-cache") pod "ee41a3ed-54ab-4e79-aa42-c549ab9d97f4" (UID: "ee41a3ed-54ab-4e79-aa42-c549ab9d97f4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:06:35.732730 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:06:35.732703    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee41a3ed-54ab-4e79-aa42-c549ab9d97f4-home" (OuterVolumeSpecName: "home") pod "ee41a3ed-54ab-4e79-aa42-c549ab9d97f4" (UID: "ee41a3ed-54ab-4e79-aa42-c549ab9d97f4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:06:35.732997 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:06:35.732965    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee41a3ed-54ab-4e79-aa42-c549ab9d97f4-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ee41a3ed-54ab-4e79-aa42-c549ab9d97f4" (UID: "ee41a3ed-54ab-4e79-aa42-c549ab9d97f4"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 14:06:35.734484 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:06:35.734455    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee41a3ed-54ab-4e79-aa42-c549ab9d97f4-dshm" (OuterVolumeSpecName: "dshm") pod "ee41a3ed-54ab-4e79-aa42-c549ab9d97f4" (UID: "ee41a3ed-54ab-4e79-aa42-c549ab9d97f4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:06:35.735225 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:06:35.735198    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee41a3ed-54ab-4e79-aa42-c549ab9d97f4-kube-api-access-t69zc" (OuterVolumeSpecName: "kube-api-access-t69zc") pod "ee41a3ed-54ab-4e79-aa42-c549ab9d97f4" (UID: "ee41a3ed-54ab-4e79-aa42-c549ab9d97f4"). InnerVolumeSpecName "kube-api-access-t69zc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 14:06:35.735338 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:06:35.735210    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee41a3ed-54ab-4e79-aa42-c549ab9d97f4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ee41a3ed-54ab-4e79-aa42-c549ab9d97f4" (UID: "ee41a3ed-54ab-4e79-aa42-c549ab9d97f4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 14:06:35.750400 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:06:35.750370    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee41a3ed-54ab-4e79-aa42-c549ab9d97f4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ee41a3ed-54ab-4e79-aa42-c549ab9d97f4" (UID: "ee41a3ed-54ab-4e79-aa42-c549ab9d97f4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:06:35.760474 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:06:35.760450    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee41a3ed-54ab-4e79-aa42-c549ab9d97f4-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ee41a3ed-54ab-4e79-aa42-c549ab9d97f4" (UID: "ee41a3ed-54ab-4e79-aa42-c549ab9d97f4"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:06:35.833486 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:06:35.833463    2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee41a3ed-54ab-4e79-aa42-c549ab9d97f4-model-cache\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 14:06:35.833588 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:06:35.833486    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t69zc\" (UniqueName: \"kubernetes.io/projected/ee41a3ed-54ab-4e79-aa42-c549ab9d97f4-kube-api-access-t69zc\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 14:06:35.833588 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:06:35.833521    2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ee41a3ed-54ab-4e79-aa42-c549ab9d97f4-cabundle-cert\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 14:06:35.833588 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:06:35.833531    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee41a3ed-54ab-4e79-aa42-c549ab9d97f4-tls-certs\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 14:06:35.833588 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:06:35.833539    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ee41a3ed-54ab-4e79-aa42-c549ab9d97f4-kserve-provision-location\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 14:06:35.833588 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:06:35.833546    2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee41a3ed-54ab-4e79-aa42-c549ab9d97f4-dshm\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 14:06:35.833588 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:06:35.833554    2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee41a3ed-54ab-4e79-aa42-c549ab9d97f4-home\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 14:06:35.833588 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:06:35.833561    2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee41a3ed-54ab-4e79-aa42-c549ab9d97f4-tmp-dir\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 14:06:36.423400 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:06:36.423364    2573 generic.go:356] "Generic (PLEG): container finished" podID="ee41a3ed-54ab-4e79-aa42-c549ab9d97f4" containerID="205110bf4195508e926357d01949cf200d50c0954ee6a51f67d10f27a8240de9" exitCode=0
Sep 08 14:06:36.423865 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:06:36.423425    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-8dfd58b86-dtn9b"
Sep 08 14:06:36.423865 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:06:36.423450    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-8dfd58b86-dtn9b" event={"ID":"ee41a3ed-54ab-4e79-aa42-c549ab9d97f4","Type":"ContainerDied","Data":"205110bf4195508e926357d01949cf200d50c0954ee6a51f67d10f27a8240de9"}
Sep 08 14:06:36.423865 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:06:36.423526    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-8dfd58b86-dtn9b" event={"ID":"ee41a3ed-54ab-4e79-aa42-c549ab9d97f4","Type":"ContainerDied","Data":"943fb573ad58e3f8b688b9d26779ae2f40f56e1dd95c154301afb8c7af6e9f76"}
Sep 08 14:06:36.423865 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:06:36.423552    2573 scope.go:117] "RemoveContainer" containerID="205110bf4195508e926357d01949cf200d50c0954ee6a51f67d10f27a8240de9"
Sep 08 14:06:36.431995 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:06:36.431980    2573 scope.go:117] "RemoveContainer" containerID="9ba564e1f3a5bdf165079ea1a8855e57da4143c09a044973f4869bc8950ec59a"
Sep 08 14:06:36.439441 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:06:36.439422    2573 scope.go:117] "RemoveContainer" containerID="205110bf4195508e926357d01949cf200d50c0954ee6a51f67d10f27a8240de9"
Sep 08 14:06:36.439749 ip-10-0-132-254 kubenswrapper[2573]: E0908 14:06:36.439726    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"205110bf4195508e926357d01949cf200d50c0954ee6a51f67d10f27a8240de9\": container with ID starting with 205110bf4195508e926357d01949cf200d50c0954ee6a51f67d10f27a8240de9 not found: ID does not exist" containerID="205110bf4195508e926357d01949cf200d50c0954ee6a51f67d10f27a8240de9"
Sep 08 14:06:36.439837 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:06:36.439758    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"205110bf4195508e926357d01949cf200d50c0954ee6a51f67d10f27a8240de9"} err="failed to get container status \"205110bf4195508e926357d01949cf200d50c0954ee6a51f67d10f27a8240de9\": rpc error: code = NotFound desc = could not find container \"205110bf4195508e926357d01949cf200d50c0954ee6a51f67d10f27a8240de9\": container with ID starting with 205110bf4195508e926357d01949cf200d50c0954ee6a51f67d10f27a8240de9 not found: ID does not exist"
Sep 08 14:06:36.439837 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:06:36.439781    2573 scope.go:117] "RemoveContainer" containerID="9ba564e1f3a5bdf165079ea1a8855e57da4143c09a044973f4869bc8950ec59a"
Sep 08 14:06:36.440064 ip-10-0-132-254 kubenswrapper[2573]: E0908 14:06:36.440045    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ba564e1f3a5bdf165079ea1a8855e57da4143c09a044973f4869bc8950ec59a\": container with ID starting with 9ba564e1f3a5bdf165079ea1a8855e57da4143c09a044973f4869bc8950ec59a not found: ID does not exist" containerID="9ba564e1f3a5bdf165079ea1a8855e57da4143c09a044973f4869bc8950ec59a"
Sep 08 14:06:36.440109 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:06:36.440070    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ba564e1f3a5bdf165079ea1a8855e57da4143c09a044973f4869bc8950ec59a"} err="failed to get container status \"9ba564e1f3a5bdf165079ea1a8855e57da4143c09a044973f4869bc8950ec59a\": rpc error: code = NotFound desc = could not find container \"9ba564e1f3a5bdf165079ea1a8855e57da4143c09a044973f4869bc8950ec59a\": container with ID starting with 9ba564e1f3a5bdf165079ea1a8855e57da4143c09a044973f4869bc8950ec59a not found: ID does not exist"
Sep 08 14:06:36.440842 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:06:36.440813    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-8dfd58b86-dtn9b"]
Sep 08 14:06:36.444485 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:06:36.444464    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-8dfd58b86-dtn9b"]
Sep 08 14:06:38.174608 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:06:38.174578    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ee41a3ed-54ab-4e79-aa42-c549ab9d97f4" path="/var/lib/kubelet/pods/ee41a3ed-54ab-4e79-aa42-c549ab9d97f4/volumes"
Sep 08 14:07:09.688978 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:07:09.688942    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:07:09.786008 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:07:09.785970    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:07:16.036519 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:07:16.036467    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:07:16.498722 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:07:16.498690    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-f45d57d9c-8zltg"]
Sep 08 14:07:16.499059 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:07:16.499042    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee41a3ed-54ab-4e79-aa42-c549ab9d97f4" containerName="storage-initializer"
Sep 08 14:07:16.499156 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:07:16.499062    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee41a3ed-54ab-4e79-aa42-c549ab9d97f4" containerName="storage-initializer"
Sep 08 14:07:16.499156 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:07:16.499088    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee41a3ed-54ab-4e79-aa42-c549ab9d97f4" containerName="main"
Sep 08 14:07:16.499156 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:07:16.499097    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee41a3ed-54ab-4e79-aa42-c549ab9d97f4" containerName="main"
Sep 08 14:07:16.499320 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:07:16.499177    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee41a3ed-54ab-4e79-aa42-c549ab9d97f4" containerName="main"
Sep 08 14:07:16.502281 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:07:16.502262    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-f45d57d9c-8zltg"
Sep 08 14:07:16.504730 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:07:16.504707    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"seaweedfs-s3-creds\""
Sep 08 14:07:16.504852 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:07:16.504730    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"kube-root-ca.crt\""
Sep 08 14:07:16.511352 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:07:16.505200    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"odh-kserve-custom-ca-bundle\""
Sep 08 14:07:16.511352 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:07:16.505405    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs\""
Sep 08 14:07:16.511352 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:07:16.505428    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"openshift-service-ca.crt\""
Sep 08 14:07:16.511352 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:07:16.506004    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"default-dockercfg-9dd8p\""
Sep 08 14:07:16.518725 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:07:16.518703    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-f45d57d9c-8zltg"]
Sep 08 14:07:16.524983 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:07:16.524947    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:07:16.645344 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:07:16.645314    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa5026b4-dfea-44dd-819c-bcb604815093-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-f45d57d9c-8zltg\" (UID: \"aa5026b4-dfea-44dd-819c-bcb604815093\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-f45d57d9c-8zltg"
Sep 08 14:07:16.645575 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:07:16.645351    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa5026b4-dfea-44dd-819c-bcb604815093-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-f45d57d9c-8zltg\" (UID: \"aa5026b4-dfea-44dd-819c-bcb604815093\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-f45d57d9c-8zltg"
Sep 08 14:07:16.645575 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:07:16.645386    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa5026b4-dfea-44dd-819c-bcb604815093-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-f45d57d9c-8zltg\" (UID: \"aa5026b4-dfea-44dd-819c-bcb604815093\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-f45d57d9c-8zltg"
Sep 08 14:07:16.645575 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:07:16.645441    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa5026b4-dfea-44dd-819c-bcb604815093-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-f45d57d9c-8zltg\" (UID: \"aa5026b4-dfea-44dd-819c-bcb604815093\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-f45d57d9c-8zltg"
Sep 08 14:07:16.645575 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:07:16.645477    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/aa5026b4-dfea-44dd-819c-bcb604815093-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-f45d57d9c-8zltg\" (UID: \"aa5026b4-dfea-44dd-819c-bcb604815093\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-f45d57d9c-8zltg"
Sep 08 14:07:16.645575 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:07:16.645535    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa5026b4-dfea-44dd-819c-bcb604815093-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-f45d57d9c-8zltg\" (UID: \"aa5026b4-dfea-44dd-819c-bcb604815093\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-f45d57d9c-8zltg"
Sep 08 14:07:16.645814 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:07:16.645615    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aa5026b4-dfea-44dd-819c-bcb604815093-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-f45d57d9c-8zltg\" (UID: \"aa5026b4-dfea-44dd-819c-bcb604815093\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-f45d57d9c-8zltg"
Sep 08 14:07:16.645814 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:07:16.645690    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txznm\" (UniqueName: \"kubernetes.io/projected/aa5026b4-dfea-44dd-819c-bcb604815093-kube-api-access-txznm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-f45d57d9c-8zltg\" (UID: \"aa5026b4-dfea-44dd-819c-bcb604815093\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-f45d57d9c-8zltg"
Sep 08 14:07:16.735225 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:07:16.735166    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:07:16.746346 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:07:16.746312    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/aa5026b4-dfea-44dd-819c-bcb604815093-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-f45d57d9c-8zltg\" (UID: \"aa5026b4-dfea-44dd-819c-bcb604815093\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-f45d57d9c-8zltg"
Sep 08 14:07:16.746438 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:07:16.746360    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa5026b4-dfea-44dd-819c-bcb604815093-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-f45d57d9c-8zltg\" (UID: \"aa5026b4-dfea-44dd-819c-bcb604815093\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-f45d57d9c-8zltg"
Sep 08 14:07:16.746486 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:07:16.746471    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aa5026b4-dfea-44dd-819c-bcb604815093-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-f45d57d9c-8zltg\" (UID: \"aa5026b4-dfea-44dd-819c-bcb604815093\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-f45d57d9c-8zltg"
Sep 08 14:07:16.746558 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:07:16.746514    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txznm\" (UniqueName: \"kubernetes.io/projected/aa5026b4-dfea-44dd-819c-bcb604815093-kube-api-access-txznm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-f45d57d9c-8zltg\" (UID: \"aa5026b4-dfea-44dd-819c-bcb604815093\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-f45d57d9c-8zltg"
Sep 08 14:07:16.746619 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:07:16.746557    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa5026b4-dfea-44dd-819c-bcb604815093-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-f45d57d9c-8zltg\" (UID: \"aa5026b4-dfea-44dd-819c-bcb604815093\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-f45d57d9c-8zltg"
Sep 08 14:07:16.746619 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:07:16.746574    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa5026b4-dfea-44dd-819c-bcb604815093-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-f45d57d9c-8zltg\" (UID: \"aa5026b4-dfea-44dd-819c-bcb604815093\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-f45d57d9c-8zltg"
Sep 08 14:07:16.746619 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:07:16.746601    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa5026b4-dfea-44dd-819c-bcb604815093-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-f45d57d9c-8zltg\" (UID: \"aa5026b4-dfea-44dd-819c-bcb604815093\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-f45d57d9c-8zltg"
Sep 08 14:07:16.746784 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:07:16.746632    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa5026b4-dfea-44dd-819c-bcb604815093-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-f45d57d9c-8zltg\" (UID: \"aa5026b4-dfea-44dd-819c-bcb604815093\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-f45d57d9c-8zltg"
Sep 08 14:07:16.746784 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:07:16.746743    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa5026b4-dfea-44dd-819c-bcb604815093-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-f45d57d9c-8zltg\" (UID: \"aa5026b4-dfea-44dd-819c-bcb604815093\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-f45d57d9c-8zltg"
Sep 08 14:07:16.746894 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:07:16.746851    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aa5026b4-dfea-44dd-819c-bcb604815093-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-f45d57d9c-8zltg\" (UID: \"aa5026b4-dfea-44dd-819c-bcb604815093\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-f45d57d9c-8zltg"
Sep 08 14:07:16.747006 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:07:16.746984    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa5026b4-dfea-44dd-819c-bcb604815093-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-f45d57d9c-8zltg\" (UID: \"aa5026b4-dfea-44dd-819c-bcb604815093\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-f45d57d9c-8zltg"
Sep 08 14:07:16.747140 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:07:16.747116    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/aa5026b4-dfea-44dd-819c-bcb604815093-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-f45d57d9c-8zltg\" (UID: \"aa5026b4-dfea-44dd-819c-bcb604815093\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-f45d57d9c-8zltg"
Sep 08 14:07:16.747223 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:07:16.747207    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa5026b4-dfea-44dd-819c-bcb604815093-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-f45d57d9c-8zltg\" (UID: \"aa5026b4-dfea-44dd-819c-bcb604815093\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-f45d57d9c-8zltg"
Sep 08 14:07:16.748790 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:07:16.748738    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa5026b4-dfea-44dd-819c-bcb604815093-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-f45d57d9c-8zltg\" (UID: \"aa5026b4-dfea-44dd-819c-bcb604815093\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-f45d57d9c-8zltg"
Sep 08 14:07:16.749137 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:07:16.749117    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa5026b4-dfea-44dd-819c-bcb604815093-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-f45d57d9c-8zltg\" (UID: \"aa5026b4-dfea-44dd-819c-bcb604815093\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-f45d57d9c-8zltg"
Sep 08 14:07:16.755366 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:07:16.755342    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-txznm\" (UniqueName: \"kubernetes.io/projected/aa5026b4-dfea-44dd-819c-bcb604815093-kube-api-access-txznm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-f45d57d9c-8zltg\" (UID: \"aa5026b4-dfea-44dd-819c-bcb604815093\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-f45d57d9c-8zltg"
Sep 08 14:07:16.819304 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:07:16.819262    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-f45d57d9c-8zltg"
Sep 08 14:07:17.102311 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:07:17.102282    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-f45d57d9c-8zltg"]
Sep 08 14:07:17.104486 ip-10-0-132-254 kubenswrapper[2573]: W0908 14:07:17.104460    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa5026b4_dfea_44dd_819c_bcb604815093.slice/crio-a42f47c4f6146e6758c5de3c61ee787a5354f76bf71e450ba2e3b3173d3dc893 WatchSource:0}: Error finding container a42f47c4f6146e6758c5de3c61ee787a5354f76bf71e450ba2e3b3173d3dc893: Status 404 returned error can't find the container with id a42f47c4f6146e6758c5de3c61ee787a5354f76bf71e450ba2e3b3173d3dc893
Sep 08 14:07:17.581832 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:07:17.581794    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-f45d57d9c-8zltg" event={"ID":"aa5026b4-dfea-44dd-819c-bcb604815093","Type":"ContainerStarted","Data":"a559e9ab346aac54f5722b074ab99d4035ec49fbdc65f79152fdf876a829d4f1"}
Sep 08 14:07:17.582010 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:07:17.581852    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-f45d57d9c-8zltg" event={"ID":"aa5026b4-dfea-44dd-819c-bcb604815093","Type":"ContainerStarted","Data":"a42f47c4f6146e6758c5de3c61ee787a5354f76bf71e450ba2e3b3173d3dc893"}
Sep 08 14:07:20.594475 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:07:20.594390    2573 generic.go:356] "Generic (PLEG): container finished" podID="aa5026b4-dfea-44dd-819c-bcb604815093" containerID="a559e9ab346aac54f5722b074ab99d4035ec49fbdc65f79152fdf876a829d4f1" exitCode=0
Sep 08 14:07:20.594475 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:07:20.594460    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-f45d57d9c-8zltg" event={"ID":"aa5026b4-dfea-44dd-819c-bcb604815093","Type":"ContainerDied","Data":"a559e9ab346aac54f5722b074ab99d4035ec49fbdc65f79152fdf876a829d4f1"}
Sep 08 14:07:21.601249 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:07:21.601213    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-f45d57d9c-8zltg" event={"ID":"aa5026b4-dfea-44dd-819c-bcb604815093","Type":"ContainerStarted","Data":"06c00f92b97d0972d804f13cc7ba477fe51e80f3229de9be866adb114dd30f50"}
Sep 08 14:07:21.626659 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:07:21.626605    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-f45d57d9c-8zltg" podStartSLOduration=2.624374229 podStartE2EDuration="5.626585023s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.002210794s" isStatefulPod=false podCreationTimestamp="2026-09-08 14:07:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:07:21.624050199 +0000 UTC m=+3091.895498152" watchObservedRunningTime="2026-09-08 14:07:21.626585023 +0000 UTC m=+3091.898032957"
Sep 08 14:07:21.819940 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:07:21.819906    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-f45d57d9c-8zltg"
Sep 08 14:07:21.892619 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:07:21.892586    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:07:21.957698 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:07:21.957663    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:07:26.819639 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:07:26.819597    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-f45d57d9c-8zltg"
Sep 08 14:07:26.821205 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:07:26.821176    2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-f45d57d9c-8zltg" podUID="aa5026b4-dfea-44dd-819c-bcb604815093" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": dial tcp 10.133.0.43:8000: connect: connection refused"
Sep 08 14:07:30.451435 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:07:30.451395    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:07:36.820042 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:07:36.819989    2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-f45d57d9c-8zltg" podUID="aa5026b4-dfea-44dd-819c-bcb604815093" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": dial tcp 10.133.0.43:8000: connect: connection refused"
Sep 08 14:07:46.819938 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:07:46.819893    2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-f45d57d9c-8zltg" podUID="aa5026b4-dfea-44dd-819c-bcb604815093" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": dial tcp 10.133.0.43:8000: connect: connection refused"
Sep 08 14:07:56.819828 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:07:56.819793    2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-f45d57d9c-8zltg" podUID="aa5026b4-dfea-44dd-819c-bcb604815093" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": dial tcp 10.133.0.43:8000: connect: connection refused"
Sep 08 14:08:06.820120 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:08:06.820079    2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-f45d57d9c-8zltg" podUID="aa5026b4-dfea-44dd-819c-bcb604815093" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": dial tcp 10.133.0.43:8000: connect: connection refused"
Sep 08 14:08:16.820691 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:08:16.820601    2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-f45d57d9c-8zltg" podUID="aa5026b4-dfea-44dd-819c-bcb604815093" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": dial tcp 10.133.0.43:8000: connect: connection refused"
Sep 08 14:08:24.109853 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:08:24.109820    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:08:24.211372 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:08:24.211344    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:08:26.819992 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:08:26.819952    2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-f45d57d9c-8zltg" podUID="aa5026b4-dfea-44dd-819c-bcb604815093" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": dial tcp 10.133.0.43:8000: connect: connection refused"
Sep 08 14:08:33.273963 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:08:33.273931    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:08:33.458643 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:08:33.458612    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:08:36.829188 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:08:36.829161    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-f45d57d9c-8zltg"
Sep 08 14:08:36.836768 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:08:36.836737    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-f45d57d9c-8zltg"
Sep 08 14:08:38.366646 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:08:38.366616    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-f45d57d9c-8zltg"]
Sep 08 14:08:38.368143 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:08:38.368119    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:08:38.457552 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:08:38.457521    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:08:53.388951 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:08:53.388902    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-f45d57d9c-8zltg" podUID="aa5026b4-dfea-44dd-819c-bcb604815093" containerName="main" containerID="cri-o://06c00f92b97d0972d804f13cc7ba477fe51e80f3229de9be866adb114dd30f50" gracePeriod=45
Sep 08 14:08:53.830173 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:08:53.830133    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-f45d57d9c-8zltg" podUID="aa5026b4-dfea-44dd-819c-bcb604815093" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": dial tcp 10.133.0.43:8000: connect: connection refused"
Sep 08 14:08:53.934494 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:08:53.934469    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-f45d57d9c-8zltg"
Sep 08 14:08:53.981410 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:08:53.981373    2573 generic.go:356] "Generic (PLEG): container finished" podID="aa5026b4-dfea-44dd-819c-bcb604815093" containerID="06c00f92b97d0972d804f13cc7ba477fe51e80f3229de9be866adb114dd30f50" exitCode=0
Sep 08 14:08:53.981622 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:08:53.981417    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-f45d57d9c-8zltg"
Sep 08 14:08:53.981622 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:08:53.981438    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-f45d57d9c-8zltg" event={"ID":"aa5026b4-dfea-44dd-819c-bcb604815093","Type":"ContainerDied","Data":"06c00f92b97d0972d804f13cc7ba477fe51e80f3229de9be866adb114dd30f50"}
Sep 08 14:08:53.981622 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:08:53.981495    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-f45d57d9c-8zltg" event={"ID":"aa5026b4-dfea-44dd-819c-bcb604815093","Type":"ContainerDied","Data":"a42f47c4f6146e6758c5de3c61ee787a5354f76bf71e450ba2e3b3173d3dc893"}
Sep 08 14:08:53.981622 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:08:53.981549    2573 scope.go:117] "RemoveContainer" containerID="06c00f92b97d0972d804f13cc7ba477fe51e80f3229de9be866adb114dd30f50"
Sep 08 14:08:53.989316 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:08:53.989299    2573 scope.go:117] "RemoveContainer" containerID="a559e9ab346aac54f5722b074ab99d4035ec49fbdc65f79152fdf876a829d4f1"
Sep 08 14:08:53.996668 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:08:53.996651    2573 scope.go:117] "RemoveContainer" containerID="06c00f92b97d0972d804f13cc7ba477fe51e80f3229de9be866adb114dd30f50"
Sep 08 14:08:53.996934 ip-10-0-132-254 kubenswrapper[2573]: E0908 14:08:53.996920    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06c00f92b97d0972d804f13cc7ba477fe51e80f3229de9be866adb114dd30f50\": container with ID starting with 06c00f92b97d0972d804f13cc7ba477fe51e80f3229de9be866adb114dd30f50 not found: ID does not exist" containerID="06c00f92b97d0972d804f13cc7ba477fe51e80f3229de9be866adb114dd30f50"
Sep 08 14:08:53.996986 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:08:53.996943    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06c00f92b97d0972d804f13cc7ba477fe51e80f3229de9be866adb114dd30f50"} err="failed to get container status \"06c00f92b97d0972d804f13cc7ba477fe51e80f3229de9be866adb114dd30f50\": rpc error: code = NotFound desc = could not find container \"06c00f92b97d0972d804f13cc7ba477fe51e80f3229de9be866adb114dd30f50\": container with ID starting with 06c00f92b97d0972d804f13cc7ba477fe51e80f3229de9be866adb114dd30f50 not found: ID does not exist"
Sep 08 14:08:53.996986 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:08:53.996961    2573 scope.go:117] "RemoveContainer" containerID="a559e9ab346aac54f5722b074ab99d4035ec49fbdc65f79152fdf876a829d4f1"
Sep 08 14:08:53.997190 ip-10-0-132-254 kubenswrapper[2573]: E0908 14:08:53.997172    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a559e9ab346aac54f5722b074ab99d4035ec49fbdc65f79152fdf876a829d4f1\": container with ID starting with a559e9ab346aac54f5722b074ab99d4035ec49fbdc65f79152fdf876a829d4f1 not found: ID does not exist" containerID="a559e9ab346aac54f5722b074ab99d4035ec49fbdc65f79152fdf876a829d4f1"
Sep 08 14:08:53.997247 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:08:53.997195    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a559e9ab346aac54f5722b074ab99d4035ec49fbdc65f79152fdf876a829d4f1"} err="failed to get container status \"a559e9ab346aac54f5722b074ab99d4035ec49fbdc65f79152fdf876a829d4f1\": rpc error: code = NotFound desc = could not find container \"a559e9ab346aac54f5722b074ab99d4035ec49fbdc65f79152fdf876a829d4f1\": container with ID starting with a559e9ab346aac54f5722b074ab99d4035ec49fbdc65f79152fdf876a829d4f1 not found: ID does not exist"
Sep 08 14:08:54.050244 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:08:54.050189    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-txznm\" (UniqueName: \"kubernetes.io/projected/aa5026b4-dfea-44dd-819c-bcb604815093-kube-api-access-txznm\") pod \"aa5026b4-dfea-44dd-819c-bcb604815093\" (UID: \"aa5026b4-dfea-44dd-819c-bcb604815093\") "
Sep 08 14:08:54.050244 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:08:54.050223    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa5026b4-dfea-44dd-819c-bcb604815093-model-cache\") pod \"aa5026b4-dfea-44dd-819c-bcb604815093\" (UID: \"aa5026b4-dfea-44dd-819c-bcb604815093\") "
Sep 08 14:08:54.050383 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:08:54.050253    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa5026b4-dfea-44dd-819c-bcb604815093-tmp-dir\") pod \"aa5026b4-dfea-44dd-819c-bcb604815093\" (UID: \"aa5026b4-dfea-44dd-819c-bcb604815093\") "
Sep 08 14:08:54.050383 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:08:54.050291    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aa5026b4-dfea-44dd-819c-bcb604815093-kserve-provision-location\") pod \"aa5026b4-dfea-44dd-819c-bcb604815093\" (UID: \"aa5026b4-dfea-44dd-819c-bcb604815093\") "
Sep 08 14:08:54.050383 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:08:54.050332    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa5026b4-dfea-44dd-819c-bcb604815093-dshm\") pod \"aa5026b4-dfea-44dd-819c-bcb604815093\" (UID: \"aa5026b4-dfea-44dd-819c-bcb604815093\") "
Sep 08 14:08:54.050383 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:08:54.050356    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/aa5026b4-dfea-44dd-819c-bcb604815093-cabundle-cert\") pod \"aa5026b4-dfea-44dd-819c-bcb604815093\" (UID: \"aa5026b4-dfea-44dd-819c-bcb604815093\") "
Sep 08 14:08:54.050383 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:08:54.050384    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa5026b4-dfea-44dd-819c-bcb604815093-home\") pod \"aa5026b4-dfea-44dd-819c-bcb604815093\" (UID: \"aa5026b4-dfea-44dd-819c-bcb604815093\") "
Sep 08 14:08:54.050720 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:08:54.050419    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa5026b4-dfea-44dd-819c-bcb604815093-tls-certs\") pod \"aa5026b4-dfea-44dd-819c-bcb604815093\" (UID: \"aa5026b4-dfea-44dd-819c-bcb604815093\") "
Sep 08 14:08:54.050720 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:08:54.050577    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa5026b4-dfea-44dd-819c-bcb604815093-model-cache" (OuterVolumeSpecName: "model-cache") pod "aa5026b4-dfea-44dd-819c-bcb604815093" (UID: "aa5026b4-dfea-44dd-819c-bcb604815093"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:08:54.050847 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:08:54.050777    2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa5026b4-dfea-44dd-819c-bcb604815093-model-cache\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 14:08:54.050916 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:08:54.050892    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa5026b4-dfea-44dd-819c-bcb604815093-home" (OuterVolumeSpecName: "home") pod "aa5026b4-dfea-44dd-819c-bcb604815093" (UID: "aa5026b4-dfea-44dd-819c-bcb604815093"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:08:54.050977 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:08:54.050942    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa5026b4-dfea-44dd-819c-bcb604815093-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "aa5026b4-dfea-44dd-819c-bcb604815093" (UID: "aa5026b4-dfea-44dd-819c-bcb604815093"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 14:08:54.051035 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:08:54.050977    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa5026b4-dfea-44dd-819c-bcb604815093-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "aa5026b4-dfea-44dd-819c-bcb604815093" (UID: "aa5026b4-dfea-44dd-819c-bcb604815093"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:08:54.052701 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:08:54.052675    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa5026b4-dfea-44dd-819c-bcb604815093-dshm" (OuterVolumeSpecName: "dshm") pod "aa5026b4-dfea-44dd-819c-bcb604815093" (UID: "aa5026b4-dfea-44dd-819c-bcb604815093"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:08:54.052916 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:08:54.052884    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa5026b4-dfea-44dd-819c-bcb604815093-kube-api-access-txznm" (OuterVolumeSpecName: "kube-api-access-txznm") pod "aa5026b4-dfea-44dd-819c-bcb604815093" (UID: "aa5026b4-dfea-44dd-819c-bcb604815093"). InnerVolumeSpecName "kube-api-access-txznm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 14:08:54.052916 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:08:54.052895    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa5026b4-dfea-44dd-819c-bcb604815093-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "aa5026b4-dfea-44dd-819c-bcb604815093" (UID: "aa5026b4-dfea-44dd-819c-bcb604815093"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 14:08:54.072131 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:08:54.072107    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa5026b4-dfea-44dd-819c-bcb604815093-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "aa5026b4-dfea-44dd-819c-bcb604815093" (UID: "aa5026b4-dfea-44dd-819c-bcb604815093"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:08:54.151308 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:08:54.151265    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txznm\" (UniqueName: \"kubernetes.io/projected/aa5026b4-dfea-44dd-819c-bcb604815093-kube-api-access-txznm\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 14:08:54.151308 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:08:54.151302    2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa5026b4-dfea-44dd-819c-bcb604815093-tmp-dir\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 14:08:54.151308 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:08:54.151313    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aa5026b4-dfea-44dd-819c-bcb604815093-kserve-provision-location\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 14:08:54.151563 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:08:54.151321    2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa5026b4-dfea-44dd-819c-bcb604815093-dshm\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 14:08:54.151563 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:08:54.151330    2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/aa5026b4-dfea-44dd-819c-bcb604815093-cabundle-cert\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 14:08:54.151563 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:08:54.151338    2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa5026b4-dfea-44dd-819c-bcb604815093-home\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 14:08:54.151563 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:08:54.151346    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa5026b4-dfea-44dd-819c-bcb604815093-tls-certs\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 14:08:54.297574 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:08:54.297542    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-f45d57d9c-8zltg"]
Sep 08 14:08:54.301975 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:08:54.301915    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-f45d57d9c-8zltg"]
Sep 08 14:08:56.174913 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:08:56.174877    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aa5026b4-dfea-44dd-819c-bcb604815093" path="/var/lib/kubelet/pods/aa5026b4-dfea-44dd-819c-bcb604815093/volumes"
Sep 08 14:09:09.722376 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:09:09.722344    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:09:09.817904 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:09:09.817872    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:09:16.298048 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:09:16.298016    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:09:22.387520 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:09:22.387479    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:09:22.490162 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:09:22.490108    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:09:28.424117 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:09:28.424086    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:09:28.579651 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:09:28.579619    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:09:28.738736 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:09:28.738651    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:09:30.764696 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:09:30.764664    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:09:30.878938 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:09:30.878905    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:09:30.990144 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:09:30.990116    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:09:35.356969 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:09:35.356937    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:09:37.078986 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:09:37.078953    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:09:37.161787 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:09:37.161753    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:09:37.771206 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:09:37.771177    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:09:37.965966 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:09:37.965932    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:09:38.054281 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:09:38.054212    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:09:38.156994 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:09:38.156962    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:09:41.708297 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:09:41.708265    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:09:41.811013 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:09:41.810988    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:09:42.765733 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:09:42.765654    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:09:42.867600 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:09:42.867571    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:09:42.972314 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:09:42.972285    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:09:44.783634 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:09:44.783601    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:09:44.908424 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:09:44.908393    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:09:47.751113 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:09:47.751079    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:09:48.000235 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:09:48.000206    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:09:48.100842 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:09:48.100765    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:09:50.285744 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:09:50.285706    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:09:50.402463 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:09:50.402437    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:09:53.028335 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:09:53.028302    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:09:53.151996 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:09:53.151953    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:09:55.104150 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:09:55.104116    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:09:55.203160 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:09:55.203129    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:09:57.938820 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:09:57.938793    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:09:59.544161 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:09:59.544129    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:10:00.910648 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:10:00.910620    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:10:05.427241 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:10:05.427209    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-895d8956b-4rjdm"]
Sep 08 14:10:05.427632 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:10:05.427518    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa5026b4-dfea-44dd-819c-bcb604815093" containerName="main"
Sep 08 14:10:05.427632 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:10:05.427531    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa5026b4-dfea-44dd-819c-bcb604815093" containerName="main"
Sep 08 14:10:05.427632 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:10:05.427549    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa5026b4-dfea-44dd-819c-bcb604815093" containerName="storage-initializer"
Sep 08 14:10:05.427632 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:10:05.427555    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa5026b4-dfea-44dd-819c-bcb604815093" containerName="storage-initializer"
Sep 08 14:10:05.427632 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:10:05.427612    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa5026b4-dfea-44dd-819c-bcb604815093" containerName="main"
Sep 08 14:10:05.430637 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:10:05.430613    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-895d8956b-4rjdm"
Sep 08 14:10:05.433897 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:10:05.433879    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"seaweedfs-s3-creds\""
Sep 08 14:10:05.434027 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:10:05.433914    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"default-dockercfg-jmqm4\""
Sep 08 14:10:05.434242 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:10:05.434223    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-kserve-self-signed-certs\""
Sep 08 14:10:05.434299 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:10:05.434269    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"kube-root-ca.crt\""
Sep 08 14:10:05.434729 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:10:05.434698    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"odh-kserve-custom-ca-bundle\""
Sep 08 14:10:05.434857 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:10:05.434736    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"openshift-service-ca.crt\""
Sep 08 14:10:05.447357 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:10:05.447334    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-895d8956b-4rjdm"]
Sep 08 14:10:05.463168 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:10:05.463145    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:10:05.467263 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:10:05.467235    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/de693a35-1091-423d-a5e0-78e0bd93bb69-kserve-provision-location\") pod \"stop-feature-test-kserve-895d8956b-4rjdm\" (UID: \"de693a35-1091-423d-a5e0-78e0bd93bb69\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-895d8956b-4rjdm"
Sep 08 14:10:05.467354 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:10:05.467262    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/de693a35-1091-423d-a5e0-78e0bd93bb69-cabundle-cert\") pod \"stop-feature-test-kserve-895d8956b-4rjdm\" (UID: \"de693a35-1091-423d-a5e0-78e0bd93bb69\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-895d8956b-4rjdm"
Sep 08 14:10:05.467354 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:10:05.467330    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxjfv\" (UniqueName: \"kubernetes.io/projected/de693a35-1091-423d-a5e0-78e0bd93bb69-kube-api-access-cxjfv\") pod \"stop-feature-test-kserve-895d8956b-4rjdm\" (UID: \"de693a35-1091-423d-a5e0-78e0bd93bb69\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-895d8956b-4rjdm"
Sep 08 14:10:05.467452 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:10:05.467372    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de693a35-1091-423d-a5e0-78e0bd93bb69-model-cache\") pod \"stop-feature-test-kserve-895d8956b-4rjdm\" (UID: \"de693a35-1091-423d-a5e0-78e0bd93bb69\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-895d8956b-4rjdm"
Sep 08 14:10:05.467452 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:10:05.467404    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de693a35-1091-423d-a5e0-78e0bd93bb69-dshm\") pod \"stop-feature-test-kserve-895d8956b-4rjdm\" (UID: \"de693a35-1091-423d-a5e0-78e0bd93bb69\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-895d8956b-4rjdm"
Sep 08 14:10:05.467452 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:10:05.467438    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de693a35-1091-423d-a5e0-78e0bd93bb69-home\") pod \"stop-feature-test-kserve-895d8956b-4rjdm\" (UID: \"de693a35-1091-423d-a5e0-78e0bd93bb69\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-895d8956b-4rjdm"
Sep 08 14:10:05.467574 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:10:05.467462    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de693a35-1091-423d-a5e0-78e0bd93bb69-tmp-dir\") pod \"stop-feature-test-kserve-895d8956b-4rjdm\" (UID: \"de693a35-1091-423d-a5e0-78e0bd93bb69\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-895d8956b-4rjdm"
Sep 08 14:10:05.467574 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:10:05.467492    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de693a35-1091-423d-a5e0-78e0bd93bb69-tls-certs\") pod \"stop-feature-test-kserve-895d8956b-4rjdm\" (UID: \"de693a35-1091-423d-a5e0-78e0bd93bb69\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-895d8956b-4rjdm"
Sep 08 14:10:05.567779 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:10:05.567758    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/de693a35-1091-423d-a5e0-78e0bd93bb69-cabundle-cert\") pod \"stop-feature-test-kserve-895d8956b-4rjdm\" (UID: \"de693a35-1091-423d-a5e0-78e0bd93bb69\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-895d8956b-4rjdm"
Sep 08 14:10:05.567874 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:10:05.567795    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxjfv\" (UniqueName: \"kubernetes.io/projected/de693a35-1091-423d-a5e0-78e0bd93bb69-kube-api-access-cxjfv\") pod \"stop-feature-test-kserve-895d8956b-4rjdm\" (UID: \"de693a35-1091-423d-a5e0-78e0bd93bb69\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-895d8956b-4rjdm"
Sep 08 14:10:05.567874 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:10:05.567811    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de693a35-1091-423d-a5e0-78e0bd93bb69-model-cache\") pod \"stop-feature-test-kserve-895d8956b-4rjdm\" (UID: \"de693a35-1091-423d-a5e0-78e0bd93bb69\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-895d8956b-4rjdm"
Sep 08 14:10:05.567874 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:10:05.567830    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de693a35-1091-423d-a5e0-78e0bd93bb69-dshm\") pod \"stop-feature-test-kserve-895d8956b-4rjdm\" (UID: \"de693a35-1091-423d-a5e0-78e0bd93bb69\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-895d8956b-4rjdm"
Sep 08 14:10:05.567874 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:10:05.567857    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de693a35-1091-423d-a5e0-78e0bd93bb69-home\") pod \"stop-feature-test-kserve-895d8956b-4rjdm\" (UID: \"de693a35-1091-423d-a5e0-78e0bd93bb69\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-895d8956b-4rjdm"
Sep 08 14:10:05.568075 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:10:05.567890    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de693a35-1091-423d-a5e0-78e0bd93bb69-tmp-dir\") pod \"stop-feature-test-kserve-895d8956b-4rjdm\" (UID: \"de693a35-1091-423d-a5e0-78e0bd93bb69\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-895d8956b-4rjdm"
Sep 08 14:10:05.568075 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:10:05.567921    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de693a35-1091-423d-a5e0-78e0bd93bb69-tls-certs\") pod \"stop-feature-test-kserve-895d8956b-4rjdm\" (UID: \"de693a35-1091-423d-a5e0-78e0bd93bb69\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-895d8956b-4rjdm"
Sep 08 14:10:05.568075 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:10:05.567969    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/de693a35-1091-423d-a5e0-78e0bd93bb69-kserve-provision-location\") pod \"stop-feature-test-kserve-895d8956b-4rjdm\" (UID: \"de693a35-1091-423d-a5e0-78e0bd93bb69\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-895d8956b-4rjdm"
Sep 08 14:10:05.568235 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:10:05.568217    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de693a35-1091-423d-a5e0-78e0bd93bb69-model-cache\") pod \"stop-feature-test-kserve-895d8956b-4rjdm\" (UID: \"de693a35-1091-423d-a5e0-78e0bd93bb69\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-895d8956b-4rjdm"
Sep 08 14:10:05.568386 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:10:05.568364    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/de693a35-1091-423d-a5e0-78e0bd93bb69-kserve-provision-location\") pod \"stop-feature-test-kserve-895d8956b-4rjdm\" (UID: \"de693a35-1091-423d-a5e0-78e0bd93bb69\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-895d8956b-4rjdm"
Sep 08 14:10:05.568461 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:10:05.568422    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de693a35-1091-423d-a5e0-78e0bd93bb69-home\") pod \"stop-feature-test-kserve-895d8956b-4rjdm\" (UID: \"de693a35-1091-423d-a5e0-78e0bd93bb69\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-895d8956b-4rjdm"
Sep 08 14:10:05.568544 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:10:05.568481    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de693a35-1091-423d-a5e0-78e0bd93bb69-tmp-dir\") pod \"stop-feature-test-kserve-895d8956b-4rjdm\" (UID: \"de693a35-1091-423d-a5e0-78e0bd93bb69\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-895d8956b-4rjdm"
Sep 08 14:10:05.568624 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:10:05.568605    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/de693a35-1091-423d-a5e0-78e0bd93bb69-cabundle-cert\") pod \"stop-feature-test-kserve-895d8956b-4rjdm\" (UID: \"de693a35-1091-423d-a5e0-78e0bd93bb69\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-895d8956b-4rjdm"
Sep 08 14:10:05.570041 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:10:05.570021    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de693a35-1091-423d-a5e0-78e0bd93bb69-dshm\") pod \"stop-feature-test-kserve-895d8956b-4rjdm\" (UID: \"de693a35-1091-423d-a5e0-78e0bd93bb69\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-895d8956b-4rjdm"
Sep 08 14:10:05.570440 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:10:05.570421    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de693a35-1091-423d-a5e0-78e0bd93bb69-tls-certs\") pod \"stop-feature-test-kserve-895d8956b-4rjdm\" (UID: \"de693a35-1091-423d-a5e0-78e0bd93bb69\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-895d8956b-4rjdm"
Sep 08 14:10:05.577282 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:10:05.577260    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxjfv\" (UniqueName: \"kubernetes.io/projected/de693a35-1091-423d-a5e0-78e0bd93bb69-kube-api-access-cxjfv\") pod \"stop-feature-test-kserve-895d8956b-4rjdm\" (UID: \"de693a35-1091-423d-a5e0-78e0bd93bb69\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-895d8956b-4rjdm"
Sep 08 14:10:05.682867 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:10:05.682788    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:10:05.741240 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:10:05.741215    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-895d8956b-4rjdm"
Sep 08 14:10:06.020901 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:10:06.020874    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-895d8956b-4rjdm"]
Sep 08 14:10:06.023176 ip-10-0-132-254 kubenswrapper[2573]: W0908 14:10:06.023148    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde693a35_1091_423d_a5e0_78e0bd93bb69.slice/crio-eb23591f7c49c2b10dc3a2a2ab99f4467c7ef2f4089634b851c1f903617e13d3 WatchSource:0}: Error finding container eb23591f7c49c2b10dc3a2a2ab99f4467c7ef2f4089634b851c1f903617e13d3: Status 404 returned error can't find the container with id eb23591f7c49c2b10dc3a2a2ab99f4467c7ef2f4089634b851c1f903617e13d3
Sep 08 14:10:06.253537 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:10:06.253491    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-895d8956b-4rjdm" event={"ID":"de693a35-1091-423d-a5e0-78e0bd93bb69","Type":"ContainerStarted","Data":"fd69f04ff10f4e58c791627078d3a604855ee76fc32669fcae613c4f81bc0fb5"}
Sep 08 14:10:06.253723 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:10:06.253551    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-895d8956b-4rjdm" event={"ID":"de693a35-1091-423d-a5e0-78e0bd93bb69","Type":"ContainerStarted","Data":"eb23591f7c49c2b10dc3a2a2ab99f4467c7ef2f4089634b851c1f903617e13d3"}
Sep 08 14:10:08.263817 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:10:08.263785    2573 generic.go:356] "Generic (PLEG): container finished" podID="de693a35-1091-423d-a5e0-78e0bd93bb69" containerID="fd69f04ff10f4e58c791627078d3a604855ee76fc32669fcae613c4f81bc0fb5" exitCode=0
Sep 08 14:10:08.264141 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:10:08.263856    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-895d8956b-4rjdm" event={"ID":"de693a35-1091-423d-a5e0-78e0bd93bb69","Type":"ContainerDied","Data":"fd69f04ff10f4e58c791627078d3a604855ee76fc32669fcae613c4f81bc0fb5"}
Sep 08 14:10:09.270152 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:10:09.270114    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-895d8956b-4rjdm" event={"ID":"de693a35-1091-423d-a5e0-78e0bd93bb69","Type":"ContainerStarted","Data":"c6221bbfc7fc5e25b09a04f12facbb6567453e910cd05c5a079e1467f04bf0f0"}
Sep 08 14:10:09.294999 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:10:09.294912    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-895d8956b-4rjdm" podStartSLOduration=2.236263386 podStartE2EDuration="4.294892322s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.058628936s" isStatefulPod=false podCreationTimestamp="2026-09-08 14:10:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:10:09.292682277 +0000 UTC m=+3259.564130221" watchObservedRunningTime="2026-09-08 14:10:09.294892322 +0000 UTC m=+3259.566340254"
Sep 08 14:10:09.741702 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:10:09.741665    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-895d8956b-4rjdm"
Sep 08 14:10:15.742024 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:10:15.741985    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-895d8956b-4rjdm"
Sep 08 14:10:15.743577 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:10:15.743524    2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-895d8956b-4rjdm" podUID="de693a35-1091-423d-a5e0-78e0bd93bb69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": dial tcp 10.133.0.44:8000: connect: connection refused"
Sep 08 14:10:18.246355 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:10:18.246321    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:10:25.741925 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:10:25.741878    2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-895d8956b-4rjdm" podUID="de693a35-1091-423d-a5e0-78e0bd93bb69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": dial tcp 10.133.0.44:8000: connect: connection refused"
Sep 08 14:10:35.741960 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:10:35.741913    2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-895d8956b-4rjdm" podUID="de693a35-1091-423d-a5e0-78e0bd93bb69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": dial tcp 10.133.0.44:8000: connect: connection refused"
Sep 08 14:10:45.742168 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:10:45.742119    2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-895d8956b-4rjdm" podUID="de693a35-1091-423d-a5e0-78e0bd93bb69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": dial tcp 10.133.0.44:8000: connect: connection refused"
Sep 08 14:10:50.377708 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:10:50.377680    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8h8cn_b936238e-bdf1-4dac-8bf8-6417c313597c/ovn-acl-logging/0.log"
Sep 08 14:10:50.386028 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:10:50.386005    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8h8cn_b936238e-bdf1-4dac-8bf8-6417c313597c/ovn-acl-logging/0.log"
Sep 08 14:10:55.742737 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:10:55.742693    2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-895d8956b-4rjdm" podUID="de693a35-1091-423d-a5e0-78e0bd93bb69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": dial tcp 10.133.0.44:8000: connect: connection refused"
Sep 08 14:11:05.742363 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:11:05.742319    2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-895d8956b-4rjdm" podUID="de693a35-1091-423d-a5e0-78e0bd93bb69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": dial tcp 10.133.0.44:8000: connect: connection refused"
Sep 08 14:11:15.741969 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:11:15.741873    2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-895d8956b-4rjdm" podUID="de693a35-1091-423d-a5e0-78e0bd93bb69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": dial tcp 10.133.0.44:8000: connect: connection refused"
Sep 08 14:11:25.742647 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:11:25.742593    2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-895d8956b-4rjdm" podUID="de693a35-1091-423d-a5e0-78e0bd93bb69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": dial tcp 10.133.0.44:8000: connect: connection refused"
Sep 08 14:11:35.742527 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:11:35.742461    2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-895d8956b-4rjdm" podUID="de693a35-1091-423d-a5e0-78e0bd93bb69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": dial tcp 10.133.0.44:8000: connect: connection refused"
Sep 08 14:11:42.531741 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:11:42.531700    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:11:42.625941 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:11:42.625899    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:11:45.742688 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:11:45.742647    2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-895d8956b-4rjdm" podUID="de693a35-1091-423d-a5e0-78e0bd93bb69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": dial tcp 10.133.0.44:8000: connect: connection refused"
Sep 08 14:11:55.741935 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:11:55.741885    2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-895d8956b-4rjdm" podUID="de693a35-1091-423d-a5e0-78e0bd93bb69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": dial tcp 10.133.0.44:8000: connect: connection refused"
Sep 08 14:12:05.751489 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:12:05.751453    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-895d8956b-4rjdm"
Sep 08 14:12:05.759454 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:12:05.759428    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-895d8956b-4rjdm"
Sep 08 14:12:07.423302 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:12:07.423270    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-895d8956b-4rjdm"]
Sep 08 14:12:07.450925 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:12:07.450898    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:12:07.519434 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:12:07.519401    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:12:07.656548 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:12:07.656494    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:12:15.319569 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:12:15.319536    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:12:22.763141 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:12:22.763109    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:12:22.778949 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:12:22.778918    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-895d8956b-4rjdm" podUID="de693a35-1091-423d-a5e0-78e0bd93bb69" containerName="main" containerID="cri-o://c6221bbfc7fc5e25b09a04f12facbb6567453e910cd05c5a079e1467f04bf0f0" gracePeriod=45
Sep 08 14:12:22.900428 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:12:22.900393    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-6zd7n"]
Sep 08 14:12:22.903862 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:12:22.903845    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-6zd7n"
Sep 08 14:12:22.906135 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:12:22.906109    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-epp-sa-dockercfg-ltqjx\""
Sep 08 14:12:22.916825 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:12:22.916804    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-6zd7n"]
Sep 08 14:12:22.944900 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:12:22.944865    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:12:22.991915 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:12:22.991885    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a2d4516-36b8-4301-a85d-2ab9d8824b5c-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-6zd7n\" (UID: \"4a2d4516-36b8-4301-a85d-2ab9d8824b5c\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-6zd7n"
Sep 08 14:12:22.992087 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:12:22.991943    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npq5s\" (UniqueName: \"kubernetes.io/projected/4a2d4516-36b8-4301-a85d-2ab9d8824b5c-kube-api-access-npq5s\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-6zd7n\" (UID: \"4a2d4516-36b8-4301-a85d-2ab9d8824b5c\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-6zd7n"
Sep 08 14:12:23.092769 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:12:23.092673    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a2d4516-36b8-4301-a85d-2ab9d8824b5c-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-6zd7n\" (UID: \"4a2d4516-36b8-4301-a85d-2ab9d8824b5c\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-6zd7n"
Sep 08 14:12:23.092769 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:12:23.092730    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npq5s\" (UniqueName: \"kubernetes.io/projected/4a2d4516-36b8-4301-a85d-2ab9d8824b5c-kube-api-access-npq5s\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-6zd7n\" (UID: \"4a2d4516-36b8-4301-a85d-2ab9d8824b5c\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-6zd7n"
Sep 08 14:12:23.095230 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:12:23.095203    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a2d4516-36b8-4301-a85d-2ab9d8824b5c-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-6zd7n\" (UID: \"4a2d4516-36b8-4301-a85d-2ab9d8824b5c\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-6zd7n"
Sep 08 14:12:23.101356 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:12:23.101329    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-npq5s\" (UniqueName: \"kubernetes.io/projected/4a2d4516-36b8-4301-a85d-2ab9d8824b5c-kube-api-access-npq5s\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-6zd7n\" (UID: \"4a2d4516-36b8-4301-a85d-2ab9d8824b5c\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-6zd7n"
Sep 08 14:12:23.215689 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:12:23.215655    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-6zd7n"
Sep 08 14:12:23.486757 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:12:23.486727    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-6zd7n"]
Sep 08 14:12:23.488875 ip-10-0-132-254 kubenswrapper[2573]: W0908 14:12:23.488847    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a2d4516_36b8_4301_a85d_2ab9d8824b5c.slice/crio-3a041f41f139609fd49c2fe8c56474924fcb9993bf8925d3512b7a48bf7578c2 WatchSource:0}: Error finding container 3a041f41f139609fd49c2fe8c56474924fcb9993bf8925d3512b7a48bf7578c2: Status 404 returned error can't find the container with id 3a041f41f139609fd49c2fe8c56474924fcb9993bf8925d3512b7a48bf7578c2
Sep 08 14:12:23.541330 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:12:23.541310    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-895d8956b-4rjdm"
Sep 08 14:12:23.697698 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:12:23.697608    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de693a35-1091-423d-a5e0-78e0bd93bb69-tls-certs\") pod \"de693a35-1091-423d-a5e0-78e0bd93bb69\" (UID: \"de693a35-1091-423d-a5e0-78e0bd93bb69\") "
Sep 08 14:12:23.697698 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:12:23.697663    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de693a35-1091-423d-a5e0-78e0bd93bb69-dshm\") pod \"de693a35-1091-423d-a5e0-78e0bd93bb69\" (UID: \"de693a35-1091-423d-a5e0-78e0bd93bb69\") "
Sep 08 14:12:23.697698 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:12:23.697697    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/de693a35-1091-423d-a5e0-78e0bd93bb69-cabundle-cert\") pod \"de693a35-1091-423d-a5e0-78e0bd93bb69\" (UID: \"de693a35-1091-423d-a5e0-78e0bd93bb69\") "
Sep 08 14:12:23.697992 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:12:23.697854    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cxjfv\" (UniqueName: \"kubernetes.io/projected/de693a35-1091-423d-a5e0-78e0bd93bb69-kube-api-access-cxjfv\") pod \"de693a35-1091-423d-a5e0-78e0bd93bb69\" (UID: \"de693a35-1091-423d-a5e0-78e0bd93bb69\") "
Sep 08 14:12:23.697992 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:12:23.697916    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/de693a35-1091-423d-a5e0-78e0bd93bb69-kserve-provision-location\") pod \"de693a35-1091-423d-a5e0-78e0bd93bb69\" (UID: \"de693a35-1091-423d-a5e0-78e0bd93bb69\") "
Sep 08 14:12:23.697992 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:12:23.697973    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de693a35-1091-423d-a5e0-78e0bd93bb69-model-cache\") pod \"de693a35-1091-423d-a5e0-78e0bd93bb69\" (UID: \"de693a35-1091-423d-a5e0-78e0bd93bb69\") "
Sep 08 14:12:23.698161 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:12:23.697996    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de693a35-1091-423d-a5e0-78e0bd93bb69-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "de693a35-1091-423d-a5e0-78e0bd93bb69" (UID: "de693a35-1091-423d-a5e0-78e0bd93bb69"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 14:12:23.698161 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:12:23.698053    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de693a35-1091-423d-a5e0-78e0bd93bb69-home\") pod \"de693a35-1091-423d-a5e0-78e0bd93bb69\" (UID: \"de693a35-1091-423d-a5e0-78e0bd93bb69\") "
Sep 08 14:12:23.698161 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:12:23.698104    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de693a35-1091-423d-a5e0-78e0bd93bb69-tmp-dir\") pod \"de693a35-1091-423d-a5e0-78e0bd93bb69\" (UID: \"de693a35-1091-423d-a5e0-78e0bd93bb69\") "
Sep 08 14:12:23.698322 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:12:23.698254    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de693a35-1091-423d-a5e0-78e0bd93bb69-model-cache" (OuterVolumeSpecName: "model-cache") pod "de693a35-1091-423d-a5e0-78e0bd93bb69" (UID: "de693a35-1091-423d-a5e0-78e0bd93bb69"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:12:23.698448 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:12:23.698427    2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/de693a35-1091-423d-a5e0-78e0bd93bb69-cabundle-cert\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 14:12:23.698540 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:12:23.698453    2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de693a35-1091-423d-a5e0-78e0bd93bb69-model-cache\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 14:12:23.699105 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:12:23.699075    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de693a35-1091-423d-a5e0-78e0bd93bb69-home" (OuterVolumeSpecName: "home") pod "de693a35-1091-423d-a5e0-78e0bd93bb69" (UID: "de693a35-1091-423d-a5e0-78e0bd93bb69"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:12:23.700048 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:12:23.700012    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de693a35-1091-423d-a5e0-78e0bd93bb69-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "de693a35-1091-423d-a5e0-78e0bd93bb69" (UID: "de693a35-1091-423d-a5e0-78e0bd93bb69"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 14:12:23.700147 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:12:23.700082    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de693a35-1091-423d-a5e0-78e0bd93bb69-dshm" (OuterVolumeSpecName: "dshm") pod "de693a35-1091-423d-a5e0-78e0bd93bb69" (UID: "de693a35-1091-423d-a5e0-78e0bd93bb69"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:12:23.700147 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:12:23.700132    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de693a35-1091-423d-a5e0-78e0bd93bb69-kube-api-access-cxjfv" (OuterVolumeSpecName: "kube-api-access-cxjfv") pod "de693a35-1091-423d-a5e0-78e0bd93bb69" (UID: "de693a35-1091-423d-a5e0-78e0bd93bb69"). InnerVolumeSpecName "kube-api-access-cxjfv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 14:12:23.710412 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:12:23.710383    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de693a35-1091-423d-a5e0-78e0bd93bb69-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "de693a35-1091-423d-a5e0-78e0bd93bb69" (UID: "de693a35-1091-423d-a5e0-78e0bd93bb69"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:12:23.720080 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:12:23.720057    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de693a35-1091-423d-a5e0-78e0bd93bb69-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "de693a35-1091-423d-a5e0-78e0bd93bb69" (UID: "de693a35-1091-423d-a5e0-78e0bd93bb69"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:12:23.799131 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:12:23.799109    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de693a35-1091-423d-a5e0-78e0bd93bb69-tls-certs\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 14:12:23.799131 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:12:23.799130    2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de693a35-1091-423d-a5e0-78e0bd93bb69-dshm\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 14:12:23.799444 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:12:23.799138    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxjfv\" (UniqueName: \"kubernetes.io/projected/de693a35-1091-423d-a5e0-78e0bd93bb69-kube-api-access-cxjfv\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 14:12:23.799444 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:12:23.799149    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/de693a35-1091-423d-a5e0-78e0bd93bb69-kserve-provision-location\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 14:12:23.799444 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:12:23.799157    2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de693a35-1091-423d-a5e0-78e0bd93bb69-home\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 14:12:23.799444 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:12:23.799166    2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de693a35-1091-423d-a5e0-78e0bd93bb69-tmp-dir\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 14:12:23.821285 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:12:23.821259    2573 generic.go:356] "Generic (PLEG): container finished" podID="de693a35-1091-423d-a5e0-78e0bd93bb69" containerID="c6221bbfc7fc5e25b09a04f12facbb6567453e910cd05c5a079e1467f04bf0f0" exitCode=0
Sep 08 14:12:23.821374 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:12:23.821327    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-895d8956b-4rjdm" event={"ID":"de693a35-1091-423d-a5e0-78e0bd93bb69","Type":"ContainerDied","Data":"c6221bbfc7fc5e25b09a04f12facbb6567453e910cd05c5a079e1467f04bf0f0"}
Sep 08 14:12:23.821429 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:12:23.821412    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-895d8956b-4rjdm" event={"ID":"de693a35-1091-423d-a5e0-78e0bd93bb69","Type":"ContainerDied","Data":"eb23591f7c49c2b10dc3a2a2ab99f4467c7ef2f4089634b851c1f903617e13d3"}
Sep 08 14:12:23.821481 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:12:23.821339    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-895d8956b-4rjdm"
Sep 08 14:12:23.821481 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:12:23.821440    2573 scope.go:117] "RemoveContainer" containerID="c6221bbfc7fc5e25b09a04f12facbb6567453e910cd05c5a079e1467f04bf0f0"
Sep 08 14:12:23.823055 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:12:23.823031    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-6zd7n" event={"ID":"4a2d4516-36b8-4301-a85d-2ab9d8824b5c","Type":"ContainerStarted","Data":"ff49266ca6213c83d4018dd24f6c52689d94f41cf81b1d353416e1165ddf2f70"}
Sep 08 14:12:23.823158 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:12:23.823086    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-6zd7n" event={"ID":"4a2d4516-36b8-4301-a85d-2ab9d8824b5c","Type":"ContainerStarted","Data":"3a041f41f139609fd49c2fe8c56474924fcb9993bf8925d3512b7a48bf7578c2"}
Sep 08 14:12:23.823158 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:12:23.823119    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-6zd7n"
Sep 08 14:12:23.830395 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:12:23.830378    2573 scope.go:117] "RemoveContainer" containerID="fd69f04ff10f4e58c791627078d3a604855ee76fc32669fcae613c4f81bc0fb5"
Sep 08 14:12:23.837373 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:12:23.837356    2573 scope.go:117] "RemoveContainer" containerID="c6221bbfc7fc5e25b09a04f12facbb6567453e910cd05c5a079e1467f04bf0f0"
Sep 08 14:12:23.837681 ip-10-0-132-254 kubenswrapper[2573]: E0908 14:12:23.837656    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6221bbfc7fc5e25b09a04f12facbb6567453e910cd05c5a079e1467f04bf0f0\": container with ID starting with c6221bbfc7fc5e25b09a04f12facbb6567453e910cd05c5a079e1467f04bf0f0 not found: ID does not exist" containerID="c6221bbfc7fc5e25b09a04f12facbb6567453e910cd05c5a079e1467f04bf0f0"
Sep 08 14:12:23.837738 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:12:23.837681    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6221bbfc7fc5e25b09a04f12facbb6567453e910cd05c5a079e1467f04bf0f0"} err="failed to get container status \"c6221bbfc7fc5e25b09a04f12facbb6567453e910cd05c5a079e1467f04bf0f0\": rpc error: code = NotFound desc = could not find container \"c6221bbfc7fc5e25b09a04f12facbb6567453e910cd05c5a079e1467f04bf0f0\": container with ID starting with c6221bbfc7fc5e25b09a04f12facbb6567453e910cd05c5a079e1467f04bf0f0 not found: ID does not exist"
Sep 08 14:12:23.837738 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:12:23.837695    2573 scope.go:117] "RemoveContainer" containerID="fd69f04ff10f4e58c791627078d3a604855ee76fc32669fcae613c4f81bc0fb5"
Sep 08 14:12:23.837933 ip-10-0-132-254 kubenswrapper[2573]: E0908 14:12:23.837916    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd69f04ff10f4e58c791627078d3a604855ee76fc32669fcae613c4f81bc0fb5\": container with ID starting with fd69f04ff10f4e58c791627078d3a604855ee76fc32669fcae613c4f81bc0fb5 not found: ID does not exist" containerID="fd69f04ff10f4e58c791627078d3a604855ee76fc32669fcae613c4f81bc0fb5"
Sep 08 14:12:23.837983 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:12:23.837934    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd69f04ff10f4e58c791627078d3a604855ee76fc32669fcae613c4f81bc0fb5"} err="failed to get container status \"fd69f04ff10f4e58c791627078d3a604855ee76fc32669fcae613c4f81bc0fb5\": rpc error: code = NotFound desc = could not find container \"fd69f04ff10f4e58c791627078d3a604855ee76fc32669fcae613c4f81bc0fb5\": container with ID starting with fd69f04ff10f4e58c791627078d3a604855ee76fc32669fcae613c4f81bc0fb5 not found: ID does not exist"
Sep 08 14:12:23.842707 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:12:23.842669    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-6zd7n" podStartSLOduration=1.842657123 podStartE2EDuration="1.842657123s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 14:12:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:12:23.8415906 +0000 UTC m=+3394.113038560" watchObservedRunningTime="2026-09-08 14:12:23.842657123 +0000 UTC m=+3394.114105054"
Sep 08 14:12:23.857004 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:12:23.856980    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-895d8956b-4rjdm"]
Sep 08 14:12:23.861111 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:12:23.861093    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-895d8956b-4rjdm"]
Sep 08 14:12:24.175475 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:12:24.175448    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="de693a35-1091-423d-a5e0-78e0bd93bb69" path="/var/lib/kubelet/pods/de693a35-1091-423d-a5e0-78e0bd93bb69/volumes"
Sep 08 14:12:24.292947 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:12:24.292925    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:12:27.718584 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:12:27.718552    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:12:33.763351 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:12:33.763321    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:12:33.850815 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:12:33.850785    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:12:36.383966 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:12:36.383930    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:12:39.464597 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:12:39.464559    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:12:45.367552 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:12:45.367476    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:12:45.470806 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:12:45.470780    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:12:47.384642 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:12:47.384608    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:12:47.730234 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:12:47.730152    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:12:47.824721 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:12:47.824689    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:12:54.830385 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:12:54.830350    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-6zd7n"
Sep 08 14:13:44.691862 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:13:44.691823    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:13:44.764916 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:13:44.764884    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:14:44.830649 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:14:44.830605    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-6zd7n" podUID="4a2d4516-36b8-4301-a85d-2ab9d8824b5c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 08 14:14:44.835279 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:14:44.835253    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:14:44.854585 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:14:44.854551    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-6zd7n"]
Sep 08 14:14:44.923894 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:14:44.923865    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:14:46.423866 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:14:46.423831    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:14:54.830473 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:14:54.830434    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-6zd7n" podUID="4a2d4516-36b8-4301-a85d-2ab9d8824b5c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 08 14:14:59.873282 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:14:59.873240    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-6zd7n" podUID="4a2d4516-36b8-4301-a85d-2ab9d8824b5c" containerName="main" containerID="cri-o://ff49266ca6213c83d4018dd24f6c52689d94f41cf81b1d353416e1165ddf2f70" gracePeriod=45
Sep 08 14:15:04.830123 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:15:04.830083    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-6zd7n" podUID="4a2d4516-36b8-4301-a85d-2ab9d8824b5c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 08 14:15:04.830535 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:15:04.830186    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-6zd7n"
Sep 08 14:15:12.277541 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:15:12.277490    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:15:14.830293 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:15:14.830255    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-6zd7n" podUID="4a2d4516-36b8-4301-a85d-2ab9d8824b5c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 08 14:15:15.092752 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:15:15.092669    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:15:24.830007 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:15:24.829965    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-6zd7n" podUID="4a2d4516-36b8-4301-a85d-2ab9d8824b5c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 08 14:15:30.106463 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:15:30.106437    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-6zd7n"
Sep 08 14:15:30.288296 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:15:30.288221    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a2d4516-36b8-4301-a85d-2ab9d8824b5c-tls-certs\") pod \"4a2d4516-36b8-4301-a85d-2ab9d8824b5c\" (UID: \"4a2d4516-36b8-4301-a85d-2ab9d8824b5c\") "
Sep 08 14:15:30.288296 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:15:30.288269    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-npq5s\" (UniqueName: \"kubernetes.io/projected/4a2d4516-36b8-4301-a85d-2ab9d8824b5c-kube-api-access-npq5s\") pod \"4a2d4516-36b8-4301-a85d-2ab9d8824b5c\" (UID: \"4a2d4516-36b8-4301-a85d-2ab9d8824b5c\") "
Sep 08 14:15:30.290251 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:15:30.290224    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a2d4516-36b8-4301-a85d-2ab9d8824b5c-kube-api-access-npq5s" (OuterVolumeSpecName: "kube-api-access-npq5s") pod "4a2d4516-36b8-4301-a85d-2ab9d8824b5c" (UID: "4a2d4516-36b8-4301-a85d-2ab9d8824b5c"). InnerVolumeSpecName "kube-api-access-npq5s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 14:15:30.290357 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:15:30.290251    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a2d4516-36b8-4301-a85d-2ab9d8824b5c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4a2d4516-36b8-4301-a85d-2ab9d8824b5c" (UID: "4a2d4516-36b8-4301-a85d-2ab9d8824b5c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 14:15:30.389318 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:15:30.389295    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a2d4516-36b8-4301-a85d-2ab9d8824b5c-tls-certs\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 14:15:30.389318 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:15:30.389317    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-npq5s\" (UniqueName: \"kubernetes.io/projected/4a2d4516-36b8-4301-a85d-2ab9d8824b5c-kube-api-access-npq5s\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 14:15:30.540633 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:15:30.540565    2573 generic.go:356] "Generic (PLEG): container finished" podID="4a2d4516-36b8-4301-a85d-2ab9d8824b5c" containerID="ff49266ca6213c83d4018dd24f6c52689d94f41cf81b1d353416e1165ddf2f70" exitCode=0
Sep 08 14:15:30.540633 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:15:30.540611    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-6zd7n"
Sep 08 14:15:30.540633 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:15:30.540624    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-6zd7n" event={"ID":"4a2d4516-36b8-4301-a85d-2ab9d8824b5c","Type":"ContainerDied","Data":"ff49266ca6213c83d4018dd24f6c52689d94f41cf81b1d353416e1165ddf2f70"}
Sep 08 14:15:30.540826 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:15:30.540663    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-6zd7n" event={"ID":"4a2d4516-36b8-4301-a85d-2ab9d8824b5c","Type":"ContainerDied","Data":"3a041f41f139609fd49c2fe8c56474924fcb9993bf8925d3512b7a48bf7578c2"}
Sep 08 14:15:30.540826 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:15:30.540684    2573 scope.go:117] "RemoveContainer" containerID="ff49266ca6213c83d4018dd24f6c52689d94f41cf81b1d353416e1165ddf2f70"
Sep 08 14:15:30.549050 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:15:30.549028    2573 scope.go:117] "RemoveContainer" containerID="ff49266ca6213c83d4018dd24f6c52689d94f41cf81b1d353416e1165ddf2f70"
Sep 08 14:15:30.549331 ip-10-0-132-254 kubenswrapper[2573]: E0908 14:15:30.549312    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff49266ca6213c83d4018dd24f6c52689d94f41cf81b1d353416e1165ddf2f70\": container with ID starting with ff49266ca6213c83d4018dd24f6c52689d94f41cf81b1d353416e1165ddf2f70 not found: ID does not exist" containerID="ff49266ca6213c83d4018dd24f6c52689d94f41cf81b1d353416e1165ddf2f70"
Sep 08 14:15:30.549404 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:15:30.549338    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff49266ca6213c83d4018dd24f6c52689d94f41cf81b1d353416e1165ddf2f70"} err="failed to get container status \"ff49266ca6213c83d4018dd24f6c52689d94f41cf81b1d353416e1165ddf2f70\": rpc error: code = NotFound desc = could not find container \"ff49266ca6213c83d4018dd24f6c52689d94f41cf81b1d353416e1165ddf2f70\": container with ID starting with ff49266ca6213c83d4018dd24f6c52689d94f41cf81b1d353416e1165ddf2f70 not found: ID does not exist"
Sep 08 14:15:30.562040 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:15:30.562019    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-6zd7n"]
Sep 08 14:15:30.568553 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:15:30.568534    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-6zd7n"]
Sep 08 14:15:32.174656 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:15:32.174617    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a2d4516-36b8-4301-a85d-2ab9d8824b5c" path="/var/lib/kubelet/pods/4a2d4516-36b8-4301-a85d-2ab9d8824b5c/volumes"
Sep 08 14:15:34.739423 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:15:34.739383    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:15:34.842492 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:15:34.842451    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:15:41.205714 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:15:41.205665    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:15:47.068878 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:15:47.068794    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:15:47.186431 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:15:47.186400    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:15:50.407338 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:15:50.407312    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8h8cn_b936238e-bdf1-4dac-8bf8-6417c313597c/ovn-acl-logging/0.log"
Sep 08 14:15:50.413328 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:15:50.413308    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8h8cn_b936238e-bdf1-4dac-8bf8-6417c313597c/ovn-acl-logging/0.log"
Sep 08 14:15:52.846669 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:15:52.846632    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:16:02.424450 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:16:02.424417    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:16:02.518713 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:16:02.518681    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:16:07.465669 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:16:07.465637    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:16:29.171309 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:16:29.171271    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:16:29.488299 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:16:29.488206    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:16:29.589661 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:16:29.589626    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:16:31.132072 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:16:31.132038    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:16:34.827519 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:16:34.827473    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8759d6f58-w6dvw"]
Sep 08 14:16:34.827919 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:16:34.827765    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de693a35-1091-423d-a5e0-78e0bd93bb69" containerName="storage-initializer"
Sep 08 14:16:34.827919 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:16:34.827777    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="de693a35-1091-423d-a5e0-78e0bd93bb69" containerName="storage-initializer"
Sep 08 14:16:34.827919 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:16:34.827786    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a2d4516-36b8-4301-a85d-2ab9d8824b5c" containerName="main"
Sep 08 14:16:34.827919 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:16:34.827792    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a2d4516-36b8-4301-a85d-2ab9d8824b5c" containerName="main"
Sep 08 14:16:34.827919 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:16:34.827807    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de693a35-1091-423d-a5e0-78e0bd93bb69" containerName="main"
Sep 08 14:16:34.827919 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:16:34.827812    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="de693a35-1091-423d-a5e0-78e0bd93bb69" containerName="main"
Sep 08 14:16:34.827919 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:16:34.827861    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a2d4516-36b8-4301-a85d-2ab9d8824b5c" containerName="main"
Sep 08 14:16:34.827919 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:16:34.827868    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="de693a35-1091-423d-a5e0-78e0bd93bb69" containerName="main"
Sep 08 14:16:34.831021 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:16:34.831004    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8759d6f58-w6dvw"
Sep 08 14:16:34.833404 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:16:34.833366    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"seaweedfs-s3-creds\""
Sep 08 14:16:34.833614 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:16:34.833466    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"kube-root-ca.crt\""
Sep 08 14:16:34.833614 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:16:34.833477    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"openshift-service-ca.crt\""
Sep 08 14:16:34.833614 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:16:34.833528    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"default-dockercfg-d4mnv\""
Sep 08 14:16:34.833813 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:16:34.833788    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"odh-kserve-custom-ca-bundle\""
Sep 08 14:16:34.833872 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:16:34.833855    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisv44d181485fad85e662eb092f3749502f-kserve-self-signed-certs\""
Sep 08 14:16:34.837288 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:16:34.837264    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-dockercfg-x66sq\""
Sep 08 14:16:34.853846 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:16:34.853823    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8759d6f58-w6dvw"]
Sep 08 14:16:34.905725 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:16:34.905691    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:16:34.975433 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:16:34.975402    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4cc6cc62-69e6-465b-a24d-6c75b5c24c11-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8759d6f58-w6dvw\" (UID: \"4cc6cc62-69e6-465b-a24d-6c75b5c24c11\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8759d6f58-w6dvw"
Sep 08 14:16:34.975433 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:16:34.975436    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4cc6cc62-69e6-465b-a24d-6c75b5c24c11-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8759d6f58-w6dvw\" (UID: \"4cc6cc62-69e6-465b-a24d-6c75b5c24c11\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8759d6f58-w6dvw"
Sep 08 14:16:34.975647 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:16:34.975457    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4cc6cc62-69e6-465b-a24d-6c75b5c24c11-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8759d6f58-w6dvw\" (UID: \"4cc6cc62-69e6-465b-a24d-6c75b5c24c11\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8759d6f58-w6dvw"
Sep 08 14:16:34.975647 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:16:34.975518    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4cc6cc62-69e6-465b-a24d-6c75b5c24c11-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8759d6f58-w6dvw\" (UID: \"4cc6cc62-69e6-465b-a24d-6c75b5c24c11\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8759d6f58-w6dvw"
Sep 08 14:16:34.975647 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:16:34.975557    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4cc6cc62-69e6-465b-a24d-6c75b5c24c11-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8759d6f58-w6dvw\" (UID: \"4cc6cc62-69e6-465b-a24d-6c75b5c24c11\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8759d6f58-w6dvw"
Sep 08 14:16:34.975647 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:16:34.975606    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4cc6cc62-69e6-465b-a24d-6c75b5c24c11-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8759d6f58-w6dvw\" (UID: \"4cc6cc62-69e6-465b-a24d-6c75b5c24c11\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8759d6f58-w6dvw"
Sep 08 14:16:34.975795 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:16:34.975669    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4cc6cc62-69e6-465b-a24d-6c75b5c24c11-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8759d6f58-w6dvw\" (UID: \"4cc6cc62-69e6-465b-a24d-6c75b5c24c11\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8759d6f58-w6dvw"
Sep 08 14:16:34.975795 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:16:34.975708    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sj5b2\" (UniqueName: \"kubernetes.io/projected/4cc6cc62-69e6-465b-a24d-6c75b5c24c11-kube-api-access-sj5b2\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8759d6f58-w6dvw\" (UID: \"4cc6cc62-69e6-465b-a24d-6c75b5c24c11\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8759d6f58-w6dvw"
Sep 08 14:16:35.076411 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:16:35.076375    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4cc6cc62-69e6-465b-a24d-6c75b5c24c11-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8759d6f58-w6dvw\" (UID: \"4cc6cc62-69e6-465b-a24d-6c75b5c24c11\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8759d6f58-w6dvw"
Sep 08 14:16:35.076570 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:16:35.076432    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sj5b2\" (UniqueName: \"kubernetes.io/projected/4cc6cc62-69e6-465b-a24d-6c75b5c24c11-kube-api-access-sj5b2\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8759d6f58-w6dvw\" (UID: \"4cc6cc62-69e6-465b-a24d-6c75b5c24c11\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8759d6f58-w6dvw"
Sep 08 14:16:35.076570 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:16:35.076477    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4cc6cc62-69e6-465b-a24d-6c75b5c24c11-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8759d6f58-w6dvw\" (UID: \"4cc6cc62-69e6-465b-a24d-6c75b5c24c11\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8759d6f58-w6dvw"
Sep 08 14:16:35.076570 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:16:35.076541    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4cc6cc62-69e6-465b-a24d-6c75b5c24c11-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8759d6f58-w6dvw\" (UID: \"4cc6cc62-69e6-465b-a24d-6c75b5c24c11\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8759d6f58-w6dvw"
Sep 08 14:16:35.076710 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:16:35.076575    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4cc6cc62-69e6-465b-a24d-6c75b5c24c11-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8759d6f58-w6dvw\" (UID: \"4cc6cc62-69e6-465b-a24d-6c75b5c24c11\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8759d6f58-w6dvw"
Sep 08 14:16:35.076710 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:16:35.076600    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4cc6cc62-69e6-465b-a24d-6c75b5c24c11-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8759d6f58-w6dvw\" (UID: \"4cc6cc62-69e6-465b-a24d-6c75b5c24c11\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8759d6f58-w6dvw"
Sep 08 14:16:35.076710 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:16:35.076624    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4cc6cc62-69e6-465b-a24d-6c75b5c24c11-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8759d6f58-w6dvw\" (UID: \"4cc6cc62-69e6-465b-a24d-6c75b5c24c11\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8759d6f58-w6dvw"
Sep 08 14:16:35.076710 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:16:35.076681    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4cc6cc62-69e6-465b-a24d-6c75b5c24c11-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8759d6f58-w6dvw\" (UID: \"4cc6cc62-69e6-465b-a24d-6c75b5c24c11\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8759d6f58-w6dvw"
Sep 08 14:16:35.076891 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:16:35.076869    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4cc6cc62-69e6-465b-a24d-6c75b5c24c11-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8759d6f58-w6dvw\" (UID: \"4cc6cc62-69e6-465b-a24d-6c75b5c24c11\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8759d6f58-w6dvw"
Sep 08 14:16:35.077018 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:16:35.076995    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4cc6cc62-69e6-465b-a24d-6c75b5c24c11-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8759d6f58-w6dvw\" (UID: \"4cc6cc62-69e6-465b-a24d-6c75b5c24c11\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8759d6f58-w6dvw"
Sep 08 14:16:35.077201 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:16:35.077137    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4cc6cc62-69e6-465b-a24d-6c75b5c24c11-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8759d6f58-w6dvw\" (UID: \"4cc6cc62-69e6-465b-a24d-6c75b5c24c11\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8759d6f58-w6dvw"
Sep 08 14:16:35.077312 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:16:35.077292    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4cc6cc62-69e6-465b-a24d-6c75b5c24c11-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8759d6f58-w6dvw\" (UID: \"4cc6cc62-69e6-465b-a24d-6c75b5c24c11\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8759d6f58-w6dvw"
Sep 08 14:16:35.077385 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:16:35.077336    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4cc6cc62-69e6-465b-a24d-6c75b5c24c11-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8759d6f58-w6dvw\" (UID: \"4cc6cc62-69e6-465b-a24d-6c75b5c24c11\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8759d6f58-w6dvw"
Sep 08 14:16:35.079231 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:16:35.079173    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4cc6cc62-69e6-465b-a24d-6c75b5c24c11-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8759d6f58-w6dvw\" (UID: \"4cc6cc62-69e6-465b-a24d-6c75b5c24c11\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8759d6f58-w6dvw"
Sep 08 14:16:35.079231 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:16:35.079205    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4cc6cc62-69e6-465b-a24d-6c75b5c24c11-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8759d6f58-w6dvw\" (UID: \"4cc6cc62-69e6-465b-a24d-6c75b5c24c11\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8759d6f58-w6dvw"
Sep 08 14:16:35.108274 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:16:35.108241    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sj5b2\" (UniqueName: \"kubernetes.io/projected/4cc6cc62-69e6-465b-a24d-6c75b5c24c11-kube-api-access-sj5b2\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8759d6f58-w6dvw\" (UID: \"4cc6cc62-69e6-465b-a24d-6c75b5c24c11\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8759d6f58-w6dvw"
Sep 08 14:16:35.141284 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:16:35.141254    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8759d6f58-w6dvw"
Sep 08 14:16:35.150947 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:16:35.150924    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:16:35.423417 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:16:35.423384    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8759d6f58-w6dvw"]
Sep 08 14:16:35.425982 ip-10-0-132-254 kubenswrapper[2573]: W0908 14:16:35.425955    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cc6cc62_69e6_465b_a24d_6c75b5c24c11.slice/crio-66c78b26037919c83c0a0ef9f9b484801a5792b97ed9fa9c52e5a27c1bb2b0ee WatchSource:0}: Error finding container 66c78b26037919c83c0a0ef9f9b484801a5792b97ed9fa9c52e5a27c1bb2b0ee: Status 404 returned error can't find the container with id 66c78b26037919c83c0a0ef9f9b484801a5792b97ed9fa9c52e5a27c1bb2b0ee
Sep 08 14:16:35.428019 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:16:35.428000    2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 08 14:16:35.784536 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:16:35.784371    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8759d6f58-w6dvw" event={"ID":"4cc6cc62-69e6-465b-a24d-6c75b5c24c11","Type":"ContainerStarted","Data":"66c78b26037919c83c0a0ef9f9b484801a5792b97ed9fa9c52e5a27c1bb2b0ee"}
Sep 08 14:16:37.505037 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:16:37.505003    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:16:38.799786 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:16:38.799753    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8759d6f58-w6dvw" event={"ID":"4cc6cc62-69e6-465b-a24d-6c75b5c24c11","Type":"ContainerStarted","Data":"0cb36b51f9a46205a8f841392992cf2bfe8768ad6d0b760ea79844b6bbc6f649"}
Sep 08 14:16:38.800152 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:16:38.799907    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8759d6f58-w6dvw"
Sep 08 14:16:39.805543 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:16:39.805488    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8759d6f58-w6dvw" event={"ID":"4cc6cc62-69e6-465b-a24d-6c75b5c24c11","Type":"ContainerStarted","Data":"d8f4fbd7353955bb438bec6c9ad44c88f6292b52556f51f49e441f435f832ff6"}
Sep 08 14:16:41.815432 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:16:41.815394    2573 generic.go:356] "Generic (PLEG): container finished" podID="4cc6cc62-69e6-465b-a24d-6c75b5c24c11" containerID="d8f4fbd7353955bb438bec6c9ad44c88f6292b52556f51f49e441f435f832ff6" exitCode=0
Sep 08 14:16:41.815885 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:16:41.815468    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8759d6f58-w6dvw" event={"ID":"4cc6cc62-69e6-465b-a24d-6c75b5c24c11","Type":"ContainerDied","Data":"d8f4fbd7353955bb438bec6c9ad44c88f6292b52556f51f49e441f435f832ff6"}
Sep 08 14:16:42.822858 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:16:42.822813    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8759d6f58-w6dvw" event={"ID":"4cc6cc62-69e6-465b-a24d-6c75b5c24c11","Type":"ContainerStarted","Data":"67780bb1493097f2266343444a2a8ae4d159087430aadf1b9d6b6a10e60cb0c0"}
Sep 08 14:16:42.846201 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:16:42.846117    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8759d6f58-w6dvw" podStartSLOduration=3.506679009 podStartE2EDuration="8.84609648s" totalImagesPullingTime="3.271304966s" totalInitContainerRuntime="2.068112505s" isStatefulPod=false podCreationTimestamp="2026-09-08 14:16:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:16:42.845686399 +0000 UTC m=+3653.117134330" watchObservedRunningTime="2026-09-08 14:16:42.84609648 +0000 UTC m=+3653.117544412"
Sep 08 14:16:45.142145 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:16:45.142099    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8759d6f58-w6dvw"
Sep 08 14:16:45.142587 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:16:45.142166    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8759d6f58-w6dvw"
Sep 08 14:16:45.143560 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:16:45.143528    2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8759d6f58-w6dvw" podUID="4cc6cc62-69e6-465b-a24d-6c75b5c24c11" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8001/health\": dial tcp 10.133.0.46:8001: connect: connection refused"
Sep 08 14:16:45.652310 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:16:45.652274    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:16:55.142122 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:16:55.142083    2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8759d6f58-w6dvw" podUID="4cc6cc62-69e6-465b-a24d-6c75b5c24c11" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8001/health\": dial tcp 10.133.0.46:8001: connect: connection refused"
Sep 08 14:16:55.154548 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:16:55.154491    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8759d6f58-w6dvw"
Sep 08 14:16:55.681141 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:16:55.681100    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:17:01.646548 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:17:01.646486    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:17:01.748541 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:17:01.748484    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:17:05.142314 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:17:05.142261    2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8759d6f58-w6dvw" podUID="4cc6cc62-69e6-465b-a24d-6c75b5c24c11" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8001/health\": dial tcp 10.133.0.46:8001: connect: connection refused"
Sep 08 14:17:15.141964 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:17:15.141859    2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8759d6f58-w6dvw" podUID="4cc6cc62-69e6-465b-a24d-6c75b5c24c11" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8001/health\": dial tcp 10.133.0.46:8001: connect: connection refused"
Sep 08 14:17:25.142492 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:17:25.142436    2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8759d6f58-w6dvw" podUID="4cc6cc62-69e6-465b-a24d-6c75b5c24c11" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8001/health\": dial tcp 10.133.0.46:8001: connect: connection refused"
Sep 08 14:17:35.142714 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:17:35.142659    2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8759d6f58-w6dvw" podUID="4cc6cc62-69e6-465b-a24d-6c75b5c24c11" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8001/health\": dial tcp 10.133.0.46:8001: connect: connection refused"
Sep 08 14:17:45.142588 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:17:45.142526    2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8759d6f58-w6dvw" podUID="4cc6cc62-69e6-465b-a24d-6c75b5c24c11" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8001/health\": dial tcp 10.133.0.46:8001: connect: connection refused"
Sep 08 14:17:49.890513 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:17:49.890452    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:17:51.025940 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:17:51.025902    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:17:55.142342 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:17:55.142299    2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8759d6f58-w6dvw" podUID="4cc6cc62-69e6-465b-a24d-6c75b5c24c11" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8001/health\": dial tcp 10.133.0.46:8001: connect: connection refused"
Sep 08 14:18:05.141982 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:18:05.141936    2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8759d6f58-w6dvw" podUID="4cc6cc62-69e6-465b-a24d-6c75b5c24c11" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8001/health\": dial tcp 10.133.0.46:8001: connect: connection refused"
Sep 08 14:18:10.945878 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:18:10.945838    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:18:11.064158 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:18:11.064126    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:18:12.190776 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:18:12.190738    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:18:15.142156 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:18:15.142109    2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8759d6f58-w6dvw" podUID="4cc6cc62-69e6-465b-a24d-6c75b5c24c11" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8001/health\": dial tcp 10.133.0.46:8001: connect: connection refused"
Sep 08 14:18:17.036605 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:18:17.036563    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:18:17.331526 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:18:17.331369    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:18:18.239837 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:18:18.239807    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:18:25.142739 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:18:25.142687    2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8759d6f58-w6dvw" podUID="4cc6cc62-69e6-465b-a24d-6c75b5c24c11" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8001/health\": dial tcp 10.133.0.46:8001: connect: connection refused"
Sep 08 14:18:35.152389 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:18:35.152350    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8759d6f58-w6dvw"
Sep 08 14:18:35.169124 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:18:35.169097    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8759d6f58-w6dvw"
Sep 08 14:18:54.357801 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:18:54.357768    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:18:54.367162 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:18:54.367128    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8759d6f58-w6dvw"]
Sep 08 14:18:54.424083 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:18:54.424055    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:19:09.394039 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:19:09.393999    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8759d6f58-w6dvw" podUID="4cc6cc62-69e6-465b-a24d-6c75b5c24c11" containerName="main" containerID="cri-o://67780bb1493097f2266343444a2a8ae4d159087430aadf1b9d6b6a10e60cb0c0" gracePeriod=45
Sep 08 14:19:09.934824 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:19:09.934794    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8759d6f58-w6dvw" podUID="4cc6cc62-69e6-465b-a24d-6c75b5c24c11" containerName="llm-d-routing-sidecar" containerID="cri-o://0cb36b51f9a46205a8f841392992cf2bfe8768ad6d0b760ea79844b6bbc6f649" gracePeriod=45
Sep 08 14:19:10.166532 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:19:10.166510    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8759d6f58-w6dvw"
Sep 08 14:19:10.284748 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:19:10.284720    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4cc6cc62-69e6-465b-a24d-6c75b5c24c11-cabundle-cert\") pod \"4cc6cc62-69e6-465b-a24d-6c75b5c24c11\" (UID: \"4cc6cc62-69e6-465b-a24d-6c75b5c24c11\") "
Sep 08 14:19:10.284932 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:19:10.284758    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4cc6cc62-69e6-465b-a24d-6c75b5c24c11-kserve-provision-location\") pod \"4cc6cc62-69e6-465b-a24d-6c75b5c24c11\" (UID: \"4cc6cc62-69e6-465b-a24d-6c75b5c24c11\") "
Sep 08 14:19:10.284932 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:19:10.284786    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4cc6cc62-69e6-465b-a24d-6c75b5c24c11-tmp-dir\") pod \"4cc6cc62-69e6-465b-a24d-6c75b5c24c11\" (UID: \"4cc6cc62-69e6-465b-a24d-6c75b5c24c11\") "
Sep 08 14:19:10.284932 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:19:10.284839    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4cc6cc62-69e6-465b-a24d-6c75b5c24c11-dshm\") pod \"4cc6cc62-69e6-465b-a24d-6c75b5c24c11\" (UID: \"4cc6cc62-69e6-465b-a24d-6c75b5c24c11\") "
Sep 08 14:19:10.284932 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:19:10.284860    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sj5b2\" (UniqueName: \"kubernetes.io/projected/4cc6cc62-69e6-465b-a24d-6c75b5c24c11-kube-api-access-sj5b2\") pod \"4cc6cc62-69e6-465b-a24d-6c75b5c24c11\" (UID: \"4cc6cc62-69e6-465b-a24d-6c75b5c24c11\") "
Sep 08 14:19:10.284932 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:19:10.284880    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4cc6cc62-69e6-465b-a24d-6c75b5c24c11-tls-certs\") pod \"4cc6cc62-69e6-465b-a24d-6c75b5c24c11\" (UID: \"4cc6cc62-69e6-465b-a24d-6c75b5c24c11\") "
Sep 08 14:19:10.284932 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:19:10.284901    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4cc6cc62-69e6-465b-a24d-6c75b5c24c11-model-cache\") pod \"4cc6cc62-69e6-465b-a24d-6c75b5c24c11\" (UID: \"4cc6cc62-69e6-465b-a24d-6c75b5c24c11\") "
Sep 08 14:19:10.285270 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:19:10.284936    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4cc6cc62-69e6-465b-a24d-6c75b5c24c11-home\") pod \"4cc6cc62-69e6-465b-a24d-6c75b5c24c11\" (UID: \"4cc6cc62-69e6-465b-a24d-6c75b5c24c11\") "
Sep 08 14:19:10.285270 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:19:10.285097    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cc6cc62-69e6-465b-a24d-6c75b5c24c11-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4cc6cc62-69e6-465b-a24d-6c75b5c24c11" (UID: "4cc6cc62-69e6-465b-a24d-6c75b5c24c11"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 14:19:10.285270 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:19:10.285253    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cc6cc62-69e6-465b-a24d-6c75b5c24c11-model-cache" (OuterVolumeSpecName: "model-cache") pod "4cc6cc62-69e6-465b-a24d-6c75b5c24c11" (UID: "4cc6cc62-69e6-465b-a24d-6c75b5c24c11"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:19:10.285768 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:19:10.285722    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cc6cc62-69e6-465b-a24d-6c75b5c24c11-home" (OuterVolumeSpecName: "home") pod "4cc6cc62-69e6-465b-a24d-6c75b5c24c11" (UID: "4cc6cc62-69e6-465b-a24d-6c75b5c24c11"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:19:10.287404 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:19:10.287372    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cc6cc62-69e6-465b-a24d-6c75b5c24c11-dshm" (OuterVolumeSpecName: "dshm") pod "4cc6cc62-69e6-465b-a24d-6c75b5c24c11" (UID: "4cc6cc62-69e6-465b-a24d-6c75b5c24c11"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:19:10.287404 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:19:10.287395    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4cc6cc62-69e6-465b-a24d-6c75b5c24c11-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4cc6cc62-69e6-465b-a24d-6c75b5c24c11" (UID: "4cc6cc62-69e6-465b-a24d-6c75b5c24c11"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 14:19:10.287666 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:19:10.287475    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cc6cc62-69e6-465b-a24d-6c75b5c24c11-kube-api-access-sj5b2" (OuterVolumeSpecName: "kube-api-access-sj5b2") pod "4cc6cc62-69e6-465b-a24d-6c75b5c24c11" (UID: "4cc6cc62-69e6-465b-a24d-6c75b5c24c11"). InnerVolumeSpecName "kube-api-access-sj5b2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 14:19:10.297372 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:19:10.297349    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cc6cc62-69e6-465b-a24d-6c75b5c24c11-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4cc6cc62-69e6-465b-a24d-6c75b5c24c11" (UID: "4cc6cc62-69e6-465b-a24d-6c75b5c24c11"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:19:10.306255 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:19:10.306235    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cc6cc62-69e6-465b-a24d-6c75b5c24c11-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4cc6cc62-69e6-465b-a24d-6c75b5c24c11" (UID: "4cc6cc62-69e6-465b-a24d-6c75b5c24c11"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:19:10.386154 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:19:10.386131    2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4cc6cc62-69e6-465b-a24d-6c75b5c24c11-home\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 14:19:10.386154 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:19:10.386154    2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4cc6cc62-69e6-465b-a24d-6c75b5c24c11-cabundle-cert\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 14:19:10.386321 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:19:10.386165    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4cc6cc62-69e6-465b-a24d-6c75b5c24c11-kserve-provision-location\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 14:19:10.386321 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:19:10.386175    2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4cc6cc62-69e6-465b-a24d-6c75b5c24c11-tmp-dir\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 14:19:10.386321 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:19:10.386184    2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4cc6cc62-69e6-465b-a24d-6c75b5c24c11-dshm\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 14:19:10.386321 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:19:10.386192    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sj5b2\" (UniqueName: \"kubernetes.io/projected/4cc6cc62-69e6-465b-a24d-6c75b5c24c11-kube-api-access-sj5b2\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 14:19:10.386321 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:19:10.386205    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4cc6cc62-69e6-465b-a24d-6c75b5c24c11-tls-certs\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 14:19:10.386321 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:19:10.386216    2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4cc6cc62-69e6-465b-a24d-6c75b5c24c11-model-cache\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 14:19:10.455516 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:19:10.455428    2573 generic.go:356] "Generic (PLEG): container finished" podID="4cc6cc62-69e6-465b-a24d-6c75b5c24c11" containerID="67780bb1493097f2266343444a2a8ae4d159087430aadf1b9d6b6a10e60cb0c0" exitCode=0
Sep 08 14:19:10.455516 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:19:10.455449    2573 generic.go:356] "Generic (PLEG): container finished" podID="4cc6cc62-69e6-465b-a24d-6c75b5c24c11" containerID="0cb36b51f9a46205a8f841392992cf2bfe8768ad6d0b760ea79844b6bbc6f649" exitCode=0
Sep 08 14:19:10.455516 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:19:10.455482    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8759d6f58-w6dvw" event={"ID":"4cc6cc62-69e6-465b-a24d-6c75b5c24c11","Type":"ContainerDied","Data":"67780bb1493097f2266343444a2a8ae4d159087430aadf1b9d6b6a10e60cb0c0"}
Sep 08 14:19:10.455516 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:19:10.455491    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8759d6f58-w6dvw"
Sep 08 14:19:10.456081 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:19:10.455547    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8759d6f58-w6dvw" event={"ID":"4cc6cc62-69e6-465b-a24d-6c75b5c24c11","Type":"ContainerDied","Data":"0cb36b51f9a46205a8f841392992cf2bfe8768ad6d0b760ea79844b6bbc6f649"}
Sep 08 14:19:10.456081 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:19:10.455566    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8759d6f58-w6dvw" event={"ID":"4cc6cc62-69e6-465b-a24d-6c75b5c24c11","Type":"ContainerDied","Data":"66c78b26037919c83c0a0ef9f9b484801a5792b97ed9fa9c52e5a27c1bb2b0ee"}
Sep 08 14:19:10.456081 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:19:10.455585    2573 scope.go:117] "RemoveContainer" containerID="67780bb1493097f2266343444a2a8ae4d159087430aadf1b9d6b6a10e60cb0c0"
Sep 08 14:19:10.464297 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:19:10.464280    2573 scope.go:117] "RemoveContainer" containerID="d8f4fbd7353955bb438bec6c9ad44c88f6292b52556f51f49e441f435f832ff6"
Sep 08 14:19:10.471417 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:19:10.471400    2573 scope.go:117] "RemoveContainer" containerID="0cb36b51f9a46205a8f841392992cf2bfe8768ad6d0b760ea79844b6bbc6f649"
Sep 08 14:19:10.477385 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:19:10.477362    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8759d6f58-w6dvw"]
Sep 08 14:19:10.478564 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:19:10.478548    2573 scope.go:117] "RemoveContainer" containerID="67780bb1493097f2266343444a2a8ae4d159087430aadf1b9d6b6a10e60cb0c0"
Sep 08 14:19:10.478824 ip-10-0-132-254 kubenswrapper[2573]: E0908 14:19:10.478802    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67780bb1493097f2266343444a2a8ae4d159087430aadf1b9d6b6a10e60cb0c0\": container with ID starting with 67780bb1493097f2266343444a2a8ae4d159087430aadf1b9d6b6a10e60cb0c0 not found: ID does not exist" containerID="67780bb1493097f2266343444a2a8ae4d159087430aadf1b9d6b6a10e60cb0c0"
Sep 08 14:19:10.478900 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:19:10.478835    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67780bb1493097f2266343444a2a8ae4d159087430aadf1b9d6b6a10e60cb0c0"} err="failed to get container status \"67780bb1493097f2266343444a2a8ae4d159087430aadf1b9d6b6a10e60cb0c0\": rpc error: code = NotFound desc = could not find container \"67780bb1493097f2266343444a2a8ae4d159087430aadf1b9d6b6a10e60cb0c0\": container with ID starting with 67780bb1493097f2266343444a2a8ae4d159087430aadf1b9d6b6a10e60cb0c0 not found: ID does not exist"
Sep 08 14:19:10.478900 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:19:10.478859    2573 scope.go:117] "RemoveContainer" containerID="d8f4fbd7353955bb438bec6c9ad44c88f6292b52556f51f49e441f435f832ff6"
Sep 08 14:19:10.479155 ip-10-0-132-254 kubenswrapper[2573]: E0908 14:19:10.479138    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8f4fbd7353955bb438bec6c9ad44c88f6292b52556f51f49e441f435f832ff6\": container with ID starting with d8f4fbd7353955bb438bec6c9ad44c88f6292b52556f51f49e441f435f832ff6 not found: ID does not exist" containerID="d8f4fbd7353955bb438bec6c9ad44c88f6292b52556f51f49e441f435f832ff6"
Sep 08 14:19:10.479230 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:19:10.479164    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8f4fbd7353955bb438bec6c9ad44c88f6292b52556f51f49e441f435f832ff6"} err="failed to get container status \"d8f4fbd7353955bb438bec6c9ad44c88f6292b52556f51f49e441f435f832ff6\": rpc error: code = NotFound desc = could not find container \"d8f4fbd7353955bb438bec6c9ad44c88f6292b52556f51f49e441f435f832ff6\": container with ID starting with d8f4fbd7353955bb438bec6c9ad44c88f6292b52556f51f49e441f435f832ff6 not found: ID does not exist"
Sep 08 14:19:10.479230 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:19:10.479184    2573 scope.go:117] "RemoveContainer" containerID="0cb36b51f9a46205a8f841392992cf2bfe8768ad6d0b760ea79844b6bbc6f649"
Sep 08 14:19:10.479487 ip-10-0-132-254 kubenswrapper[2573]: E0908 14:19:10.479421    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0cb36b51f9a46205a8f841392992cf2bfe8768ad6d0b760ea79844b6bbc6f649\": container with ID starting with 0cb36b51f9a46205a8f841392992cf2bfe8768ad6d0b760ea79844b6bbc6f649 not found: ID does not exist" containerID="0cb36b51f9a46205a8f841392992cf2bfe8768ad6d0b760ea79844b6bbc6f649"
Sep 08 14:19:10.479487 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:19:10.479459    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0cb36b51f9a46205a8f841392992cf2bfe8768ad6d0b760ea79844b6bbc6f649"} err="failed to get container status \"0cb36b51f9a46205a8f841392992cf2bfe8768ad6d0b760ea79844b6bbc6f649\": rpc error: code = NotFound desc = could not find container \"0cb36b51f9a46205a8f841392992cf2bfe8768ad6d0b760ea79844b6bbc6f649\": container with ID starting with 0cb36b51f9a46205a8f841392992cf2bfe8768ad6d0b760ea79844b6bbc6f649 not found: ID does not exist"
Sep 08 14:19:10.479487 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:19:10.479496    2573 scope.go:117] "RemoveContainer" containerID="67780bb1493097f2266343444a2a8ae4d159087430aadf1b9d6b6a10e60cb0c0"
Sep 08 14:19:10.479824 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:19:10.479780    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67780bb1493097f2266343444a2a8ae4d159087430aadf1b9d6b6a10e60cb0c0"} err="failed to get container status \"67780bb1493097f2266343444a2a8ae4d159087430aadf1b9d6b6a10e60cb0c0\": rpc error: code = NotFound desc = could not find container \"67780bb1493097f2266343444a2a8ae4d159087430aadf1b9d6b6a10e60cb0c0\": container with ID starting with 67780bb1493097f2266343444a2a8ae4d159087430aadf1b9d6b6a10e60cb0c0 not found: ID does not exist"
Sep 08 14:19:10.479824 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:19:10.479809    2573 scope.go:117] "RemoveContainer" containerID="d8f4fbd7353955bb438bec6c9ad44c88f6292b52556f51f49e441f435f832ff6"
Sep 08 14:19:10.480177 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:19:10.480098    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8f4fbd7353955bb438bec6c9ad44c88f6292b52556f51f49e441f435f832ff6"} err="failed to get container status \"d8f4fbd7353955bb438bec6c9ad44c88f6292b52556f51f49e441f435f832ff6\": rpc error: code = NotFound desc = could not find container \"d8f4fbd7353955bb438bec6c9ad44c88f6292b52556f51f49e441f435f832ff6\": container with ID starting with d8f4fbd7353955bb438bec6c9ad44c88f6292b52556f51f49e441f435f832ff6 not found: ID does not exist"
Sep 08 14:19:10.480177 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:19:10.480124    2573 scope.go:117] "RemoveContainer" containerID="0cb36b51f9a46205a8f841392992cf2bfe8768ad6d0b760ea79844b6bbc6f649"
Sep 08 14:19:10.480421 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:19:10.480403    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0cb36b51f9a46205a8f841392992cf2bfe8768ad6d0b760ea79844b6bbc6f649"} err="failed to get container status \"0cb36b51f9a46205a8f841392992cf2bfe8768ad6d0b760ea79844b6bbc6f649\": rpc error: code = NotFound desc = could not find container \"0cb36b51f9a46205a8f841392992cf2bfe8768ad6d0b760ea79844b6bbc6f649\": container with ID starting with 0cb36b51f9a46205a8f841392992cf2bfe8768ad6d0b760ea79844b6bbc6f649 not found: ID does not exist"
Sep 08 14:19:10.481424 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:19:10.481406    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-8759d6f58-w6dvw"]
Sep 08 14:19:12.174934 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:19:12.174897    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4cc6cc62-69e6-465b-a24d-6c75b5c24c11" path="/var/lib/kubelet/pods/4cc6cc62-69e6-465b-a24d-6c75b5c24c11/volumes"
Sep 08 14:19:44.529669 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:19:44.529633    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:19:44.637392 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:19:44.637358    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:19:51.648589 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:19:51.648552    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:19:54.524565 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:19:54.524525    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:19:54.624400 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:19:54.624370    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:19:57.677916 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:19:57.677885    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:19:57.746081 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:19:57.746049    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:20:12.300111 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:20:12.300077    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5b85f6885f-p85l9"]
Sep 08 14:20:12.300549 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:20:12.300375    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4cc6cc62-69e6-465b-a24d-6c75b5c24c11" containerName="main"
Sep 08 14:20:12.300549 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:20:12.300386    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="4cc6cc62-69e6-465b-a24d-6c75b5c24c11" containerName="main"
Sep 08 14:20:12.300549 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:20:12.300421    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4cc6cc62-69e6-465b-a24d-6c75b5c24c11" containerName="llm-d-routing-sidecar"
Sep 08 14:20:12.300549 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:20:12.300428    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="4cc6cc62-69e6-465b-a24d-6c75b5c24c11" containerName="llm-d-routing-sidecar"
Sep 08 14:20:12.300549 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:20:12.300438    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4cc6cc62-69e6-465b-a24d-6c75b5c24c11" containerName="storage-initializer"
Sep 08 14:20:12.300549 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:20:12.300443    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="4cc6cc62-69e6-465b-a24d-6c75b5c24c11" containerName="storage-initializer"
Sep 08 14:20:12.300549 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:20:12.300520    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="4cc6cc62-69e6-465b-a24d-6c75b5c24c11" containerName="llm-d-routing-sidecar"
Sep 08 14:20:12.300549 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:20:12.300528    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="4cc6cc62-69e6-465b-a24d-6c75b5c24c11" containerName="main"
Sep 08 14:20:12.303493 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:20:12.303477    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5b85f6885f-p85l9"
Sep 08 14:20:12.306090 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:20:12.306069    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"kube-root-ca.crt\""
Sep 08 14:20:12.306281 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:20:12.306257    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"default-dockercfg-rjpx4\""
Sep 08 14:20:12.306476 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:20:12.306277    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"openshift-service-ca.crt\""
Sep 08 14:20:12.306585 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:20:12.306393    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-kserve-dockercfg-m6qw5\""
Sep 08 14:20:12.306713 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:20:12.306696    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"seaweedfs-s3-creds\""
Sep 08 14:20:12.306779 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:20:12.306757    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-kserve-self-signed-certs\""
Sep 08 14:20:12.306883 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:20:12.306868    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"odh-kserve-custom-ca-bundle\""
Sep 08 14:20:12.325803 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:20:12.325780    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5b85f6885f-p85l9"]
Sep 08 14:20:12.359652 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:20:12.359625    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:20:12.370298 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:20:12.370274    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5643cbd-c7a5-4f1d-a9af-4d0212fdcc44-home\") pod \"custom-route-timeout-pd-test-kserve-5b85f6885f-p85l9\" (UID: \"b5643cbd-c7a5-4f1d-a9af-4d0212fdcc44\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5b85f6885f-p85l9"
Sep 08 14:20:12.370421 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:20:12.370303    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5643cbd-c7a5-4f1d-a9af-4d0212fdcc44-model-cache\") pod \"custom-route-timeout-pd-test-kserve-5b85f6885f-p85l9\" (UID: \"b5643cbd-c7a5-4f1d-a9af-4d0212fdcc44\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5b85f6885f-p85l9"
Sep 08 14:20:12.370421 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:20:12.370322    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b5643cbd-c7a5-4f1d-a9af-4d0212fdcc44-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-5b85f6885f-p85l9\" (UID: \"b5643cbd-c7a5-4f1d-a9af-4d0212fdcc44\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5b85f6885f-p85l9"
Sep 08 14:20:12.370421 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:20:12.370392    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5643cbd-c7a5-4f1d-a9af-4d0212fdcc44-dshm\") pod \"custom-route-timeout-pd-test-kserve-5b85f6885f-p85l9\" (UID: \"b5643cbd-c7a5-4f1d-a9af-4d0212fdcc44\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5b85f6885f-p85l9"
Sep 08 14:20:12.370421 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:20:12.370411    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5643cbd-c7a5-4f1d-a9af-4d0212fdcc44-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-5b85f6885f-p85l9\" (UID: \"b5643cbd-c7a5-4f1d-a9af-4d0212fdcc44\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5b85f6885f-p85l9"
Sep 08 14:20:12.370653 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:20:12.370459    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5643cbd-c7a5-4f1d-a9af-4d0212fdcc44-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-5b85f6885f-p85l9\" (UID: \"b5643cbd-c7a5-4f1d-a9af-4d0212fdcc44\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5b85f6885f-p85l9"
Sep 08 14:20:12.370653 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:20:12.370520    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r28nk\" (UniqueName: \"kubernetes.io/projected/b5643cbd-c7a5-4f1d-a9af-4d0212fdcc44-kube-api-access-r28nk\") pod \"custom-route-timeout-pd-test-kserve-5b85f6885f-p85l9\" (UID: \"b5643cbd-c7a5-4f1d-a9af-4d0212fdcc44\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5b85f6885f-p85l9"
Sep 08 14:20:12.370653 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:20:12.370556    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b5643cbd-c7a5-4f1d-a9af-4d0212fdcc44-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-5b85f6885f-p85l9\" (UID: \"b5643cbd-c7a5-4f1d-a9af-4d0212fdcc44\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5b85f6885f-p85l9"
Sep 08 14:20:12.470903 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:20:12.470871    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5643cbd-c7a5-4f1d-a9af-4d0212fdcc44-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-5b85f6885f-p85l9\" (UID: \"b5643cbd-c7a5-4f1d-a9af-4d0212fdcc44\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5b85f6885f-p85l9"
Sep 08 14:20:12.470903 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:20:12.470903    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r28nk\" (UniqueName: \"kubernetes.io/projected/b5643cbd-c7a5-4f1d-a9af-4d0212fdcc44-kube-api-access-r28nk\") pod \"custom-route-timeout-pd-test-kserve-5b85f6885f-p85l9\" (UID: \"b5643cbd-c7a5-4f1d-a9af-4d0212fdcc44\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5b85f6885f-p85l9"
Sep 08 14:20:12.471077 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:20:12.470923    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b5643cbd-c7a5-4f1d-a9af-4d0212fdcc44-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-5b85f6885f-p85l9\" (UID: \"b5643cbd-c7a5-4f1d-a9af-4d0212fdcc44\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5b85f6885f-p85l9"
Sep 08 14:20:12.471077 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:20:12.471047    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5643cbd-c7a5-4f1d-a9af-4d0212fdcc44-home\") pod \"custom-route-timeout-pd-test-kserve-5b85f6885f-p85l9\" (UID: \"b5643cbd-c7a5-4f1d-a9af-4d0212fdcc44\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5b85f6885f-p85l9"
Sep 08 14:20:12.471158 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:20:12.471095    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5643cbd-c7a5-4f1d-a9af-4d0212fdcc44-model-cache\") pod \"custom-route-timeout-pd-test-kserve-5b85f6885f-p85l9\" (UID: \"b5643cbd-c7a5-4f1d-a9af-4d0212fdcc44\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5b85f6885f-p85l9"
Sep 08 14:20:12.471158 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:20:12.471122    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b5643cbd-c7a5-4f1d-a9af-4d0212fdcc44-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-5b85f6885f-p85l9\" (UID: \"b5643cbd-c7a5-4f1d-a9af-4d0212fdcc44\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5b85f6885f-p85l9"
Sep 08 14:20:12.471255 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:20:12.471196    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5643cbd-c7a5-4f1d-a9af-4d0212fdcc44-dshm\") pod \"custom-route-timeout-pd-test-kserve-5b85f6885f-p85l9\" (UID: \"b5643cbd-c7a5-4f1d-a9af-4d0212fdcc44\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5b85f6885f-p85l9"
Sep 08 14:20:12.471312 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:20:12.471300    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5643cbd-c7a5-4f1d-a9af-4d0212fdcc44-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-5b85f6885f-p85l9\" (UID: \"b5643cbd-c7a5-4f1d-a9af-4d0212fdcc44\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5b85f6885f-p85l9"
Sep 08 14:20:12.471448 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:20:12.471423    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5643cbd-c7a5-4f1d-a9af-4d0212fdcc44-home\") pod \"custom-route-timeout-pd-test-kserve-5b85f6885f-p85l9\" (UID: \"b5643cbd-c7a5-4f1d-a9af-4d0212fdcc44\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5b85f6885f-p85l9"
Sep 08 14:20:12.471560 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:20:12.471533    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b5643cbd-c7a5-4f1d-a9af-4d0212fdcc44-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-5b85f6885f-p85l9\" (UID: \"b5643cbd-c7a5-4f1d-a9af-4d0212fdcc44\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5b85f6885f-p85l9"
Sep 08 14:20:12.471645 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:20:12.471627    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5643cbd-c7a5-4f1d-a9af-4d0212fdcc44-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-5b85f6885f-p85l9\" (UID: \"b5643cbd-c7a5-4f1d-a9af-4d0212fdcc44\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5b85f6885f-p85l9"
Sep 08 14:20:12.471707 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:20:12.471683    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b5643cbd-c7a5-4f1d-a9af-4d0212fdcc44-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-5b85f6885f-p85l9\" (UID: \"b5643cbd-c7a5-4f1d-a9af-4d0212fdcc44\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5b85f6885f-p85l9"
Sep 08 14:20:12.471766 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:20:12.471731    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5643cbd-c7a5-4f1d-a9af-4d0212fdcc44-model-cache\") pod \"custom-route-timeout-pd-test-kserve-5b85f6885f-p85l9\" (UID: \"b5643cbd-c7a5-4f1d-a9af-4d0212fdcc44\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5b85f6885f-p85l9"
Sep 08 14:20:12.473295 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:20:12.473275    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5643cbd-c7a5-4f1d-a9af-4d0212fdcc44-dshm\") pod \"custom-route-timeout-pd-test-kserve-5b85f6885f-p85l9\" (UID: \"b5643cbd-c7a5-4f1d-a9af-4d0212fdcc44\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5b85f6885f-p85l9"
Sep 08 14:20:12.473608 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:20:12.473590    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5643cbd-c7a5-4f1d-a9af-4d0212fdcc44-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-5b85f6885f-p85l9\" (UID: \"b5643cbd-c7a5-4f1d-a9af-4d0212fdcc44\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5b85f6885f-p85l9"
Sep 08 14:20:12.480579 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:20:12.480560    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r28nk\" (UniqueName: \"kubernetes.io/projected/b5643cbd-c7a5-4f1d-a9af-4d0212fdcc44-kube-api-access-r28nk\") pod \"custom-route-timeout-pd-test-kserve-5b85f6885f-p85l9\" (UID: \"b5643cbd-c7a5-4f1d-a9af-4d0212fdcc44\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5b85f6885f-p85l9"
Sep 08 14:20:12.612879 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:20:12.612802    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5b85f6885f-p85l9"
Sep 08 14:20:12.620325 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:20:12.620301    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:20:12.897824 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:20:12.897799    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5b85f6885f-p85l9"]
Sep 08 14:20:12.900294 ip-10-0-132-254 kubenswrapper[2573]: W0908 14:20:12.900275    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5643cbd_c7a5_4f1d_a9af_4d0212fdcc44.slice/crio-62f5e9e0b22e31c7e99931adff09bbc9e7e982b3e8c7d0a56ad19b79724ac97b WatchSource:0}: Error finding container 62f5e9e0b22e31c7e99931adff09bbc9e7e982b3e8c7d0a56ad19b79724ac97b: Status 404 returned error can't find the container with id 62f5e9e0b22e31c7e99931adff09bbc9e7e982b3e8c7d0a56ad19b79724ac97b
Sep 08 14:20:13.705042 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:20:13.705006    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5b85f6885f-p85l9" event={"ID":"b5643cbd-c7a5-4f1d-a9af-4d0212fdcc44","Type":"ContainerStarted","Data":"5bc6b375f28d6ec3d1b89387e7103bc4fdfba4662d1d590d64761b5d529e1883"}
Sep 08 14:20:13.705465 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:20:13.705059    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5b85f6885f-p85l9" event={"ID":"b5643cbd-c7a5-4f1d-a9af-4d0212fdcc44","Type":"ContainerStarted","Data":"62f5e9e0b22e31c7e99931adff09bbc9e7e982b3e8c7d0a56ad19b79724ac97b"}
Sep 08 14:20:13.705465 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:20:13.705129    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5b85f6885f-p85l9"
Sep 08 14:20:14.710694 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:20:14.710658    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5b85f6885f-p85l9" event={"ID":"b5643cbd-c7a5-4f1d-a9af-4d0212fdcc44","Type":"ContainerStarted","Data":"25201fd8133fdfac8b980875d9ff8d29e00e94a7184b8ed0d6953660d99b2c5d"}
Sep 08 14:20:16.720343 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:20:16.720309    2573 generic.go:356] "Generic (PLEG): container finished" podID="b5643cbd-c7a5-4f1d-a9af-4d0212fdcc44" containerID="25201fd8133fdfac8b980875d9ff8d29e00e94a7184b8ed0d6953660d99b2c5d" exitCode=0
Sep 08 14:20:16.720768 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:20:16.720381    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5b85f6885f-p85l9" event={"ID":"b5643cbd-c7a5-4f1d-a9af-4d0212fdcc44","Type":"ContainerDied","Data":"25201fd8133fdfac8b980875d9ff8d29e00e94a7184b8ed0d6953660d99b2c5d"}
Sep 08 14:20:17.727629 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:20:17.727595    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5b85f6885f-p85l9" event={"ID":"b5643cbd-c7a5-4f1d-a9af-4d0212fdcc44","Type":"ContainerStarted","Data":"461cc3430c7e0adb45db0542e83aafc5a854df6a15cd473c565e50af7ab583fe"}
Sep 08 14:20:17.754866 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:20:17.754805    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5b85f6885f-p85l9" podStartSLOduration=3.662643969 podStartE2EDuration="5.754786723s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.092142754s" isStatefulPod=false podCreationTimestamp="2026-09-08 14:20:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:20:17.753908859 +0000 UTC m=+3868.025356815" watchObservedRunningTime="2026-09-08 14:20:17.754786723 +0000 UTC m=+3868.026234655"
Sep 08 14:20:22.409170 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:20:22.409139    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:20:22.613153 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:20:22.613114    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5b85f6885f-p85l9"
Sep 08 14:20:22.613423 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:20:22.613191    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5b85f6885f-p85l9"
Sep 08 14:20:22.614586 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:20:22.614551    2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5b85f6885f-p85l9" podUID="b5643cbd-c7a5-4f1d-a9af-4d0212fdcc44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8001/health\": dial tcp 10.133.0.47:8001: connect: connection refused"
Sep 08 14:20:22.627133 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:20:22.627108    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5b85f6885f-p85l9"
Sep 08 14:20:32.613832 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:20:32.613784    2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5b85f6885f-p85l9" podUID="b5643cbd-c7a5-4f1d-a9af-4d0212fdcc44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8001/health\": dial tcp 10.133.0.47:8001: connect: connection refused"
Sep 08 14:20:42.613619 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:20:42.613573    2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5b85f6885f-p85l9" podUID="b5643cbd-c7a5-4f1d-a9af-4d0212fdcc44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8001/health\": dial tcp 10.133.0.47:8001: connect: connection refused"
Sep 08 14:20:46.187677 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:20:46.187648    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:20:50.436777 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:20:50.436746    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8h8cn_b936238e-bdf1-4dac-8bf8-6417c313597c/ovn-acl-logging/0.log"
Sep 08 14:20:50.447586 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:20:50.447554    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8h8cn_b936238e-bdf1-4dac-8bf8-6417c313597c/ovn-acl-logging/0.log"
Sep 08 14:20:52.614164 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:20:52.614126    2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5b85f6885f-p85l9" podUID="b5643cbd-c7a5-4f1d-a9af-4d0212fdcc44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8001/health\": dial tcp 10.133.0.47:8001: connect: connection refused"
Sep 08 14:21:02.614184 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:21:02.614133    2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5b85f6885f-p85l9" podUID="b5643cbd-c7a5-4f1d-a9af-4d0212fdcc44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8001/health\": dial tcp 10.133.0.47:8001: connect: connection refused"
Sep 08 14:21:06.189724 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:21:06.189688    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:21:06.287472 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:21:06.287439    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:21:12.174886 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:21:12.174849    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:21:12.276155 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:21:12.276123    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:21:12.614027 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:21:12.613914    2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5b85f6885f-p85l9" podUID="b5643cbd-c7a5-4f1d-a9af-4d0212fdcc44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8001/health\": dial tcp 10.133.0.47:8001: connect: connection refused"
Sep 08 14:21:20.324195 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:21:20.324126    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:21:20.561187 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:21:20.561147    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:21:22.613510 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:21:22.613465    2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5b85f6885f-p85l9" podUID="b5643cbd-c7a5-4f1d-a9af-4d0212fdcc44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8001/health\": dial tcp 10.133.0.47:8001: connect: connection refused"
Sep 08 14:21:29.827367 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:21:29.827330    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:21:32.613524 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:21:32.613460    2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5b85f6885f-p85l9" podUID="b5643cbd-c7a5-4f1d-a9af-4d0212fdcc44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8001/health\": dial tcp 10.133.0.47:8001: connect: connection refused"
Sep 08 14:21:42.613195 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:21:42.613144    2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5b85f6885f-p85l9" podUID="b5643cbd-c7a5-4f1d-a9af-4d0212fdcc44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8001/health\": dial tcp 10.133.0.47:8001: connect: connection refused"
Sep 08 14:21:52.614192 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:21:52.614141    2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5b85f6885f-p85l9" podUID="b5643cbd-c7a5-4f1d-a9af-4d0212fdcc44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8001/health\": dial tcp 10.133.0.47:8001: connect: connection refused"
Sep 08 14:21:53.951762 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:21:53.951713    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:22:02.614113 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:22:02.614069    2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5b85f6885f-p85l9" podUID="b5643cbd-c7a5-4f1d-a9af-4d0212fdcc44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8001/health\": dial tcp 10.133.0.47:8001: connect: connection refused"
Sep 08 14:22:12.623359 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:22:12.623325    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5b85f6885f-p85l9"
Sep 08 14:22:12.640328 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:22:12.640303    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5b85f6885f-p85l9"
Sep 08 14:22:21.658347 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:22:21.658311    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:22:21.680959 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:22:21.680923    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5b85f6885f-p85l9"]
Sep 08 14:22:21.732999 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:22:21.732969    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:22:36.704053 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:22:36.703994    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5b85f6885f-p85l9" podUID="b5643cbd-c7a5-4f1d-a9af-4d0212fdcc44" containerName="main" containerID="cri-o://461cc3430c7e0adb45db0542e83aafc5a854df6a15cd473c565e50af7ab583fe" gracePeriod=45
Sep 08 14:22:37.321745 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:22:37.321700    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5b85f6885f-p85l9" podUID="b5643cbd-c7a5-4f1d-a9af-4d0212fdcc44" containerName="llm-d-routing-sidecar" containerID="cri-o://5bc6b375f28d6ec3d1b89387e7103bc4fdfba4662d1d590d64761b5d529e1883" gracePeriod=45
Sep 08 14:22:37.338096 ip-10-0-132-254 kubenswrapper[2573]: E0908 14:22:37.338072    2573 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5643cbd_c7a5_4f1d_a9af_4d0212fdcc44.slice/crio-5bc6b375f28d6ec3d1b89387e7103bc4fdfba4662d1d590d64761b5d529e1883.scope\": RecentStats: unable to find data in memory cache]"
Sep 08 14:22:37.568643 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:22:37.568621    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5b85f6885f-p85l9"
Sep 08 14:22:37.654636 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:22:37.654606    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r28nk\" (UniqueName: \"kubernetes.io/projected/b5643cbd-c7a5-4f1d-a9af-4d0212fdcc44-kube-api-access-r28nk\") pod \"b5643cbd-c7a5-4f1d-a9af-4d0212fdcc44\" (UID: \"b5643cbd-c7a5-4f1d-a9af-4d0212fdcc44\") "
Sep 08 14:22:37.654791 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:22:37.654647    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5643cbd-c7a5-4f1d-a9af-4d0212fdcc44-tmp-dir\") pod \"b5643cbd-c7a5-4f1d-a9af-4d0212fdcc44\" (UID: \"b5643cbd-c7a5-4f1d-a9af-4d0212fdcc44\") "
Sep 08 14:22:37.654791 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:22:37.654697    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b5643cbd-c7a5-4f1d-a9af-4d0212fdcc44-kserve-provision-location\") pod \"b5643cbd-c7a5-4f1d-a9af-4d0212fdcc44\" (UID: \"b5643cbd-c7a5-4f1d-a9af-4d0212fdcc44\") "
Sep 08 14:22:37.654791 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:22:37.654716    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5643cbd-c7a5-4f1d-a9af-4d0212fdcc44-tls-certs\") pod \"b5643cbd-c7a5-4f1d-a9af-4d0212fdcc44\" (UID: \"b5643cbd-c7a5-4f1d-a9af-4d0212fdcc44\") "
Sep 08 14:22:37.654791 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:22:37.654766    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5643cbd-c7a5-4f1d-a9af-4d0212fdcc44-dshm\") pod \"b5643cbd-c7a5-4f1d-a9af-4d0212fdcc44\" (UID: \"b5643cbd-c7a5-4f1d-a9af-4d0212fdcc44\") "
Sep 08 14:22:37.654791 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:22:37.654792    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b5643cbd-c7a5-4f1d-a9af-4d0212fdcc44-cabundle-cert\") pod \"b5643cbd-c7a5-4f1d-a9af-4d0212fdcc44\" (UID: \"b5643cbd-c7a5-4f1d-a9af-4d0212fdcc44\") "
Sep 08 14:22:37.655076 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:22:37.654827    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5643cbd-c7a5-4f1d-a9af-4d0212fdcc44-home\") pod \"b5643cbd-c7a5-4f1d-a9af-4d0212fdcc44\" (UID: \"b5643cbd-c7a5-4f1d-a9af-4d0212fdcc44\") "
Sep 08 14:22:37.655076 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:22:37.654851    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5643cbd-c7a5-4f1d-a9af-4d0212fdcc44-model-cache\") pod \"b5643cbd-c7a5-4f1d-a9af-4d0212fdcc44\" (UID: \"b5643cbd-c7a5-4f1d-a9af-4d0212fdcc44\") "
Sep 08 14:22:37.655295 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:22:37.655271    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5643cbd-c7a5-4f1d-a9af-4d0212fdcc44-model-cache" (OuterVolumeSpecName: "model-cache") pod "b5643cbd-c7a5-4f1d-a9af-4d0212fdcc44" (UID: "b5643cbd-c7a5-4f1d-a9af-4d0212fdcc44"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:22:37.655401 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:22:37.655371    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5643cbd-c7a5-4f1d-a9af-4d0212fdcc44-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b5643cbd-c7a5-4f1d-a9af-4d0212fdcc44" (UID: "b5643cbd-c7a5-4f1d-a9af-4d0212fdcc44"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 14:22:37.655882 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:22:37.655851    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5643cbd-c7a5-4f1d-a9af-4d0212fdcc44-home" (OuterVolumeSpecName: "home") pod "b5643cbd-c7a5-4f1d-a9af-4d0212fdcc44" (UID: "b5643cbd-c7a5-4f1d-a9af-4d0212fdcc44"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:22:37.657009 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:22:37.656975    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5643cbd-c7a5-4f1d-a9af-4d0212fdcc44-kube-api-access-r28nk" (OuterVolumeSpecName: "kube-api-access-r28nk") pod "b5643cbd-c7a5-4f1d-a9af-4d0212fdcc44" (UID: "b5643cbd-c7a5-4f1d-a9af-4d0212fdcc44"). InnerVolumeSpecName "kube-api-access-r28nk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 14:22:37.657102 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:22:37.657056    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5643cbd-c7a5-4f1d-a9af-4d0212fdcc44-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b5643cbd-c7a5-4f1d-a9af-4d0212fdcc44" (UID: "b5643cbd-c7a5-4f1d-a9af-4d0212fdcc44"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 14:22:37.657158 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:22:37.657133    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5643cbd-c7a5-4f1d-a9af-4d0212fdcc44-dshm" (OuterVolumeSpecName: "dshm") pod "b5643cbd-c7a5-4f1d-a9af-4d0212fdcc44" (UID: "b5643cbd-c7a5-4f1d-a9af-4d0212fdcc44"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:22:37.668115 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:22:37.668093    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5643cbd-c7a5-4f1d-a9af-4d0212fdcc44-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b5643cbd-c7a5-4f1d-a9af-4d0212fdcc44" (UID: "b5643cbd-c7a5-4f1d-a9af-4d0212fdcc44"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:22:37.679291 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:22:37.679265    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5643cbd-c7a5-4f1d-a9af-4d0212fdcc44-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b5643cbd-c7a5-4f1d-a9af-4d0212fdcc44" (UID: "b5643cbd-c7a5-4f1d-a9af-4d0212fdcc44"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:22:37.755650 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:22:37.755588    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r28nk\" (UniqueName: \"kubernetes.io/projected/b5643cbd-c7a5-4f1d-a9af-4d0212fdcc44-kube-api-access-r28nk\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 14:22:37.755650 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:22:37.755612    2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5643cbd-c7a5-4f1d-a9af-4d0212fdcc44-tmp-dir\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 14:22:37.755650 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:22:37.755623    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b5643cbd-c7a5-4f1d-a9af-4d0212fdcc44-kserve-provision-location\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 14:22:37.755650 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:22:37.755631    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5643cbd-c7a5-4f1d-a9af-4d0212fdcc44-tls-certs\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 14:22:37.755650 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:22:37.755641    2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5643cbd-c7a5-4f1d-a9af-4d0212fdcc44-dshm\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 14:22:37.755650 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:22:37.755650    2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b5643cbd-c7a5-4f1d-a9af-4d0212fdcc44-cabundle-cert\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 14:22:37.756138 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:22:37.755657    2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5643cbd-c7a5-4f1d-a9af-4d0212fdcc44-home\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 14:22:37.756138 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:22:37.755664    2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5643cbd-c7a5-4f1d-a9af-4d0212fdcc44-model-cache\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 14:22:38.305110 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:22:38.305079    2573 generic.go:356] "Generic (PLEG): container finished" podID="b5643cbd-c7a5-4f1d-a9af-4d0212fdcc44" containerID="461cc3430c7e0adb45db0542e83aafc5a854df6a15cd473c565e50af7ab583fe" exitCode=0
Sep 08 14:22:38.305110 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:22:38.305106    2573 generic.go:356] "Generic (PLEG): container finished" podID="b5643cbd-c7a5-4f1d-a9af-4d0212fdcc44" containerID="5bc6b375f28d6ec3d1b89387e7103bc4fdfba4662d1d590d64761b5d529e1883" exitCode=0
Sep 08 14:22:38.305304 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:22:38.305132    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5b85f6885f-p85l9"
Sep 08 14:22:38.305304 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:22:38.305167    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5b85f6885f-p85l9" event={"ID":"b5643cbd-c7a5-4f1d-a9af-4d0212fdcc44","Type":"ContainerDied","Data":"461cc3430c7e0adb45db0542e83aafc5a854df6a15cd473c565e50af7ab583fe"}
Sep 08 14:22:38.305304 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:22:38.305219    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5b85f6885f-p85l9" event={"ID":"b5643cbd-c7a5-4f1d-a9af-4d0212fdcc44","Type":"ContainerDied","Data":"5bc6b375f28d6ec3d1b89387e7103bc4fdfba4662d1d590d64761b5d529e1883"}
Sep 08 14:22:38.305304 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:22:38.305237    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5b85f6885f-p85l9" event={"ID":"b5643cbd-c7a5-4f1d-a9af-4d0212fdcc44","Type":"ContainerDied","Data":"62f5e9e0b22e31c7e99931adff09bbc9e7e982b3e8c7d0a56ad19b79724ac97b"}
Sep 08 14:22:38.305304 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:22:38.305257    2573 scope.go:117] "RemoveContainer" containerID="461cc3430c7e0adb45db0542e83aafc5a854df6a15cd473c565e50af7ab583fe"
Sep 08 14:22:38.313275 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:22:38.313254    2573 scope.go:117] "RemoveContainer" containerID="25201fd8133fdfac8b980875d9ff8d29e00e94a7184b8ed0d6953660d99b2c5d"
Sep 08 14:22:38.320305 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:22:38.320290    2573 scope.go:117] "RemoveContainer" containerID="5bc6b375f28d6ec3d1b89387e7103bc4fdfba4662d1d590d64761b5d529e1883"
Sep 08 14:22:38.325095 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:22:38.325069    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5b85f6885f-p85l9"]
Sep 08 14:22:38.328291 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:22:38.328265    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5b85f6885f-p85l9"]
Sep 08 14:22:38.328598 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:22:38.328573    2573 scope.go:117] "RemoveContainer" containerID="461cc3430c7e0adb45db0542e83aafc5a854df6a15cd473c565e50af7ab583fe"
Sep 08 14:22:38.328905 ip-10-0-132-254 kubenswrapper[2573]: E0908 14:22:38.328879    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"461cc3430c7e0adb45db0542e83aafc5a854df6a15cd473c565e50af7ab583fe\": container with ID starting with 461cc3430c7e0adb45db0542e83aafc5a854df6a15cd473c565e50af7ab583fe not found: ID does not exist" containerID="461cc3430c7e0adb45db0542e83aafc5a854df6a15cd473c565e50af7ab583fe"
Sep 08 14:22:38.328984 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:22:38.328914    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"461cc3430c7e0adb45db0542e83aafc5a854df6a15cd473c565e50af7ab583fe"} err="failed to get container status \"461cc3430c7e0adb45db0542e83aafc5a854df6a15cd473c565e50af7ab583fe\": rpc error: code = NotFound desc = could not find container \"461cc3430c7e0adb45db0542e83aafc5a854df6a15cd473c565e50af7ab583fe\": container with ID starting with 461cc3430c7e0adb45db0542e83aafc5a854df6a15cd473c565e50af7ab583fe not found: ID does not exist"
Sep 08 14:22:38.328984 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:22:38.328931    2573 scope.go:117] "RemoveContainer" containerID="25201fd8133fdfac8b980875d9ff8d29e00e94a7184b8ed0d6953660d99b2c5d"
Sep 08 14:22:38.329223 ip-10-0-132-254 kubenswrapper[2573]: E0908 14:22:38.329209    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25201fd8133fdfac8b980875d9ff8d29e00e94a7184b8ed0d6953660d99b2c5d\": container with ID starting with 25201fd8133fdfac8b980875d9ff8d29e00e94a7184b8ed0d6953660d99b2c5d not found: ID does not exist" containerID="25201fd8133fdfac8b980875d9ff8d29e00e94a7184b8ed0d6953660d99b2c5d"
Sep 08 14:22:38.329272 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:22:38.329230    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25201fd8133fdfac8b980875d9ff8d29e00e94a7184b8ed0d6953660d99b2c5d"} err="failed to get container status \"25201fd8133fdfac8b980875d9ff8d29e00e94a7184b8ed0d6953660d99b2c5d\": rpc error: code = NotFound desc = could not find container \"25201fd8133fdfac8b980875d9ff8d29e00e94a7184b8ed0d6953660d99b2c5d\": container with ID starting with 25201fd8133fdfac8b980875d9ff8d29e00e94a7184b8ed0d6953660d99b2c5d not found: ID does not exist"
Sep 08 14:22:38.329272 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:22:38.329246    2573 scope.go:117] "RemoveContainer" containerID="5bc6b375f28d6ec3d1b89387e7103bc4fdfba4662d1d590d64761b5d529e1883"
Sep 08 14:22:38.329526 ip-10-0-132-254 kubenswrapper[2573]: E0908 14:22:38.329492    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5bc6b375f28d6ec3d1b89387e7103bc4fdfba4662d1d590d64761b5d529e1883\": container with ID starting with 5bc6b375f28d6ec3d1b89387e7103bc4fdfba4662d1d590d64761b5d529e1883 not found: ID does not exist" containerID="5bc6b375f28d6ec3d1b89387e7103bc4fdfba4662d1d590d64761b5d529e1883"
Sep 08 14:22:38.329620 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:22:38.329530    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bc6b375f28d6ec3d1b89387e7103bc4fdfba4662d1d590d64761b5d529e1883"} err="failed to get container status \"5bc6b375f28d6ec3d1b89387e7103bc4fdfba4662d1d590d64761b5d529e1883\": rpc error: code = NotFound desc = could not find container \"5bc6b375f28d6ec3d1b89387e7103bc4fdfba4662d1d590d64761b5d529e1883\": container with ID starting with 5bc6b375f28d6ec3d1b89387e7103bc4fdfba4662d1d590d64761b5d529e1883 not found: ID does not exist"
Sep 08 14:22:38.329620 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:22:38.329544    2573 scope.go:117] "RemoveContainer" containerID="461cc3430c7e0adb45db0542e83aafc5a854df6a15cd473c565e50af7ab583fe"
Sep 08 14:22:38.329820 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:22:38.329800    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"461cc3430c7e0adb45db0542e83aafc5a854df6a15cd473c565e50af7ab583fe"} err="failed to get container status \"461cc3430c7e0adb45db0542e83aafc5a854df6a15cd473c565e50af7ab583fe\": rpc error: code = NotFound desc = could not find container \"461cc3430c7e0adb45db0542e83aafc5a854df6a15cd473c565e50af7ab583fe\": container with ID starting with 461cc3430c7e0adb45db0542e83aafc5a854df6a15cd473c565e50af7ab583fe not found: ID does not exist"
Sep 08 14:22:38.329820 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:22:38.329821    2573 scope.go:117] "RemoveContainer" containerID="25201fd8133fdfac8b980875d9ff8d29e00e94a7184b8ed0d6953660d99b2c5d"
Sep 08 14:22:38.330076 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:22:38.330055    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25201fd8133fdfac8b980875d9ff8d29e00e94a7184b8ed0d6953660d99b2c5d"} err="failed to get container status \"25201fd8133fdfac8b980875d9ff8d29e00e94a7184b8ed0d6953660d99b2c5d\": rpc error: code = NotFound desc = could not find container \"25201fd8133fdfac8b980875d9ff8d29e00e94a7184b8ed0d6953660d99b2c5d\": container with ID starting with 25201fd8133fdfac8b980875d9ff8d29e00e94a7184b8ed0d6953660d99b2c5d not found: ID does not exist"
Sep 08 14:22:38.330124 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:22:38.330077    2573 scope.go:117] "RemoveContainer" containerID="5bc6b375f28d6ec3d1b89387e7103bc4fdfba4662d1d590d64761b5d529e1883"
Sep 08 14:22:38.330358 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:22:38.330333    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bc6b375f28d6ec3d1b89387e7103bc4fdfba4662d1d590d64761b5d529e1883"} err="failed to get container status \"5bc6b375f28d6ec3d1b89387e7103bc4fdfba4662d1d590d64761b5d529e1883\": rpc error: code = NotFound desc = could not find container \"5bc6b375f28d6ec3d1b89387e7103bc4fdfba4662d1d590d64761b5d529e1883\": container with ID starting with 5bc6b375f28d6ec3d1b89387e7103bc4fdfba4662d1d590d64761b5d529e1883 not found: ID does not exist"
Sep 08 14:22:40.179474 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:22:40.179443    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b5643cbd-c7a5-4f1d-a9af-4d0212fdcc44" path="/var/lib/kubelet/pods/b5643cbd-c7a5-4f1d-a9af-4d0212fdcc44/volumes"
Sep 08 14:23:13.295802 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:23:13.295722    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:23:13.395584 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:23:13.395554    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:23:15.422844 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:23:15.422805    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:23:18.767895 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:23:18.767855    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:23:24.710002 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:23:24.709970    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:23:24.778233 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:23:24.778165    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:23:30.790277 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:23:30.790240    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6cb6464687-j992x"]
Sep 08 14:23:30.790765 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:23:30.790686    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5643cbd-c7a5-4f1d-a9af-4d0212fdcc44" containerName="storage-initializer"
Sep 08 14:23:30.790765 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:23:30.790704    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5643cbd-c7a5-4f1d-a9af-4d0212fdcc44" containerName="storage-initializer"
Sep 08 14:23:30.790765 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:23:30.790731    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5643cbd-c7a5-4f1d-a9af-4d0212fdcc44" containerName="main"
Sep 08 14:23:30.790765 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:23:30.790740    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5643cbd-c7a5-4f1d-a9af-4d0212fdcc44" containerName="main"
Sep 08 14:23:30.790765 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:23:30.790753    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5643cbd-c7a5-4f1d-a9af-4d0212fdcc44" containerName="llm-d-routing-sidecar"
Sep 08 14:23:30.790765 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:23:30.790759    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5643cbd-c7a5-4f1d-a9af-4d0212fdcc44" containerName="llm-d-routing-sidecar"
Sep 08 14:23:30.791097 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:23:30.790805    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5643cbd-c7a5-4f1d-a9af-4d0212fdcc44" containerName="main"
Sep 08 14:23:30.791097 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:23:30.790812    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5643cbd-c7a5-4f1d-a9af-4d0212fdcc44" containerName="llm-d-routing-sidecar"
Sep 08 14:23:30.793936 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:23:30.793909    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6cb6464687-j992x"
Sep 08 14:23:30.796394 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:23:30.796368    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"seaweedfs-s3-creds\""
Sep 08 14:23:30.796534 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:23:30.796391    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-kserve-dockercfg-784f2\""
Sep 08 14:23:30.796534 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:23:30.796401    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"kube-root-ca.crt\""
Sep 08 14:23:30.796534 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:23:30.796391    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-kserve-self-signed-certs\""
Sep 08 14:23:30.796534 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:23:30.796459    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"default-dockercfg-m98fm\""
Sep 08 14:23:30.797145 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:23:30.797122    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"openshift-service-ca.crt\""
Sep 08 14:23:30.797235 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:23:30.797203    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"odh-kserve-custom-ca-bundle\""
Sep 08 14:23:30.808304 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:23:30.808285    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6cb6464687-j992x"]
Sep 08 14:23:30.834228 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:23:30.834206    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3a6d6b5-8775-4a31-ac89-dd246268bdb5-tls-certs\") pod \"router-with-refs-pd-test-kserve-6cb6464687-j992x\" (UID: \"b3a6d6b5-8775-4a31-ac89-dd246268bdb5\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6cb6464687-j992x"
Sep 08 14:23:30.834350 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:23:30.834237    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b3a6d6b5-8775-4a31-ac89-dd246268bdb5-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-6cb6464687-j992x\" (UID: \"b3a6d6b5-8775-4a31-ac89-dd246268bdb5\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6cb6464687-j992x"
Sep 08 14:23:30.834350 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:23:30.834262    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3a6d6b5-8775-4a31-ac89-dd246268bdb5-tmp-dir\") pod \"router-with-refs-pd-test-kserve-6cb6464687-j992x\" (UID: \"b3a6d6b5-8775-4a31-ac89-dd246268bdb5\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6cb6464687-j992x"
Sep 08 14:23:30.834350 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:23:30.834309    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3a6d6b5-8775-4a31-ac89-dd246268bdb5-dshm\") pod \"router-with-refs-pd-test-kserve-6cb6464687-j992x\" (UID: \"b3a6d6b5-8775-4a31-ac89-dd246268bdb5\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6cb6464687-j992x"
Sep 08 14:23:30.834350 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:23:30.834341    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3a6d6b5-8775-4a31-ac89-dd246268bdb5-model-cache\") pod \"router-with-refs-pd-test-kserve-6cb6464687-j992x\" (UID: \"b3a6d6b5-8775-4a31-ac89-dd246268bdb5\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6cb6464687-j992x"
Sep 08 14:23:30.834536 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:23:30.834378    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92djm\" (UniqueName: \"kubernetes.io/projected/b3a6d6b5-8775-4a31-ac89-dd246268bdb5-kube-api-access-92djm\") pod \"router-with-refs-pd-test-kserve-6cb6464687-j992x\" (UID: \"b3a6d6b5-8775-4a31-ac89-dd246268bdb5\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6cb6464687-j992x"
Sep 08 14:23:30.834536 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:23:30.834399    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b3a6d6b5-8775-4a31-ac89-dd246268bdb5-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-6cb6464687-j992x\" (UID: \"b3a6d6b5-8775-4a31-ac89-dd246268bdb5\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6cb6464687-j992x"
Sep 08 14:23:30.834536 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:23:30.834453    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3a6d6b5-8775-4a31-ac89-dd246268bdb5-home\") pod \"router-with-refs-pd-test-kserve-6cb6464687-j992x\" (UID: \"b3a6d6b5-8775-4a31-ac89-dd246268bdb5\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6cb6464687-j992x"
Sep 08 14:23:30.920538 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:23:30.920482    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:23:30.935630 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:23:30.935604    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b3a6d6b5-8775-4a31-ac89-dd246268bdb5-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-6cb6464687-j992x\" (UID: \"b3a6d6b5-8775-4a31-ac89-dd246268bdb5\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6cb6464687-j992x"
Sep 08 14:23:30.935730 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:23:30.935639    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3a6d6b5-8775-4a31-ac89-dd246268bdb5-home\") pod \"router-with-refs-pd-test-kserve-6cb6464687-j992x\" (UID: \"b3a6d6b5-8775-4a31-ac89-dd246268bdb5\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6cb6464687-j992x"
Sep 08 14:23:30.935730 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:23:30.935676    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3a6d6b5-8775-4a31-ac89-dd246268bdb5-tls-certs\") pod \"router-with-refs-pd-test-kserve-6cb6464687-j992x\" (UID: \"b3a6d6b5-8775-4a31-ac89-dd246268bdb5\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6cb6464687-j992x"
Sep 08 14:23:30.935730 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:23:30.935691    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b3a6d6b5-8775-4a31-ac89-dd246268bdb5-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-6cb6464687-j992x\" (UID: \"b3a6d6b5-8775-4a31-ac89-dd246268bdb5\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6cb6464687-j992x"
Sep 08 14:23:30.935730 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:23:30.935720    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3a6d6b5-8775-4a31-ac89-dd246268bdb5-tmp-dir\") pod \"router-with-refs-pd-test-kserve-6cb6464687-j992x\" (UID: \"b3a6d6b5-8775-4a31-ac89-dd246268bdb5\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6cb6464687-j992x"
Sep 08 14:23:30.935923 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:23:30.935744    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3a6d6b5-8775-4a31-ac89-dd246268bdb5-dshm\") pod \"router-with-refs-pd-test-kserve-6cb6464687-j992x\" (UID: \"b3a6d6b5-8775-4a31-ac89-dd246268bdb5\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6cb6464687-j992x"
Sep 08 14:23:30.935923 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:23:30.935773    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3a6d6b5-8775-4a31-ac89-dd246268bdb5-model-cache\") pod \"router-with-refs-pd-test-kserve-6cb6464687-j992x\" (UID: \"b3a6d6b5-8775-4a31-ac89-dd246268bdb5\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6cb6464687-j992x"
Sep 08 14:23:30.935923 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:23:30.935814    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92djm\" (UniqueName: \"kubernetes.io/projected/b3a6d6b5-8775-4a31-ac89-dd246268bdb5-kube-api-access-92djm\") pod \"router-with-refs-pd-test-kserve-6cb6464687-j992x\" (UID: \"b3a6d6b5-8775-4a31-ac89-dd246268bdb5\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6cb6464687-j992x"
Sep 08 14:23:30.936101 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:23:30.936077    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3a6d6b5-8775-4a31-ac89-dd246268bdb5-tmp-dir\") pod \"router-with-refs-pd-test-kserve-6cb6464687-j992x\" (UID: \"b3a6d6b5-8775-4a31-ac89-dd246268bdb5\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6cb6464687-j992x"
Sep 08 14:23:30.936168 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:23:30.936147    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3a6d6b5-8775-4a31-ac89-dd246268bdb5-home\") pod \"router-with-refs-pd-test-kserve-6cb6464687-j992x\" (UID: \"b3a6d6b5-8775-4a31-ac89-dd246268bdb5\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6cb6464687-j992x"
Sep 08 14:23:30.936238 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:23:30.936217    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b3a6d6b5-8775-4a31-ac89-dd246268bdb5-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-6cb6464687-j992x\" (UID: \"b3a6d6b5-8775-4a31-ac89-dd246268bdb5\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6cb6464687-j992x"
Sep 08 14:23:30.936310 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:23:30.936290    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3a6d6b5-8775-4a31-ac89-dd246268bdb5-model-cache\") pod \"router-with-refs-pd-test-kserve-6cb6464687-j992x\" (UID: \"b3a6d6b5-8775-4a31-ac89-dd246268bdb5\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6cb6464687-j992x"
Sep 08 14:23:30.936428 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:23:30.936406    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b3a6d6b5-8775-4a31-ac89-dd246268bdb5-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-6cb6464687-j992x\" (UID: \"b3a6d6b5-8775-4a31-ac89-dd246268bdb5\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6cb6464687-j992x"
Sep 08 14:23:30.937960 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:23:30.937941    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3a6d6b5-8775-4a31-ac89-dd246268bdb5-dshm\") pod \"router-with-refs-pd-test-kserve-6cb6464687-j992x\" (UID: \"b3a6d6b5-8775-4a31-ac89-dd246268bdb5\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6cb6464687-j992x"
Sep 08 14:23:30.938215 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:23:30.938199    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3a6d6b5-8775-4a31-ac89-dd246268bdb5-tls-certs\") pod \"router-with-refs-pd-test-kserve-6cb6464687-j992x\" (UID: \"b3a6d6b5-8775-4a31-ac89-dd246268bdb5\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6cb6464687-j992x"
Sep 08 14:23:30.944016 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:23:30.943996    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-92djm\" (UniqueName: \"kubernetes.io/projected/b3a6d6b5-8775-4a31-ac89-dd246268bdb5-kube-api-access-92djm\") pod \"router-with-refs-pd-test-kserve-6cb6464687-j992x\" (UID: \"b3a6d6b5-8775-4a31-ac89-dd246268bdb5\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6cb6464687-j992x"
Sep 08 14:23:31.103087 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:23:31.103022    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6cb6464687-j992x"
Sep 08 14:23:31.288661 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:23:31.288624    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:23:31.381698 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:23:31.381664    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6cb6464687-j992x"]
Sep 08 14:23:31.385755 ip-10-0-132-254 kubenswrapper[2573]: W0908 14:23:31.385728    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3a6d6b5_8775_4a31_ac89_dd246268bdb5.slice/crio-8f9efbb1e06e1c8398eaacd1fb61149867012c80fc086e4bd42879d1786a9b33 WatchSource:0}: Error finding container 8f9efbb1e06e1c8398eaacd1fb61149867012c80fc086e4bd42879d1786a9b33: Status 404 returned error can't find the container with id 8f9efbb1e06e1c8398eaacd1fb61149867012c80fc086e4bd42879d1786a9b33
Sep 08 14:23:31.386716 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:23:31.386663    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:23:31.505627 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:23:31.505599    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6cb6464687-j992x" event={"ID":"b3a6d6b5-8775-4a31-ac89-dd246268bdb5","Type":"ContainerStarted","Data":"4b760aeb541205dfba719f18375d0c41f69234a36facbaf55958201b0ecaa817"}
Sep 08 14:23:31.505774 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:23:31.505647    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6cb6464687-j992x" event={"ID":"b3a6d6b5-8775-4a31-ac89-dd246268bdb5","Type":"ContainerStarted","Data":"8f9efbb1e06e1c8398eaacd1fb61149867012c80fc086e4bd42879d1786a9b33"}
Sep 08 14:23:31.505774 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:23:31.505700    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6cb6464687-j992x"
Sep 08 14:23:32.510718 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:23:32.510684    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6cb6464687-j992x" event={"ID":"b3a6d6b5-8775-4a31-ac89-dd246268bdb5","Type":"ContainerStarted","Data":"e51f1635ee1dc44b28de275c3da443e20b3f942bcb9d27d5118b42c6a5bf347f"}
Sep 08 14:23:34.521066 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:23:34.521031    2573 generic.go:356] "Generic (PLEG): container finished" podID="b3a6d6b5-8775-4a31-ac89-dd246268bdb5" containerID="e51f1635ee1dc44b28de275c3da443e20b3f942bcb9d27d5118b42c6a5bf347f" exitCode=0
Sep 08 14:23:34.521580 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:23:34.521112    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6cb6464687-j992x" event={"ID":"b3a6d6b5-8775-4a31-ac89-dd246268bdb5","Type":"ContainerDied","Data":"e51f1635ee1dc44b28de275c3da443e20b3f942bcb9d27d5118b42c6a5bf347f"}
Sep 08 14:23:35.528014 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:23:35.527977    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6cb6464687-j992x" event={"ID":"b3a6d6b5-8775-4a31-ac89-dd246268bdb5","Type":"ContainerStarted","Data":"93af103560cbf2bb7ba8fb178660d1f1b682af1b23242683c815ade32460bb61"}
Sep 08 14:23:35.561798 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:23:35.561741    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6cb6464687-j992x" podStartSLOduration=3.479249529 podStartE2EDuration="5.561719446s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.082469917s" isStatefulPod=false podCreationTimestamp="2026-09-08 14:23:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:23:35.559994254 +0000 UTC m=+4065.831442198" watchObservedRunningTime="2026-09-08 14:23:35.561719446 +0000 UTC m=+4065.833167379"
Sep 08 14:23:41.104099 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:23:41.104045    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6cb6464687-j992x"
Sep 08 14:23:41.104523 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:23:41.104131    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6cb6464687-j992x"
Sep 08 14:23:41.105488 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:23:41.105457    2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6cb6464687-j992x" podUID="b3a6d6b5-8775-4a31-ac89-dd246268bdb5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8001/health\": dial tcp 10.133.0.48:8001: connect: connection refused"
Sep 08 14:23:41.123589 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:23:41.123561    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6cb6464687-j992x"
Sep 08 14:23:48.644797 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:23:48.644765    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:23:48.711279 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:23:48.711249    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:23:50.442119 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:23:50.442077    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:23:50.543888 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:23:50.543847    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:23:51.103612 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:23:51.103565    2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6cb6464687-j992x" podUID="b3a6d6b5-8775-4a31-ac89-dd246268bdb5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8001/health\": dial tcp 10.133.0.48:8001: connect: connection refused"
Sep 08 14:24:01.103862 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:24:01.103809    2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6cb6464687-j992x" podUID="b3a6d6b5-8775-4a31-ac89-dd246268bdb5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8001/health\": dial tcp 10.133.0.48:8001: connect: connection refused"
Sep 08 14:24:11.104138 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:24:11.104088    2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6cb6464687-j992x" podUID="b3a6d6b5-8775-4a31-ac89-dd246268bdb5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8001/health\": dial tcp 10.133.0.48:8001: connect: connection refused"
Sep 08 14:24:21.104213 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:24:21.104161    2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6cb6464687-j992x" podUID="b3a6d6b5-8775-4a31-ac89-dd246268bdb5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8001/health\": dial tcp 10.133.0.48:8001: connect: connection refused"
Sep 08 14:24:31.104410 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:24:31.104360    2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6cb6464687-j992x" podUID="b3a6d6b5-8775-4a31-ac89-dd246268bdb5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8001/health\": dial tcp 10.133.0.48:8001: connect: connection refused"
Sep 08 14:24:38.753793 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:24:38.753759    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:24:38.845351 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:24:38.845316    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:24:41.103767 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:24:41.103719    2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6cb6464687-j992x" podUID="b3a6d6b5-8775-4a31-ac89-dd246268bdb5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8001/health\": dial tcp 10.133.0.48:8001: connect: connection refused"
Sep 08 14:24:42.145116 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:24:42.145084    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:24:42.374690 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:24:42.374656    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-pglfh"]
Sep 08 14:24:42.378093 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:24:42.378070    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-pglfh"
Sep 08 14:24:42.380378 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:24:42.380353    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"prestop-hook-test-epp-sa-dockercfg-v6hgl\""
Sep 08 14:24:42.380933 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:24:42.380913    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"openshift-service-ca.crt\""
Sep 08 14:24:42.381097 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:24:42.381074    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"prestop-hook-test-kserve-self-signed-certs\""
Sep 08 14:24:42.381226 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:24:42.381124    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"default-dockercfg-4b2vf\""
Sep 08 14:24:42.381226 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:24:42.381015    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"kube-root-ca.crt\""
Sep 08 14:24:42.387649 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:24:42.387627    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-pglfh"]
Sep 08 14:24:42.403915 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:24:42.403858    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:24:42.537775 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:24:42.537737    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5ljq\" (UniqueName: \"kubernetes.io/projected/6695ab0e-e030-44bf-91d3-3bbe180b5c23-kube-api-access-m5ljq\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-pglfh\" (UID: \"6695ab0e-e030-44bf-91d3-3bbe180b5c23\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-pglfh"
Sep 08 14:24:42.537938 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:24:42.537789    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6695ab0e-e030-44bf-91d3-3bbe180b5c23-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-pglfh\" (UID: \"6695ab0e-e030-44bf-91d3-3bbe180b5c23\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-pglfh"
Sep 08 14:24:42.639202 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:24:42.639172    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5ljq\" (UniqueName: \"kubernetes.io/projected/6695ab0e-e030-44bf-91d3-3bbe180b5c23-kube-api-access-m5ljq\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-pglfh\" (UID: \"6695ab0e-e030-44bf-91d3-3bbe180b5c23\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-pglfh"
Sep 08 14:24:42.639355 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:24:42.639209    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6695ab0e-e030-44bf-91d3-3bbe180b5c23-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-pglfh\" (UID: \"6695ab0e-e030-44bf-91d3-3bbe180b5c23\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-pglfh"
Sep 08 14:24:42.641888 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:24:42.641860    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6695ab0e-e030-44bf-91d3-3bbe180b5c23-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-pglfh\" (UID: \"6695ab0e-e030-44bf-91d3-3bbe180b5c23\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-pglfh"
Sep 08 14:24:42.647279 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:24:42.647259    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5ljq\" (UniqueName: \"kubernetes.io/projected/6695ab0e-e030-44bf-91d3-3bbe180b5c23-kube-api-access-m5ljq\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-pglfh\" (UID: \"6695ab0e-e030-44bf-91d3-3bbe180b5c23\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-pglfh"
Sep 08 14:24:42.691139 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:24:42.691083    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-pglfh"
Sep 08 14:24:42.974745 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:24:42.974721    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-pglfh"]
Sep 08 14:24:42.977661 ip-10-0-132-254 kubenswrapper[2573]: W0908 14:24:42.977634    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6695ab0e_e030_44bf_91d3_3bbe180b5c23.slice/crio-b5020e56d39baa6ed68646944b3aea787bcd9421d115546c2ae3bc14e602cf80 WatchSource:0}: Error finding container b5020e56d39baa6ed68646944b3aea787bcd9421d115546c2ae3bc14e602cf80: Status 404 returned error can't find the container with id b5020e56d39baa6ed68646944b3aea787bcd9421d115546c2ae3bc14e602cf80
Sep 08 14:24:43.822028 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:24:43.821991    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-pglfh" event={"ID":"6695ab0e-e030-44bf-91d3-3bbe180b5c23","Type":"ContainerStarted","Data":"85e0924ba9e4d21e44c8a4268fd24cfc09695666a1c4c9c3dd613c23ef1997a9"}
Sep 08 14:24:43.822536 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:24:43.822039    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-pglfh" event={"ID":"6695ab0e-e030-44bf-91d3-3bbe180b5c23","Type":"ContainerStarted","Data":"b5020e56d39baa6ed68646944b3aea787bcd9421d115546c2ae3bc14e602cf80"}
Sep 08 14:24:43.822536 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:24:43.822076    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-pglfh"
Sep 08 14:24:43.841236 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:24:43.841179    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-pglfh" podStartSLOduration=1.841158496 podStartE2EDuration="1.841158496s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 14:24:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:24:43.840810714 +0000 UTC m=+4134.112258700" watchObservedRunningTime="2026-09-08 14:24:43.841158496 +0000 UTC m=+4134.112606431"
Sep 08 14:24:45.086639 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:24:45.086605    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:24:51.104392 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:24:51.104337    2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6cb6464687-j992x" podUID="b3a6d6b5-8775-4a31-ac89-dd246268bdb5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8001/health\": dial tcp 10.133.0.48:8001: connect: connection refused"
Sep 08 14:24:51.259742 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:24:51.259690    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:24:51.382819 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:24:51.382761    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:25:00.223991 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:25:00.223931    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:25:01.103845 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:25:01.103798    2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6cb6464687-j992x" podUID="b3a6d6b5-8775-4a31-ac89-dd246268bdb5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8001/health\": dial tcp 10.133.0.48:8001: connect: connection refused"
Sep 08 14:25:11.104431 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:25:11.104378    2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6cb6464687-j992x" podUID="b3a6d6b5-8775-4a31-ac89-dd246268bdb5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8001/health\": dial tcp 10.133.0.48:8001: connect: connection refused"
Sep 08 14:25:14.828341 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:25:14.828311    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-pglfh"
Sep 08 14:25:21.104281 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:25:21.104236    2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6cb6464687-j992x" podUID="b3a6d6b5-8775-4a31-ac89-dd246268bdb5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8001/health\": dial tcp 10.133.0.48:8001: connect: connection refused"
Sep 08 14:25:31.112992 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:25:31.112964    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6cb6464687-j992x"
Sep 08 14:25:31.125387 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:25:31.125365    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6cb6464687-j992x"
Sep 08 14:25:40.345008 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:25:40.344974    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:25:40.372142 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:25:40.372112    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6cb6464687-j992x"]
Sep 08 14:25:41.729927 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:25:41.729891    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:25:49.544410 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:25:49.544377    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:25:50.466202 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:25:50.466173    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8h8cn_b936238e-bdf1-4dac-8bf8-6417c313597c/ovn-acl-logging/0.log"
Sep 08 14:25:50.476319 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:25:50.476291    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8h8cn_b936238e-bdf1-4dac-8bf8-6417c313597c/ovn-acl-logging/0.log"
Sep 08 14:25:55.394484 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:25:55.394436    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6cb6464687-j992x" podUID="b3a6d6b5-8775-4a31-ac89-dd246268bdb5" containerName="main" containerID="cri-o://93af103560cbf2bb7ba8fb178660d1f1b682af1b23242683c815ade32460bb61" gracePeriod=45
Sep 08 14:25:55.937902 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:25:55.937873    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6cb6464687-j992x" podUID="b3a6d6b5-8775-4a31-ac89-dd246268bdb5" containerName="llm-d-routing-sidecar" containerID="cri-o://4b760aeb541205dfba719f18375d0c41f69234a36facbaf55958201b0ecaa817" gracePeriod=45
Sep 08 14:25:56.145063 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:25:56.144996    2573 generic.go:356] "Generic (PLEG): container finished" podID="b3a6d6b5-8775-4a31-ac89-dd246268bdb5" containerID="93af103560cbf2bb7ba8fb178660d1f1b682af1b23242683c815ade32460bb61" exitCode=0
Sep 08 14:25:56.145063 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:25:56.145023    2573 generic.go:356] "Generic (PLEG): container finished" podID="b3a6d6b5-8775-4a31-ac89-dd246268bdb5" containerID="4b760aeb541205dfba719f18375d0c41f69234a36facbaf55958201b0ecaa817" exitCode=0
Sep 08 14:25:56.145225 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:25:56.145070    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6cb6464687-j992x" event={"ID":"b3a6d6b5-8775-4a31-ac89-dd246268bdb5","Type":"ContainerDied","Data":"93af103560cbf2bb7ba8fb178660d1f1b682af1b23242683c815ade32460bb61"}
Sep 08 14:25:56.145225 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:25:56.145121    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6cb6464687-j992x" event={"ID":"b3a6d6b5-8775-4a31-ac89-dd246268bdb5","Type":"ContainerDied","Data":"4b760aeb541205dfba719f18375d0c41f69234a36facbaf55958201b0ecaa817"}
Sep 08 14:25:56.180668 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:25:56.180647    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6cb6464687-j992x"
Sep 08 14:25:56.338556 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:25:56.338517    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-92djm\" (UniqueName: \"kubernetes.io/projected/b3a6d6b5-8775-4a31-ac89-dd246268bdb5-kube-api-access-92djm\") pod \"b3a6d6b5-8775-4a31-ac89-dd246268bdb5\" (UID: \"b3a6d6b5-8775-4a31-ac89-dd246268bdb5\") "
Sep 08 14:25:56.338706 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:25:56.338580    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3a6d6b5-8775-4a31-ac89-dd246268bdb5-tls-certs\") pod \"b3a6d6b5-8775-4a31-ac89-dd246268bdb5\" (UID: \"b3a6d6b5-8775-4a31-ac89-dd246268bdb5\") "
Sep 08 14:25:56.338706 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:25:56.338603    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3a6d6b5-8775-4a31-ac89-dd246268bdb5-home\") pod \"b3a6d6b5-8775-4a31-ac89-dd246268bdb5\" (UID: \"b3a6d6b5-8775-4a31-ac89-dd246268bdb5\") "
Sep 08 14:25:56.338706 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:25:56.338630    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b3a6d6b5-8775-4a31-ac89-dd246268bdb5-cabundle-cert\") pod \"b3a6d6b5-8775-4a31-ac89-dd246268bdb5\" (UID: \"b3a6d6b5-8775-4a31-ac89-dd246268bdb5\") "
Sep 08 14:25:56.338706 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:25:56.338655    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3a6d6b5-8775-4a31-ac89-dd246268bdb5-tmp-dir\") pod \"b3a6d6b5-8775-4a31-ac89-dd246268bdb5\" (UID: \"b3a6d6b5-8775-4a31-ac89-dd246268bdb5\") "
Sep 08 14:25:56.338706 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:25:56.338668    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3a6d6b5-8775-4a31-ac89-dd246268bdb5-dshm\") pod \"b3a6d6b5-8775-4a31-ac89-dd246268bdb5\" (UID: \"b3a6d6b5-8775-4a31-ac89-dd246268bdb5\") "
Sep 08 14:25:56.338977 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:25:56.338724    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b3a6d6b5-8775-4a31-ac89-dd246268bdb5-kserve-provision-location\") pod \"b3a6d6b5-8775-4a31-ac89-dd246268bdb5\" (UID: \"b3a6d6b5-8775-4a31-ac89-dd246268bdb5\") "
Sep 08 14:25:56.338977 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:25:56.338753    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3a6d6b5-8775-4a31-ac89-dd246268bdb5-model-cache\") pod \"b3a6d6b5-8775-4a31-ac89-dd246268bdb5\" (UID: \"b3a6d6b5-8775-4a31-ac89-dd246268bdb5\") "
Sep 08 14:25:56.339087 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:25:56.339010    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3a6d6b5-8775-4a31-ac89-dd246268bdb5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b3a6d6b5-8775-4a31-ac89-dd246268bdb5" (UID: "b3a6d6b5-8775-4a31-ac89-dd246268bdb5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 14:25:56.339165 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:25:56.339136    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3a6d6b5-8775-4a31-ac89-dd246268bdb5-model-cache" (OuterVolumeSpecName: "model-cache") pod "b3a6d6b5-8775-4a31-ac89-dd246268bdb5" (UID: "b3a6d6b5-8775-4a31-ac89-dd246268bdb5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:25:56.339618 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:25:56.339595    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3a6d6b5-8775-4a31-ac89-dd246268bdb5-home" (OuterVolumeSpecName: "home") pod "b3a6d6b5-8775-4a31-ac89-dd246268bdb5" (UID: "b3a6d6b5-8775-4a31-ac89-dd246268bdb5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:25:56.341131 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:25:56.341100    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3a6d6b5-8775-4a31-ac89-dd246268bdb5-dshm" (OuterVolumeSpecName: "dshm") pod "b3a6d6b5-8775-4a31-ac89-dd246268bdb5" (UID: "b3a6d6b5-8775-4a31-ac89-dd246268bdb5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:25:56.341283 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:25:56.341264    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3a6d6b5-8775-4a31-ac89-dd246268bdb5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b3a6d6b5-8775-4a31-ac89-dd246268bdb5" (UID: "b3a6d6b5-8775-4a31-ac89-dd246268bdb5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 14:25:56.341352 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:25:56.341284    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3a6d6b5-8775-4a31-ac89-dd246268bdb5-kube-api-access-92djm" (OuterVolumeSpecName: "kube-api-access-92djm") pod "b3a6d6b5-8775-4a31-ac89-dd246268bdb5" (UID: "b3a6d6b5-8775-4a31-ac89-dd246268bdb5"). InnerVolumeSpecName "kube-api-access-92djm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 14:25:56.351162 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:25:56.351136    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3a6d6b5-8775-4a31-ac89-dd246268bdb5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b3a6d6b5-8775-4a31-ac89-dd246268bdb5" (UID: "b3a6d6b5-8775-4a31-ac89-dd246268bdb5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:25:56.361239 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:25:56.361218    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3a6d6b5-8775-4a31-ac89-dd246268bdb5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b3a6d6b5-8775-4a31-ac89-dd246268bdb5" (UID: "b3a6d6b5-8775-4a31-ac89-dd246268bdb5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:25:56.439832 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:25:56.439778    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b3a6d6b5-8775-4a31-ac89-dd246268bdb5-kserve-provision-location\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 14:25:56.439832 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:25:56.439802    2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3a6d6b5-8775-4a31-ac89-dd246268bdb5-model-cache\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 14:25:56.439832 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:25:56.439812    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92djm\" (UniqueName: \"kubernetes.io/projected/b3a6d6b5-8775-4a31-ac89-dd246268bdb5-kube-api-access-92djm\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 14:25:56.439832 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:25:56.439822    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3a6d6b5-8775-4a31-ac89-dd246268bdb5-tls-certs\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 14:25:56.439832 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:25:56.439830    2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3a6d6b5-8775-4a31-ac89-dd246268bdb5-home\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 14:25:56.440249 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:25:56.439838    2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b3a6d6b5-8775-4a31-ac89-dd246268bdb5-cabundle-cert\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 14:25:56.440249 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:25:56.439846    2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3a6d6b5-8775-4a31-ac89-dd246268bdb5-tmp-dir\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 14:25:56.440249 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:25:56.439853    2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3a6d6b5-8775-4a31-ac89-dd246268bdb5-dshm\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 14:25:57.151201 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:25:57.151169    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6cb6464687-j992x" event={"ID":"b3a6d6b5-8775-4a31-ac89-dd246268bdb5","Type":"ContainerDied","Data":"8f9efbb1e06e1c8398eaacd1fb61149867012c80fc086e4bd42879d1786a9b33"}
Sep 08 14:25:57.151379 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:25:57.151215    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6cb6464687-j992x"
Sep 08 14:25:57.151379 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:25:57.151236    2573 scope.go:117] "RemoveContainer" containerID="93af103560cbf2bb7ba8fb178660d1f1b682af1b23242683c815ade32460bb61"
Sep 08 14:25:57.160137 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:25:57.160119    2573 scope.go:117] "RemoveContainer" containerID="e51f1635ee1dc44b28de275c3da443e20b3f942bcb9d27d5118b42c6a5bf347f"
Sep 08 14:25:57.167467 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:25:57.167449    2573 scope.go:117] "RemoveContainer" containerID="4b760aeb541205dfba719f18375d0c41f69234a36facbaf55958201b0ecaa817"
Sep 08 14:25:57.172434 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:25:57.172415    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6cb6464687-j992x"]
Sep 08 14:25:57.177262 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:25:57.177237    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6cb6464687-j992x"]
Sep 08 14:25:58.173578 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:25:58.173547    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b3a6d6b5-8775-4a31-ac89-dd246268bdb5" path="/var/lib/kubelet/pods/b3a6d6b5-8775-4a31-ac89-dd246268bdb5/volumes"
Sep 08 14:26:30.610110 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:26:30.610073    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:26:32.863848 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:26:32.863805    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:26:35.539285 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:26:35.539250    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:26:35.723348 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:26:35.723318    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:26:35.811787 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:26:35.811713    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:26:44.489804 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:26:44.489768    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-576469645f-bcbcn"]
Sep 08 14:26:44.492160 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:26:44.490056    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3a6d6b5-8775-4a31-ac89-dd246268bdb5" containerName="storage-initializer"
Sep 08 14:26:44.492160 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:26:44.490069    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3a6d6b5-8775-4a31-ac89-dd246268bdb5" containerName="storage-initializer"
Sep 08 14:26:44.492160 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:26:44.490080    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3a6d6b5-8775-4a31-ac89-dd246268bdb5" containerName="main"
Sep 08 14:26:44.492160 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:26:44.490086    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3a6d6b5-8775-4a31-ac89-dd246268bdb5" containerName="main"
Sep 08 14:26:44.492160 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:26:44.490094    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3a6d6b5-8775-4a31-ac89-dd246268bdb5" containerName="llm-d-routing-sidecar"
Sep 08 14:26:44.492160 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:26:44.490100    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3a6d6b5-8775-4a31-ac89-dd246268bdb5" containerName="llm-d-routing-sidecar"
Sep 08 14:26:44.492160 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:26:44.490157    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3a6d6b5-8775-4a31-ac89-dd246268bdb5" containerName="main"
Sep 08 14:26:44.492160 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:26:44.490166    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3a6d6b5-8775-4a31-ac89-dd246268bdb5" containerName="llm-d-routing-sidecar"
Sep 08 14:26:44.493046 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:26:44.493031    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-576469645f-bcbcn"
Sep 08 14:26:44.495258 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:26:44.495237    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"seaweedfs-s3-creds\""
Sep 08 14:26:44.495258 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:26:44.495254    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"odh-kserve-custom-ca-bundle\""
Sep 08 14:26:44.504337 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:26:44.504312    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-576469645f-bcbcn"]
Sep 08 14:26:44.605115 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:26:44.605089    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e311cb0f-c5fe-4b36-8743-4902852728df-tmp-dir\") pod \"prestop-hook-test-kserve-576469645f-bcbcn\" (UID: \"e311cb0f-c5fe-4b36-8743-4902852728df\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-576469645f-bcbcn"
Sep 08 14:26:44.605237 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:26:44.605131    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e311cb0f-c5fe-4b36-8743-4902852728df-dshm\") pod \"prestop-hook-test-kserve-576469645f-bcbcn\" (UID: \"e311cb0f-c5fe-4b36-8743-4902852728df\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-576469645f-bcbcn"
Sep 08 14:26:44.605237 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:26:44.605153    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e311cb0f-c5fe-4b36-8743-4902852728df-cabundle-cert\") pod \"prestop-hook-test-kserve-576469645f-bcbcn\" (UID: \"e311cb0f-c5fe-4b36-8743-4902852728df\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-576469645f-bcbcn"
Sep 08 14:26:44.605323 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:26:44.605270    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e311cb0f-c5fe-4b36-8743-4902852728df-model-cache\") pod \"prestop-hook-test-kserve-576469645f-bcbcn\" (UID: \"e311cb0f-c5fe-4b36-8743-4902852728df\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-576469645f-bcbcn"
Sep 08 14:26:44.605323 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:26:44.605300    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e311cb0f-c5fe-4b36-8743-4902852728df-kserve-provision-location\") pod \"prestop-hook-test-kserve-576469645f-bcbcn\" (UID: \"e311cb0f-c5fe-4b36-8743-4902852728df\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-576469645f-bcbcn"
Sep 08 14:26:44.605323 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:26:44.605317    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7rdh\" (UniqueName: \"kubernetes.io/projected/e311cb0f-c5fe-4b36-8743-4902852728df-kube-api-access-m7rdh\") pod \"prestop-hook-test-kserve-576469645f-bcbcn\" (UID: \"e311cb0f-c5fe-4b36-8743-4902852728df\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-576469645f-bcbcn"
Sep 08 14:26:44.605471 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:26:44.605334    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e311cb0f-c5fe-4b36-8743-4902852728df-tls-certs\") pod \"prestop-hook-test-kserve-576469645f-bcbcn\" (UID: \"e311cb0f-c5fe-4b36-8743-4902852728df\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-576469645f-bcbcn"
Sep 08 14:26:44.605471 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:26:44.605353    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e311cb0f-c5fe-4b36-8743-4902852728df-home\") pod \"prestop-hook-test-kserve-576469645f-bcbcn\" (UID: \"e311cb0f-c5fe-4b36-8743-4902852728df\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-576469645f-bcbcn"
Sep 08 14:26:44.706650 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:26:44.706627    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e311cb0f-c5fe-4b36-8743-4902852728df-model-cache\") pod \"prestop-hook-test-kserve-576469645f-bcbcn\" (UID: \"e311cb0f-c5fe-4b36-8743-4902852728df\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-576469645f-bcbcn"
Sep 08 14:26:44.706768 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:26:44.706655    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e311cb0f-c5fe-4b36-8743-4902852728df-kserve-provision-location\") pod \"prestop-hook-test-kserve-576469645f-bcbcn\" (UID: \"e311cb0f-c5fe-4b36-8743-4902852728df\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-576469645f-bcbcn"
Sep 08 14:26:44.706768 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:26:44.706673    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7rdh\" (UniqueName: \"kubernetes.io/projected/e311cb0f-c5fe-4b36-8743-4902852728df-kube-api-access-m7rdh\") pod \"prestop-hook-test-kserve-576469645f-bcbcn\" (UID: \"e311cb0f-c5fe-4b36-8743-4902852728df\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-576469645f-bcbcn"
Sep 08 14:26:44.706768 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:26:44.706688    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e311cb0f-c5fe-4b36-8743-4902852728df-tls-certs\") pod \"prestop-hook-test-kserve-576469645f-bcbcn\" (UID: \"e311cb0f-c5fe-4b36-8743-4902852728df\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-576469645f-bcbcn"
Sep 08 14:26:44.706924 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:26:44.706769    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e311cb0f-c5fe-4b36-8743-4902852728df-home\") pod \"prestop-hook-test-kserve-576469645f-bcbcn\" (UID: \"e311cb0f-c5fe-4b36-8743-4902852728df\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-576469645f-bcbcn"
Sep 08 14:26:44.706924 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:26:44.706809    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e311cb0f-c5fe-4b36-8743-4902852728df-tmp-dir\") pod \"prestop-hook-test-kserve-576469645f-bcbcn\" (UID: \"e311cb0f-c5fe-4b36-8743-4902852728df\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-576469645f-bcbcn"
Sep 08 14:26:44.706924 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:26:44.706860    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e311cb0f-c5fe-4b36-8743-4902852728df-dshm\") pod \"prestop-hook-test-kserve-576469645f-bcbcn\" (UID: \"e311cb0f-c5fe-4b36-8743-4902852728df\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-576469645f-bcbcn"
Sep 08 14:26:44.706924 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:26:44.706896    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e311cb0f-c5fe-4b36-8743-4902852728df-cabundle-cert\") pod \"prestop-hook-test-kserve-576469645f-bcbcn\" (UID: \"e311cb0f-c5fe-4b36-8743-4902852728df\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-576469645f-bcbcn"
Sep 08 14:26:44.707142 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:26:44.707058    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e311cb0f-c5fe-4b36-8743-4902852728df-model-cache\") pod \"prestop-hook-test-kserve-576469645f-bcbcn\" (UID: \"e311cb0f-c5fe-4b36-8743-4902852728df\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-576469645f-bcbcn"
Sep 08 14:26:44.707142 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:26:44.707111    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e311cb0f-c5fe-4b36-8743-4902852728df-kserve-provision-location\") pod \"prestop-hook-test-kserve-576469645f-bcbcn\" (UID: \"e311cb0f-c5fe-4b36-8743-4902852728df\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-576469645f-bcbcn"
Sep 08 14:26:44.707221 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:26:44.707157    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e311cb0f-c5fe-4b36-8743-4902852728df-home\") pod \"prestop-hook-test-kserve-576469645f-bcbcn\" (UID: \"e311cb0f-c5fe-4b36-8743-4902852728df\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-576469645f-bcbcn"
Sep 08 14:26:44.707257 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:26:44.707228    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e311cb0f-c5fe-4b36-8743-4902852728df-tmp-dir\") pod \"prestop-hook-test-kserve-576469645f-bcbcn\" (UID: \"e311cb0f-c5fe-4b36-8743-4902852728df\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-576469645f-bcbcn"
Sep 08 14:26:44.707632 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:26:44.707612    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e311cb0f-c5fe-4b36-8743-4902852728df-cabundle-cert\") pod \"prestop-hook-test-kserve-576469645f-bcbcn\" (UID: \"e311cb0f-c5fe-4b36-8743-4902852728df\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-576469645f-bcbcn"
Sep 08 14:26:44.709062 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:26:44.709043    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e311cb0f-c5fe-4b36-8743-4902852728df-dshm\") pod \"prestop-hook-test-kserve-576469645f-bcbcn\" (UID: \"e311cb0f-c5fe-4b36-8743-4902852728df\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-576469645f-bcbcn"
Sep 08 14:26:44.709253 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:26:44.709234    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e311cb0f-c5fe-4b36-8743-4902852728df-tls-certs\") pod \"prestop-hook-test-kserve-576469645f-bcbcn\" (UID: \"e311cb0f-c5fe-4b36-8743-4902852728df\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-576469645f-bcbcn"
Sep 08 14:26:44.715670 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:26:44.715639    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7rdh\" (UniqueName: \"kubernetes.io/projected/e311cb0f-c5fe-4b36-8743-4902852728df-kube-api-access-m7rdh\") pod \"prestop-hook-test-kserve-576469645f-bcbcn\" (UID: \"e311cb0f-c5fe-4b36-8743-4902852728df\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-576469645f-bcbcn"
Sep 08 14:26:44.803446 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:26:44.803368    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-576469645f-bcbcn"
Sep 08 14:26:45.080539 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:26:45.080496    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-576469645f-bcbcn"]
Sep 08 14:26:45.082600 ip-10-0-132-254 kubenswrapper[2573]: W0908 14:26:45.082571    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode311cb0f_c5fe_4b36_8743_4902852728df.slice/crio-462e5156b414bb86c511bebf7245639b346d40529fdc3270b6cb3973e5fadfa5 WatchSource:0}: Error finding container 462e5156b414bb86c511bebf7245639b346d40529fdc3270b6cb3973e5fadfa5: Status 404 returned error can't find the container with id 462e5156b414bb86c511bebf7245639b346d40529fdc3270b6cb3973e5fadfa5
Sep 08 14:26:45.341267 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:26:45.341173    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-576469645f-bcbcn" event={"ID":"e311cb0f-c5fe-4b36-8743-4902852728df","Type":"ContainerStarted","Data":"b750eb9f64af9b7d50b8471cb0f6679bd9a8aa81eda22385b41d862bdb428bec"}
Sep 08 14:26:45.341267 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:26:45.341242    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-576469645f-bcbcn" event={"ID":"e311cb0f-c5fe-4b36-8743-4902852728df","Type":"ContainerStarted","Data":"462e5156b414bb86c511bebf7245639b346d40529fdc3270b6cb3973e5fadfa5"}
Sep 08 14:26:47.354436 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:26:47.354398    2573 generic.go:356] "Generic (PLEG): container finished" podID="e311cb0f-c5fe-4b36-8743-4902852728df" containerID="b750eb9f64af9b7d50b8471cb0f6679bd9a8aa81eda22385b41d862bdb428bec" exitCode=0
Sep 08 14:26:47.354865 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:26:47.354444    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-576469645f-bcbcn" event={"ID":"e311cb0f-c5fe-4b36-8743-4902852728df","Type":"ContainerDied","Data":"b750eb9f64af9b7d50b8471cb0f6679bd9a8aa81eda22385b41d862bdb428bec"}
Sep 08 14:26:48.361514 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:26:48.361459    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-576469645f-bcbcn" event={"ID":"e311cb0f-c5fe-4b36-8743-4902852728df","Type":"ContainerStarted","Data":"84054ac30454a4c423cc9ba699dfffbfc9c93e4cab8289137041f78f50143187"}
Sep 08 14:26:48.386960 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:26:48.386893    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-576469645f-bcbcn" podStartSLOduration=2.309424166 podStartE2EDuration="4.386869736s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.07744557s" isStatefulPod=false podCreationTimestamp="2026-09-08 14:26:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:26:48.383843138 +0000 UTC m=+4258.655291082" watchObservedRunningTime="2026-09-08 14:26:48.386869736 +0000 UTC m=+4258.658317668"
Sep 08 14:26:48.803647 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:26:48.803603    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-576469645f-bcbcn"
Sep 08 14:26:51.879389 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:26:51.879356    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:26:52.884581 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:26:52.884548    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:26:54.803768 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:26:54.803723    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-576469645f-bcbcn"
Sep 08 14:26:54.805140 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:26:54.805110    2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-576469645f-bcbcn" podUID="e311cb0f-c5fe-4b36-8743-4902852728df" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": dial tcp 10.133.0.50:8000: connect: connection refused"
Sep 08 14:26:57.179123 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:26:57.179091    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:26:59.118975 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:26:59.118942    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:27:04.804354 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:27:04.804316    2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-576469645f-bcbcn" podUID="e311cb0f-c5fe-4b36-8743-4902852728df" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": dial tcp 10.133.0.50:8000: connect: connection refused"
Sep 08 14:27:05.251474 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:27:05.251438    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:27:13.593478 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:27:13.593428    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:27:14.804677 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:27:14.804634    2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-576469645f-bcbcn" podUID="e311cb0f-c5fe-4b36-8743-4902852728df" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": dial tcp 10.133.0.50:8000: connect: connection refused"
Sep 08 14:27:20.204600 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:27:20.204561    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:27:20.279867 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:27:20.279833    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:27:23.226529 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:27:23.226478    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-pglfh"]
Sep 08 14:27:23.237610 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:27:23.237580    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-576469645f-bcbcn"]
Sep 08 14:27:24.828763 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:27:24.828720    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-pglfh" podUID="6695ab0e-e030-44bf-91d3-3bbe180b5c23" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 08 14:27:25.205270 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:27:25.205236    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:27:25.272979 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:27:25.272952    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:27:30.542954 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:27:30.542912    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:27:30.622877 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:27:30.622845    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:27:34.828544 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:27:34.828489    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-pglfh" podUID="6695ab0e-e030-44bf-91d3-3bbe180b5c23" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 08 14:27:38.248787 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:27:38.248730    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-pglfh" podUID="6695ab0e-e030-44bf-91d3-3bbe180b5c23" containerName="main" containerID="cri-o://85e0924ba9e4d21e44c8a4268fd24cfc09695666a1c4c9c3dd613c23ef1997a9" gracePeriod=45
Sep 08 14:27:38.262162 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:27:38.262130    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-576469645f-bcbcn" podUID="e311cb0f-c5fe-4b36-8743-4902852728df" containerName="main" containerID="cri-o://84054ac30454a4c423cc9ba699dfffbfc9c93e4cab8289137041f78f50143187" gracePeriod=45
Sep 08 14:27:40.004893 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:27:40.004865    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-prestop-hook-22727e61_prestop-hook-test-kserve-576469645f-bcbcn_e311cb0f-c5fe-4b36-8743-4902852728df/main/0.log"
Sep 08 14:27:40.007663 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:27:40.007642    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-576469645f-bcbcn"
Sep 08 14:27:40.069392 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:27:40.069327    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m7rdh\" (UniqueName: \"kubernetes.io/projected/e311cb0f-c5fe-4b36-8743-4902852728df-kube-api-access-m7rdh\") pod \"e311cb0f-c5fe-4b36-8743-4902852728df\" (UID: \"e311cb0f-c5fe-4b36-8743-4902852728df\") "
Sep 08 14:27:40.069392 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:27:40.069363    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e311cb0f-c5fe-4b36-8743-4902852728df-tmp-dir\") pod \"e311cb0f-c5fe-4b36-8743-4902852728df\" (UID: \"e311cb0f-c5fe-4b36-8743-4902852728df\") "
Sep 08 14:27:40.069666 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:27:40.069393    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e311cb0f-c5fe-4b36-8743-4902852728df-home\") pod \"e311cb0f-c5fe-4b36-8743-4902852728df\" (UID: \"e311cb0f-c5fe-4b36-8743-4902852728df\") "
Sep 08 14:27:40.069666 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:27:40.069436    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e311cb0f-c5fe-4b36-8743-4902852728df-cabundle-cert\") pod \"e311cb0f-c5fe-4b36-8743-4902852728df\" (UID: \"e311cb0f-c5fe-4b36-8743-4902852728df\") "
Sep 08 14:27:40.069666 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:27:40.069491    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e311cb0f-c5fe-4b36-8743-4902852728df-model-cache\") pod \"e311cb0f-c5fe-4b36-8743-4902852728df\" (UID: \"e311cb0f-c5fe-4b36-8743-4902852728df\") "
Sep 08 14:27:40.069666 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:27:40.069551    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e311cb0f-c5fe-4b36-8743-4902852728df-tls-certs\") pod \"e311cb0f-c5fe-4b36-8743-4902852728df\" (UID: \"e311cb0f-c5fe-4b36-8743-4902852728df\") "
Sep 08 14:27:40.069666 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:27:40.069589    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e311cb0f-c5fe-4b36-8743-4902852728df-dshm\") pod \"e311cb0f-c5fe-4b36-8743-4902852728df\" (UID: \"e311cb0f-c5fe-4b36-8743-4902852728df\") "
Sep 08 14:27:40.069666 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:27:40.069612    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e311cb0f-c5fe-4b36-8743-4902852728df-kserve-provision-location\") pod \"e311cb0f-c5fe-4b36-8743-4902852728df\" (UID: \"e311cb0f-c5fe-4b36-8743-4902852728df\") "
Sep 08 14:27:40.069994 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:27:40.069747    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e311cb0f-c5fe-4b36-8743-4902852728df-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e311cb0f-c5fe-4b36-8743-4902852728df" (UID: "e311cb0f-c5fe-4b36-8743-4902852728df"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 14:27:40.069994 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:27:40.069763    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e311cb0f-c5fe-4b36-8743-4902852728df-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e311cb0f-c5fe-4b36-8743-4902852728df" (UID: "e311cb0f-c5fe-4b36-8743-4902852728df"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:27:40.069994 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:27:40.069769    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e311cb0f-c5fe-4b36-8743-4902852728df-home" (OuterVolumeSpecName: "home") pod "e311cb0f-c5fe-4b36-8743-4902852728df" (UID: "e311cb0f-c5fe-4b36-8743-4902852728df"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:27:40.069994 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:27:40.069786    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e311cb0f-c5fe-4b36-8743-4902852728df-model-cache" (OuterVolumeSpecName: "model-cache") pod "e311cb0f-c5fe-4b36-8743-4902852728df" (UID: "e311cb0f-c5fe-4b36-8743-4902852728df"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:27:40.070218 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:27:40.070001    2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e311cb0f-c5fe-4b36-8743-4902852728df-tmp-dir\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 14:27:40.070218 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:27:40.070022    2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e311cb0f-c5fe-4b36-8743-4902852728df-home\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 14:27:40.070218 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:27:40.070032    2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e311cb0f-c5fe-4b36-8743-4902852728df-cabundle-cert\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 14:27:40.070218 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:27:40.070040    2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e311cb0f-c5fe-4b36-8743-4902852728df-model-cache\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 14:27:40.071803 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:27:40.071765    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e311cb0f-c5fe-4b36-8743-4902852728df-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e311cb0f-c5fe-4b36-8743-4902852728df" (UID: "e311cb0f-c5fe-4b36-8743-4902852728df"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 14:27:40.071905 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:27:40.071812    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e311cb0f-c5fe-4b36-8743-4902852728df-dshm" (OuterVolumeSpecName: "dshm") pod "e311cb0f-c5fe-4b36-8743-4902852728df" (UID: "e311cb0f-c5fe-4b36-8743-4902852728df"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:27:40.071905 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:27:40.071878    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e311cb0f-c5fe-4b36-8743-4902852728df-kube-api-access-m7rdh" (OuterVolumeSpecName: "kube-api-access-m7rdh") pod "e311cb0f-c5fe-4b36-8743-4902852728df" (UID: "e311cb0f-c5fe-4b36-8743-4902852728df"). InnerVolumeSpecName "kube-api-access-m7rdh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 14:27:40.092157 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:27:40.092135    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e311cb0f-c5fe-4b36-8743-4902852728df-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e311cb0f-c5fe-4b36-8743-4902852728df" (UID: "e311cb0f-c5fe-4b36-8743-4902852728df"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:27:40.175659 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:27:40.170887    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e311cb0f-c5fe-4b36-8743-4902852728df-tls-certs\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 14:27:40.175659 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:27:40.170927    2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e311cb0f-c5fe-4b36-8743-4902852728df-dshm\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 14:27:40.175659 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:27:40.170943    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e311cb0f-c5fe-4b36-8743-4902852728df-kserve-provision-location\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 14:27:40.175659 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:27:40.170959    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7rdh\" (UniqueName: \"kubernetes.io/projected/e311cb0f-c5fe-4b36-8743-4902852728df-kube-api-access-m7rdh\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 14:27:40.596796 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:27:40.596771    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-prestop-hook-22727e61_prestop-hook-test-kserve-576469645f-bcbcn_e311cb0f-c5fe-4b36-8743-4902852728df/main/0.log"
Sep 08 14:27:40.597123 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:27:40.597100    2573 generic.go:356] "Generic (PLEG): container finished" podID="e311cb0f-c5fe-4b36-8743-4902852728df" containerID="84054ac30454a4c423cc9ba699dfffbfc9c93e4cab8289137041f78f50143187" exitCode=130
Sep 08 14:27:40.597210 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:27:40.597157    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-576469645f-bcbcn"
Sep 08 14:27:40.597210 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:27:40.597181    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-576469645f-bcbcn" event={"ID":"e311cb0f-c5fe-4b36-8743-4902852728df","Type":"ContainerDied","Data":"84054ac30454a4c423cc9ba699dfffbfc9c93e4cab8289137041f78f50143187"}
Sep 08 14:27:40.597296 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:27:40.597236    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-576469645f-bcbcn" event={"ID":"e311cb0f-c5fe-4b36-8743-4902852728df","Type":"ContainerDied","Data":"462e5156b414bb86c511bebf7245639b346d40529fdc3270b6cb3973e5fadfa5"}
Sep 08 14:27:40.597296 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:27:40.597258    2573 scope.go:117] "RemoveContainer" containerID="84054ac30454a4c423cc9ba699dfffbfc9c93e4cab8289137041f78f50143187"
Sep 08 14:27:40.607474 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:27:40.607454    2573 scope.go:117] "RemoveContainer" containerID="b750eb9f64af9b7d50b8471cb0f6679bd9a8aa81eda22385b41d862bdb428bec"
Sep 08 14:27:40.613885 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:27:40.613846    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-576469645f-bcbcn"]
Sep 08 14:27:40.615303 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:27:40.615289    2573 scope.go:117] "RemoveContainer" containerID="84054ac30454a4c423cc9ba699dfffbfc9c93e4cab8289137041f78f50143187"
Sep 08 14:27:40.615587 ip-10-0-132-254 kubenswrapper[2573]: E0908 14:27:40.615565    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84054ac30454a4c423cc9ba699dfffbfc9c93e4cab8289137041f78f50143187\": container with ID starting with 84054ac30454a4c423cc9ba699dfffbfc9c93e4cab8289137041f78f50143187 not found: ID does not exist" containerID="84054ac30454a4c423cc9ba699dfffbfc9c93e4cab8289137041f78f50143187"
Sep 08 14:27:40.615652 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:27:40.615597    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84054ac30454a4c423cc9ba699dfffbfc9c93e4cab8289137041f78f50143187"} err="failed to get container status \"84054ac30454a4c423cc9ba699dfffbfc9c93e4cab8289137041f78f50143187\": rpc error: code = NotFound desc = could not find container \"84054ac30454a4c423cc9ba699dfffbfc9c93e4cab8289137041f78f50143187\": container with ID starting with 84054ac30454a4c423cc9ba699dfffbfc9c93e4cab8289137041f78f50143187 not found: ID does not exist"
Sep 08 14:27:40.615652 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:27:40.615614    2573 scope.go:117] "RemoveContainer" containerID="b750eb9f64af9b7d50b8471cb0f6679bd9a8aa81eda22385b41d862bdb428bec"
Sep 08 14:27:40.615843 ip-10-0-132-254 kubenswrapper[2573]: E0908 14:27:40.615826    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b750eb9f64af9b7d50b8471cb0f6679bd9a8aa81eda22385b41d862bdb428bec\": container with ID starting with b750eb9f64af9b7d50b8471cb0f6679bd9a8aa81eda22385b41d862bdb428bec not found: ID does not exist" containerID="b750eb9f64af9b7d50b8471cb0f6679bd9a8aa81eda22385b41d862bdb428bec"
Sep 08 14:27:40.615902 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:27:40.615851    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b750eb9f64af9b7d50b8471cb0f6679bd9a8aa81eda22385b41d862bdb428bec"} err="failed to get container status \"b750eb9f64af9b7d50b8471cb0f6679bd9a8aa81eda22385b41d862bdb428bec\": rpc error: code = NotFound desc = could not find container \"b750eb9f64af9b7d50b8471cb0f6679bd9a8aa81eda22385b41d862bdb428bec\": container with ID starting with b750eb9f64af9b7d50b8471cb0f6679bd9a8aa81eda22385b41d862bdb428bec not found: ID does not exist"
Sep 08 14:27:40.620181 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:27:40.620162    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-576469645f-bcbcn"]
Sep 08 14:27:42.175407 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:27:42.175362    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e311cb0f-c5fe-4b36-8743-4902852728df" path="/var/lib/kubelet/pods/e311cb0f-c5fe-4b36-8743-4902852728df/volumes"
Sep 08 14:27:44.828885 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:27:44.828801    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-pglfh" podUID="6695ab0e-e030-44bf-91d3-3bbe180b5c23" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 08 14:27:44.829341 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:27:44.828907    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-pglfh"
Sep 08 14:27:54.828403 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:27:54.828354    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-pglfh" podUID="6695ab0e-e030-44bf-91d3-3bbe180b5c23" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 08 14:28:04.828455 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:28:04.828410    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-pglfh" podUID="6695ab0e-e030-44bf-91d3-3bbe180b5c23" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 08 14:28:08.488553 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:28:08.488528    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-pglfh"
Sep 08 14:28:08.597262 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:28:08.597170    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m5ljq\" (UniqueName: \"kubernetes.io/projected/6695ab0e-e030-44bf-91d3-3bbe180b5c23-kube-api-access-m5ljq\") pod \"6695ab0e-e030-44bf-91d3-3bbe180b5c23\" (UID: \"6695ab0e-e030-44bf-91d3-3bbe180b5c23\") "
Sep 08 14:28:08.597262 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:28:08.597264    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6695ab0e-e030-44bf-91d3-3bbe180b5c23-tls-certs\") pod \"6695ab0e-e030-44bf-91d3-3bbe180b5c23\" (UID: \"6695ab0e-e030-44bf-91d3-3bbe180b5c23\") "
Sep 08 14:28:08.599269 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:28:08.599242    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6695ab0e-e030-44bf-91d3-3bbe180b5c23-kube-api-access-m5ljq" (OuterVolumeSpecName: "kube-api-access-m5ljq") pod "6695ab0e-e030-44bf-91d3-3bbe180b5c23" (UID: "6695ab0e-e030-44bf-91d3-3bbe180b5c23"). InnerVolumeSpecName "kube-api-access-m5ljq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 14:28:08.599363 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:28:08.599252    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6695ab0e-e030-44bf-91d3-3bbe180b5c23-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6695ab0e-e030-44bf-91d3-3bbe180b5c23" (UID: "6695ab0e-e030-44bf-91d3-3bbe180b5c23"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 14:28:08.698630 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:28:08.698588    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5ljq\" (UniqueName: \"kubernetes.io/projected/6695ab0e-e030-44bf-91d3-3bbe180b5c23-kube-api-access-m5ljq\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 14:28:08.698630 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:28:08.698628    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6695ab0e-e030-44bf-91d3-3bbe180b5c23-tls-certs\") on node \"ip-10-0-132-254.ec2.internal\" DevicePath \"\""
Sep 08 14:28:08.713066 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:28:08.713031    2573 generic.go:356] "Generic (PLEG): container finished" podID="6695ab0e-e030-44bf-91d3-3bbe180b5c23" containerID="85e0924ba9e4d21e44c8a4268fd24cfc09695666a1c4c9c3dd613c23ef1997a9" exitCode=0
Sep 08 14:28:08.713205 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:28:08.713077    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-pglfh"
Sep 08 14:28:08.713205 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:28:08.713118    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-pglfh" event={"ID":"6695ab0e-e030-44bf-91d3-3bbe180b5c23","Type":"ContainerDied","Data":"85e0924ba9e4d21e44c8a4268fd24cfc09695666a1c4c9c3dd613c23ef1997a9"}
Sep 08 14:28:08.713205 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:28:08.713175    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-pglfh" event={"ID":"6695ab0e-e030-44bf-91d3-3bbe180b5c23","Type":"ContainerDied","Data":"b5020e56d39baa6ed68646944b3aea787bcd9421d115546c2ae3bc14e602cf80"}
Sep 08 14:28:08.713205 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:28:08.713203    2573 scope.go:117] "RemoveContainer" containerID="85e0924ba9e4d21e44c8a4268fd24cfc09695666a1c4c9c3dd613c23ef1997a9"
Sep 08 14:28:08.721348 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:28:08.721331    2573 scope.go:117] "RemoveContainer" containerID="85e0924ba9e4d21e44c8a4268fd24cfc09695666a1c4c9c3dd613c23ef1997a9"
Sep 08 14:28:08.721633 ip-10-0-132-254 kubenswrapper[2573]: E0908 14:28:08.721613    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85e0924ba9e4d21e44c8a4268fd24cfc09695666a1c4c9c3dd613c23ef1997a9\": container with ID starting with 85e0924ba9e4d21e44c8a4268fd24cfc09695666a1c4c9c3dd613c23ef1997a9 not found: ID does not exist" containerID="85e0924ba9e4d21e44c8a4268fd24cfc09695666a1c4c9c3dd613c23ef1997a9"
Sep 08 14:28:08.721695 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:28:08.721640    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85e0924ba9e4d21e44c8a4268fd24cfc09695666a1c4c9c3dd613c23ef1997a9"} err="failed to get container status \"85e0924ba9e4d21e44c8a4268fd24cfc09695666a1c4c9c3dd613c23ef1997a9\": rpc error: code = NotFound desc = could not find container \"85e0924ba9e4d21e44c8a4268fd24cfc09695666a1c4c9c3dd613c23ef1997a9\": container with ID starting with 85e0924ba9e4d21e44c8a4268fd24cfc09695666a1c4c9c3dd613c23ef1997a9 not found: ID does not exist"
Sep 08 14:28:08.734012 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:28:08.733988    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-pglfh"]
Sep 08 14:28:08.737773 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:28:08.737754    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-pglfh"]
Sep 08 14:28:10.174780 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:28:10.174752    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6695ab0e-e030-44bf-91d3-3bbe180b5c23" path="/var/lib/kubelet/pods/6695ab0e-e030-44bf-91d3-3bbe180b5c23/volumes"
Sep 08 14:28:11.086964 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:28:11.086933    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:28:18.055278 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:28:18.055248    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:28:18.128322 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:28:18.128282    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:28:20.250652 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:28:20.250621    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:28:20.326036 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:28:20.325997    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:28:24.668851 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:28:24.668818    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:28:24.767229 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:28:24.767197    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:28:29.522859 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:28:29.522826    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:28:31.073063 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:28:31.073021    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:28:44.926660 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:28:44.926631    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-69549dc55b-qs44g_871377db-9869-404a-b14a-343321527fdb/router/0.log"
Sep 08 14:28:45.886543 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:28:45.886511    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-69549dc55b-qs44g_871377db-9869-404a-b14a-343321527fdb/router/0.log"
Sep 08 14:28:47.080279 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:28:47.080253    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-ls9ww_d422a1d5-7838-4d16-919a-c6c81d36b614/manager/0.log"
Sep 08 14:28:47.261399 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:28:47.261366    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-r4wjf_05b740c9-d688-487a-8d60-cc248a226a1c/limitador/0.log"
Sep 08 14:28:47.280453 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:28:47.280427    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-mmlq6_c099226c-7892-45ed-81c4-e3a353c6c340/manager/0.log"
Sep 08 14:28:51.624417 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:28:51.624381    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:28:51.835689 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:28:51.835657    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:28:51.947154 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:28:51.947071    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:28:52.445803 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:28:52.445775    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:28:52.554749 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:28:52.554719    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:28:54.990950 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:28:54.990916    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-6qqfz_9e7df160-d688-4877-895e-edc5f73db16a/global-pull-secret-syncer/0.log"
Sep 08 14:28:55.177061 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:28:55.177029    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log"
Sep 08 14:28:55.433219 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:28:55.433187    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-132-254.ec2.internal_7dfffcd72254a48b318c2c1ea59c531d/haproxy/0.log"
Sep 08 14:29:00.261587 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:29:00.261554    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:29:00.422264 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:29:00.422234    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:29:01.042899 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:29:01.042870    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-ls9ww_d422a1d5-7838-4d16-919a-c6c81d36b614/manager/0.log"
Sep 08 14:29:01.430893 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:29:01.430866    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-r4wjf_05b740c9-d688-487a-8d60-cc248a226a1c/limitador/0.log"
Sep 08 14:29:01.477766 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:29:01.477742    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-mmlq6_c099226c-7892-45ed-81c4-e3a353c6c340/manager/0.log"
Sep 08 14:29:02.808532 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:29:02.808453    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wrhb_9c39b80f-3adc-45fd-b1ec-39df3c237b02/konnectivity-agent/0.log" limit=1024
Sep 08 14:29:03.509982 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:29:03.509954    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-64b7f4f884-ds5dg_0a3e2a72-2d91-42e4-b48e-ecaccc6cafd8/metrics-server/0.log"
Sep 08 14:29:03.620980 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:29:03.620958    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nv6pk_07f9b84f-8894-43de-95df-ee95e733be31/node-exporter/0.log"
Sep 08 14:29:03.663463 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:29:03.663442    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nv6pk_07f9b84f-8894-43de-95df-ee95e733be31/kube-rbac-proxy/0.log"
Sep 08 14:29:03.705231 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:29:03.705205    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nv6pk_07f9b84f-8894-43de-95df-ee95e733be31/init-textfile/0.log"
Sep 08 14:29:04.132903 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:29:04.132864    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_89edf18f-ede2-4d2b-9ad4-8fcdaa4d3cff/prometheus/0.log"
Sep 08 14:29:04.168114 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:29:04.168086    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_89edf18f-ede2-4d2b-9ad4-8fcdaa4d3cff/config-reloader/0.log"
Sep 08 14:29:04.208932 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:29:04.208911    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_89edf18f-ede2-4d2b-9ad4-8fcdaa4d3cff/thanos-sidecar/0.log"
Sep 08 14:29:04.247470 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:29:04.247451    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_89edf18f-ede2-4d2b-9ad4-8fcdaa4d3cff/kube-rbac-proxy-web/0.log"
Sep 08 14:29:04.290901 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:29:04.290882    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_89edf18f-ede2-4d2b-9ad4-8fcdaa4d3cff/kube-rbac-proxy/0.log"
Sep 08 14:29:04.340358 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:29:04.340338    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_89edf18f-ede2-4d2b-9ad4-8fcdaa4d3cff/kube-rbac-proxy-thanos/0.log"
Sep 08 14:29:04.387746 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:29:04.387697    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_89edf18f-ede2-4d2b-9ad4-8fcdaa4d3cff/init-config-reloader/0.log"
Sep 08 14:29:04.434094 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:29:04.434072    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-6f4946db56-qnzxl_b3d87f50-8434-482b-b645-5118e2960fda/prometheus-operator/0.log"
Sep 08 14:29:04.469924 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:29:04.469904    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-6f4946db56-qnzxl_b3d87f50-8434-482b-b645-5118e2960fda/kube-rbac-proxy/0.log"
Sep 08 14:29:04.842529 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:29:04.842443    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-nm4b8/perf-node-gather-daemonset-p9mkq"]
Sep 08 14:29:04.842733 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:29:04.842721    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e311cb0f-c5fe-4b36-8743-4902852728df" containerName="storage-initializer"
Sep 08 14:29:04.842783 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:29:04.842735    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="e311cb0f-c5fe-4b36-8743-4902852728df" containerName="storage-initializer"
Sep 08 14:29:04.842783 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:29:04.842749    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e311cb0f-c5fe-4b36-8743-4902852728df" containerName="main"
Sep 08 14:29:04.842783 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:29:04.842756    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="e311cb0f-c5fe-4b36-8743-4902852728df" containerName="main"
Sep 08 14:29:04.842783 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:29:04.842766    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6695ab0e-e030-44bf-91d3-3bbe180b5c23" containerName="main"
Sep 08 14:29:04.842783 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:29:04.842771    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="6695ab0e-e030-44bf-91d3-3bbe180b5c23" containerName="main"
Sep 08 14:29:04.842944 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:29:04.842824    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="6695ab0e-e030-44bf-91d3-3bbe180b5c23" containerName="main"
Sep 08 14:29:04.842944 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:29:04.842834    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="e311cb0f-c5fe-4b36-8743-4902852728df" containerName="main"
Sep 08 14:29:04.846967 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:29:04.846952    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-nm4b8/perf-node-gather-daemonset-p9mkq"
Sep 08 14:29:04.855891 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:29:04.855873    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-nm4b8\"/\"openshift-service-ca.crt\""
Sep 08 14:29:04.856012 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:29:04.855897    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-nm4b8\"/\"kube-root-ca.crt\""
Sep 08 14:29:04.856067 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:29:04.856020    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-nm4b8\"/\"default-dockercfg-fsrkr\""
Sep 08 14:29:04.867182 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:29:04.867160    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-nm4b8/perf-node-gather-daemonset-p9mkq"]
Sep 08 14:29:04.919352 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:29:04.919324    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ba8dfefb-23de-49ae-a543-6c75e379a752-lib-modules\") pod \"perf-node-gather-daemonset-p9mkq\" (UID: \"ba8dfefb-23de-49ae-a543-6c75e379a752\") " pod="openshift-must-gather-nm4b8/perf-node-gather-daemonset-p9mkq"
Sep 08 14:29:04.919473 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:29:04.919370    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ba8dfefb-23de-49ae-a543-6c75e379a752-sys\") pod \"perf-node-gather-daemonset-p9mkq\" (UID: \"ba8dfefb-23de-49ae-a543-6c75e379a752\") " pod="openshift-must-gather-nm4b8/perf-node-gather-daemonset-p9mkq"
Sep 08 14:29:04.919473 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:29:04.919393    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ba8dfefb-23de-49ae-a543-6c75e379a752-podres\") pod \"perf-node-gather-daemonset-p9mkq\" (UID: \"ba8dfefb-23de-49ae-a543-6c75e379a752\") " pod="openshift-must-gather-nm4b8/perf-node-gather-daemonset-p9mkq"
Sep 08 14:29:04.919593 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:29:04.919478    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ba8dfefb-23de-49ae-a543-6c75e379a752-proc\") pod \"perf-node-gather-daemonset-p9mkq\" (UID: \"ba8dfefb-23de-49ae-a543-6c75e379a752\") " pod="openshift-must-gather-nm4b8/perf-node-gather-daemonset-p9mkq"
Sep 08 14:29:04.919593 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:29:04.919569    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ph8pt\" (UniqueName: \"kubernetes.io/projected/ba8dfefb-23de-49ae-a543-6c75e379a752-kube-api-access-ph8pt\") pod \"perf-node-gather-daemonset-p9mkq\" (UID: \"ba8dfefb-23de-49ae-a543-6c75e379a752\") " pod="openshift-must-gather-nm4b8/perf-node-gather-daemonset-p9mkq"
Sep 08 14:29:05.020164 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:29:05.020138    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ba8dfefb-23de-49ae-a543-6c75e379a752-lib-modules\") pod \"perf-node-gather-daemonset-p9mkq\" (UID: \"ba8dfefb-23de-49ae-a543-6c75e379a752\") " pod="openshift-must-gather-nm4b8/perf-node-gather-daemonset-p9mkq"
Sep 08 14:29:05.020279 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:29:05.020174    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ba8dfefb-23de-49ae-a543-6c75e379a752-sys\") pod \"perf-node-gather-daemonset-p9mkq\" (UID: \"ba8dfefb-23de-49ae-a543-6c75e379a752\") " pod="openshift-must-gather-nm4b8/perf-node-gather-daemonset-p9mkq"
Sep 08 14:29:05.020279 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:29:05.020191    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ba8dfefb-23de-49ae-a543-6c75e379a752-podres\") pod \"perf-node-gather-daemonset-p9mkq\" (UID: \"ba8dfefb-23de-49ae-a543-6c75e379a752\") " pod="openshift-must-gather-nm4b8/perf-node-gather-daemonset-p9mkq"
Sep 08 14:29:05.020279 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:29:05.020233    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ba8dfefb-23de-49ae-a543-6c75e379a752-proc\") pod \"perf-node-gather-daemonset-p9mkq\" (UID: \"ba8dfefb-23de-49ae-a543-6c75e379a752\") " pod="openshift-must-gather-nm4b8/perf-node-gather-daemonset-p9mkq"
Sep 08 14:29:05.020279 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:29:05.020257    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ph8pt\" (UniqueName: \"kubernetes.io/projected/ba8dfefb-23de-49ae-a543-6c75e379a752-kube-api-access-ph8pt\") pod \"perf-node-gather-daemonset-p9mkq\" (UID: \"ba8dfefb-23de-49ae-a543-6c75e379a752\") " pod="openshift-must-gather-nm4b8/perf-node-gather-daemonset-p9mkq"
Sep 08 14:29:05.020279 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:29:05.020269    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ba8dfefb-23de-49ae-a543-6c75e379a752-sys\") pod \"perf-node-gather-daemonset-p9mkq\" (UID: \"ba8dfefb-23de-49ae-a543-6c75e379a752\") " pod="openshift-must-gather-nm4b8/perf-node-gather-daemonset-p9mkq"
Sep 08 14:29:05.020467 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:29:05.020298    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ba8dfefb-23de-49ae-a543-6c75e379a752-lib-modules\") pod \"perf-node-gather-daemonset-p9mkq\" (UID: \"ba8dfefb-23de-49ae-a543-6c75e379a752\") " pod="openshift-must-gather-nm4b8/perf-node-gather-daemonset-p9mkq"
Sep 08 14:29:05.020467 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:29:05.020325    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ba8dfefb-23de-49ae-a543-6c75e379a752-proc\") pod \"perf-node-gather-daemonset-p9mkq\" (UID: \"ba8dfefb-23de-49ae-a543-6c75e379a752\") " pod="openshift-must-gather-nm4b8/perf-node-gather-daemonset-p9mkq"
Sep 08 14:29:05.020467 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:29:05.020380    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ba8dfefb-23de-49ae-a543-6c75e379a752-podres\") pod \"perf-node-gather-daemonset-p9mkq\" (UID: \"ba8dfefb-23de-49ae-a543-6c75e379a752\") " pod="openshift-must-gather-nm4b8/perf-node-gather-daemonset-p9mkq"
Sep 08 14:29:05.034199 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:29:05.034177    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ph8pt\" (UniqueName: \"kubernetes.io/projected/ba8dfefb-23de-49ae-a543-6c75e379a752-kube-api-access-ph8pt\") pod \"perf-node-gather-daemonset-p9mkq\" (UID: \"ba8dfefb-23de-49ae-a543-6c75e379a752\") " pod="openshift-must-gather-nm4b8/perf-node-gather-daemonset-p9mkq"
Sep 08 14:29:05.156251 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:29:05.156230    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-nm4b8/perf-node-gather-daemonset-p9mkq"
Sep 08 14:29:05.444294 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:29:05.444221    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-nm4b8/perf-node-gather-daemonset-p9mkq"]
Sep 08 14:29:05.447162 ip-10-0-132-254 kubenswrapper[2573]: W0908 14:29:05.447128    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podba8dfefb_23de_49ae_a543_6c75e379a752.slice/crio-a5cfc94f530df4fd4580e94f23e61460bba52ab978a2bb5d4e7f75e744e4dcaf WatchSource:0}: Error finding container a5cfc94f530df4fd4580e94f23e61460bba52ab978a2bb5d4e7f75e744e4dcaf: Status 404 returned error can't find the container with id a5cfc94f530df4fd4580e94f23e61460bba52ab978a2bb5d4e7f75e744e4dcaf
Sep 08 14:29:05.934681 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:29:05.934645    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nm4b8/perf-node-gather-daemonset-p9mkq" event={"ID":"ba8dfefb-23de-49ae-a543-6c75e379a752","Type":"ContainerStarted","Data":"e1f58766fa66af70cc9604a8fd031468703098cf25e93e1f32fd72740e0ee1d8"}
Sep 08 14:29:05.934845 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:29:05.934705    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nm4b8/perf-node-gather-daemonset-p9mkq" event={"ID":"ba8dfefb-23de-49ae-a543-6c75e379a752","Type":"ContainerStarted","Data":"a5cfc94f530df4fd4580e94f23e61460bba52ab978a2bb5d4e7f75e744e4dcaf"}
Sep 08 14:29:05.934845 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:29:05.934778    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-nm4b8/perf-node-gather-daemonset-p9mkq"
Sep 08 14:29:06.200532 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:29:06.200435    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-79977f9844-wfwsl_ec967b4f-3738-4934-84bc-2f6ab72f8a24/networking-console-plugin/0.log"
Sep 08 14:29:09.260690 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:29:09.260659    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-knfp2_21ee5269-0851-405f-a9a5-f9d2741e8500/dns/0.log"
Sep 08 14:29:09.309623 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:29:09.309594    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-knfp2_21ee5269-0851-405f-a9a5-f9d2741e8500/kube-rbac-proxy/0.log"
Sep 08 14:29:09.424400 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:29:09.424368    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-rbnnh_42aa8a5e-3aba-4581-9f95-f3b978c250f0/dns-node-resolver/0.log"
Sep 08 14:29:10.081365 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:29:10.081335    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-24f78_137be755-3092-45ba-a6d2-8a025c5e6226/node-ca/0.log"
Sep 08 14:29:11.384815 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:29:11.384789    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-69549dc55b-qs44g_871377db-9869-404a-b14a-343321527fdb/router/0.log"
Sep 08 14:29:11.947566 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:29:11.947535    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-nm4b8/perf-node-gather-daemonset-p9mkq"
Sep 08 14:29:11.970621 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:29:11.970575    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-nm4b8/perf-node-gather-daemonset-p9mkq" podStartSLOduration=7.970560593 podStartE2EDuration="7.970560593s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 14:29:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:29:05.966579831 +0000 UTC m=+4396.238027774" watchObservedRunningTime="2026-09-08 14:29:11.970560593 +0000 UTC m=+4402.242008523"
Sep 08 14:29:12.033016 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:29:12.032990    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-qmrbg_831913e7-ea98-4d0b-bbaf-411e16552f73/serve-healthcheck-canary/0.log"
Sep 08 14:29:12.724016 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:29:12.723984    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-kgz9w_6934e1da-3606-4819-a625-60e175d0c1ce/kube-rbac-proxy/0.log"
Sep 08 14:29:12.754843 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:29:12.754815    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-kgz9w_6934e1da-3606-4819-a625-60e175d0c1ce/exporter/0.log"
Sep 08 14:29:12.779410 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:29:12.779353    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-kgz9w_6934e1da-3606-4819-a625-60e175d0c1ce/extractor/0.log"
Sep 08 14:29:15.518438 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:29:15.518415    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-69549dc55b-qs44g_871377db-9869-404a-b14a-343321527fdb/router/0.log"
Sep 08 14:29:16.616261 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:29:16.616227    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-66c568d8-gsld6_edb013d6-1adf-4545-b602-1d3617f9fc25/manager/0.log"
Sep 08 14:29:16.816798 ip-10-0-132-254 kubenswrapper[2573]: I0908 14:29:16.816726    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-69549dc55b-qs44g_871377db-9869-404a-b14a-343321527fdb/router/0.log"